Fusing Physical and Cognitive Stimuli: An Eye Movement Emotion Recognition Framework Based on Hierarchical Attention Mechanism

Eye movement signals, as physiological signals that are resistant to interference and closely related to emotions, have been widely applied in multimodal emotion recognition research. However, existing studies often focus on directly utilizing eye movement signals for emotion recognition, with few exploring the interaction between different video stimuli and eye movement signals from the perspective of Human-Computer Interaction (HCI). Research in cognitive neuroscience has revealed that eye movement signals are not only directly influenced by physical visual information such as light and brightness during observation but also by high-level visual features resulting from top-down cognitive processing of the stimulus materials in the brain. The sequential stimulation of these two types of visual features both affects eye movement signals and reflects the corresponding emotional states through these signals. Inspired by these findings, this study designs a hierarchical attention mechanism-based emotion recognition framework(HAMER) to simulate the process by which eye movement signals respond to stimuli, enabling interaction between video information and physiological signals in HCI. The framework demonstrates excellent performance on two emotion recognition datasets, VLMED and MAHNOB-HCI, which contain eye movement signals and video information, providing innovative perspectives and empirical support for emotion recognition based on physiological signals and video information in the field of HCI.