|
|
基于脑电信号和眼动信号的情感识别
|
Abstract:
本文探讨了基于脑电信号和眼动信号的多模态情感识别方法，使用了上海交通大学SEED-IV和SEED-V数据集。通过预处理脑电信号并提取功率谱密度和微分熵特征，结合眼动信号的时域和频域特征，提出了一种基于对比学习的数据融合方法，显著提高了情感识别的准确率。实验结果表明，多模态融合，尤其是脑电信号与眼动信号的结合，能显著提升识别性能；同时，微分熵特征优于功率谱密度特征，具有更好的分类准确性。
This paper explores a multimodal emotion recognition method based on EEG and eye movement signals, using the SEED-IV and SEED-V datasets from Shanghai Jiao Tong University. After preprocessing the EEG signals, power spectral density and differential entropy features were extracted, while time-domain and frequency-domain features were extracted from the eye movement signals. A contrastive learning-based data fusion method is proposed, which effectively bridges the distribution gap between different modalities and significantly improves emotion recognition accuracy. Experimental results show that multimodal fusion, particularly the combination of EEG and eye movement signals, significantly enhances recognition performance. Additionally, the differential entropy feature of EEG signals outperforms the power spectral density feature in classification accuracy, indicating better separability in the dataset.
| [1] | Maithri, M., Raghavendra, U., Gudigar, A., Samanth, J., Prabal Datta Barua,, Murugappan, M., et al. (2022) Automated Emotion Recognition: Current Trends and Future Perspectives. Computer Methods and Programs in Biomedicine, 215, Article ID: 106646. https://doi.org/10.1016/j.cmpb.2022.106646 |
| [2] | Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., García, D.L., et al. (2023) Emotion Recognition in EEG Signals Using Deep Learning Methods: A Review. Computers in Biology and Medicine, 165, Article ID: 107450. https://doi.org/10.1016/j.compbiomed.2023.107450 |
| [3] | Zhang, X., Liu, J., Shen, J., Li, S., Hou, K., Hu, B., et al. (2021) Emotion Recognition from Multimodal Physiological Signals Using a Regularized Deep Fusion of Kernel Machine. IEEE Transactions on Cybernetics, 51, 4386-4399. https://doi.org/10.1109/tcyb.2020.2987575 |
| [4] | Wang, Q., Wang, M., Yang, Y. and Zhang, X. (2022) Multi-Modal Emotion Recognition Using EEG and Speech Signals. Computers in Biology and Medicine, 149, Article ID: 105907. https://doi.org/10.1016/j.compbiomed.2022.105907 |
| [5] | Zhu, Q., Lu, G. and Yan, J. (2020) Valence-Arousal Model Based Emotion Recognition Using EEG, Peripheral Physiological Signals and Facial Expression. Proceedings of the 4th International Conference on Machine Learning and Soft Computing, Haiphong City, 17-19 January 2020, 81-85. https://doi.org/10.1145/3380688.3380694 |
| [6] | Zhang, J., Yin, Z., Chen, P. and Nichele, S. (2020) Emotion Recognition Using Multi-Modal Data and Machine Learning Techniques: A Tutorial and Review. Information Fusion, 59, 103-126. https://doi.org/10.1016/j.inffus.2020.01.011 |
| [7] | Zheng, W., Liu, W., Lu, Y., Lu, B. and Cichocki, A. (2019) Emotionmeter: A Multimodal Framework for Recognizing Human Emotions. IEEE Transactions on Cybernetics, 49, 1110-1122. https://doi.org/10.1109/tcyb.2018.2797176 |
| [8] | Liu, W., Qiu, J., Zheng, W. and Lu, B. (2022) Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition. IEEE Transactions on Cognitive and Developmental Systems, 14, 715-729. https://doi.org/10.1109/tcds.2021.3071170 |
| [9] | Liu, H., Zhang, J., Liu, Q. and Cao, J. (2022) Minimum Spanning Tree Based Graph Neural Network for Emotion Classification Using Eeg. Neural Networks, 145, 308-318. https://doi.org/10.1016/j.neunet.2021.10.023 |
| [10] | Bouazizi, S., Benmohamed, E. and Ltifi, H. (2023) Enhancing EEG-Based Emotion Recognition Using PSD-Grouped Deep Echo State Network. JUCS—Journal of Universal Computer Science, 29, 1116-1138. https://doi.org/10.3897/jucs.98789 |
| [11] | Hwang, S., Hong, K., Son, G. and Byun, H. (2019) Learning CNN Features from DE Features for EEG-Based Emotion Recognition. Pattern Analysis and Applications, 23, 1323-1335. https://doi.org/10.1007/s10044-019-00860-w |
| [12] | Li, M., Yang, B., Levy, J., Stolcke, A., Rozgic, V., Matsoukas, S., et al. (2021) Contrastive Unsupervised Learning for Speech Emotion Recognition. ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, 6-11 June 2021, 6329-6333. https://doi.org/10.1109/icassp39728.2021.9413910 |
| [13] | Pan, C., Shi, C., Mu, H., Li, J. and Gao, X. (2020) EEG-Based Emotion Recognition Using Logistic Regression with Gaussian Kernel and Laplacian Prior and Investigation of Critical Frequency Bands. Applied Sciences, 10, Article 1619. https://doi.org/10.3390/app10051619 |
| [14] | Chen, L., Su, W., Feng, Y., Wu, M., She, J. and Hirota, K. (2020) Two-Layer Fuzzy Multiple Random Forest for Speech Emotion Recognition in Human-Robot Interaction. Information Sciences, 509, 150-163. https://doi.org/10.1016/j.ins.2019.09.005 |
| [15] | Shen, P., Changjun, Z. and Chen, X. (2011) Automatic Speech Emotion Recognition Using Support Vector Machine. Proceedings of 2011 International Conference on Electronic & Mechanical Engineering and Information Technology, Harbin, 12-14 August 2011, 621-625. https://doi.org/10.1109/emeit.2011.6023178 |
| [16] | Al Dujaili, M.J., Ebrahimi-Moghadam, A. and Fatlawi, A. (2021) Speech Emotion Recognition Based on SVM and KNN Classifications Fusion. International Journal of Electrical and Computer Engineering (IJECE), 11, 1259-1264. https://doi.org/10.11591/ijece.v11i2.pp1259-1264 |
| [17] | Oktavia, N.Y., Wibawa, A.D., Pane, E.S. and Purnomo, M.H. (2019) Human Emotion Classification Based on EEG Signals Using Naïve Bayes Method. 2019 International Seminar on Application for Technology of Information and Communication (iSemantic), Semarang, 21-22 September 2019, 319-324. https://doi.org/10.1109/isemantic.2019.8884224 |
