Emotion Recognition in EEG Signals Using Phase Lock Value and Differential Entropy Features with the Help of One-Dimensional Convolutional Neural Network

Year 2023, , 725 - 734, 01.09.2023

Hakan Uyanık , Salih Taha Alperen Özçelik , Abdülkadir Şengür

https://doi.org/10.35234/fumbd.1242223

Abstract

Emotion recognition is one of the most researched fields in today's scientific world. It is a subject that is closely examined by disciplines such as neuroscience and psychology, and it is more and more involved in our daily lives, especially in human-computer interaction area. Although methods such as speech signals, facial expressions, body language, and facial expressions are used for emotion analysis, these methods do not give as reliable results as biological signals because they are open to manipulation. In this study, a new method for emotion recognition with electroencephalography (EEG) signals, which is a bioelectrical signal prepared with the help of virtual reality (VR) technology, is proposed. In this method, differential entropy (DE) and phase-locking value (PLV) properties of sub-bands of EEG signals were used to recognize positive and negative emotions with the help of a designed one-dimensional convolutional neural network (1D-CNN). The feature matrices obtained with the help of both features were tested ten times and average accuracy values were obtained. As a result of these tests, the highest average accuracy scores with DE and FKD features were obtained as 74.061±1.41% and 63.7590±1.72%, respectively, by combining all sub-band feature matrices. In addition, the higher accuracy rates of the tests of the high-frequency signal components obtained in the study compared to the low-frequency bands, supported the results of similar studies carried out in this area before.

Keywords

Emotion recognition, electroencephalography, differential entropy, phase locking value, one dimensional convolutional neural network

Bir Boyutlu Evrişimsel Sinir Ağı Yardımıyla Faz Kilitleme Değeri ve Diferansiyel Entropi Özellikleri Kullanılarak EEG Sinyallerinde Duygu Tanınması

Abstract

Duygu analizi günümüz bilim dünyasında üzerinde en çok araştırma yapılan alanların başında gelmektedir. Özellikle insan-bilgisayar etkileşimi gibi günlük hayatımıza her geçen gün daha çok dahil olan alanların yanı sıra nörobilim ve psikoloji gibi bilim dallarının da yakından incelediği bir konudur. Duygu analizi için konuşma sinyalleri, mimikler, vücut dili, yüz ifadeleri gibi yöntemler kullanılsa da bu yöntemler manipülasyona açık oldukları için biyolojik sinyaller kadar güvenilir sonuçlar vermezler. Bu çalışmada sanal gerçeklik (SG) teknolojisi yardımıyla hazırlanmış, biyoelektriksel bir sinyal olan elektroansefalografi (EEG) sinyalleri ile duygu tanıma için yeni bir yöntem önerilmiştir. Bu yöntemde EEG sinyallerinin alt bantlarının diferansiyel entropi (DE) ve faz kilitleme değeri (FKD) özellikleri, tasarlanan bir boyutlu evrişimsel sinir ağı (1B-ESA) yardımı ile pozitif ve negatif duyguların tanınması için kullanılmıştır. Her iki özellik yardımıyla elde edilen özellik matrisleri on defa teste tâbi tutularak ortalama başarı değerleri elde edilmiştir. Bu testler sonucunda DE ve FKD özellikleri ile en yüksek ortalama başarı puanları, tüm alt bant özellik matrislerinin birleştirilmesi ile sırasıyla %74,0611,41 ve %63,75901,72 olarak elde edilmiştir. Ayrıca çalışmada elde edilen yüksek frekanstaki sinyal bileşenlerine ait testlerin başarı oranlarının düşük frekans bantlarına göre daha yüksek elde edilmesi daha önce bu alanda yapılan benzer çalışmaların sonuçlarını destekler nitelikte olmuştur.

Keywords

Duygu tanıma, elektroansefalografi, diferansiyel entropi, faz kilitleme değeri, bir boyutlu evrişimsel sinir ağı

References

• Chalmers D. The hard problem of consciousness. The Blackwell companion to consciousness, 2007, pp. 225–235.
• Klinger E. Thought flow: Properties and mechanisms underlying shifts in content. Singer, 1999, pp. 29–50.
• Robinson MD, Watkins E, Harmon-Jones E (Eds.). Handbook of cognition and emotion. The Guilford Press, pp. 3-16. (2013).
• Adolphs R, Anderson DJ. The Neuroscience of Emotion: A New Synthesis. Princeton University Press, 2018.
• Davidson RJ, Fox AS, Lapate RC, Shackman AJ. Nature of Emotion: Fundamental Questions. Oxford University Press, 2018.
• Adolphs R, Mlodinow L, Barrett LF. What is an emotion?. Current biology, 2019, CB, 29(20), R1060–R1064.
• Yadav SP, Zaidi S, Mishra A, Yadav V. Survey on machine learning in speech emotion recognition and vision systems using a recurrent neural network (RNN). Archives of Computational Methods in Engineering, 2022, 29(3), 1753-1770.
• Aggarwal A, Srivastava A, Agarwal A, Chahal N, Singh D, Alnuaim A, Lee HN. Two-way feature extraction for speech emotion recognition using deep learning. Sensors, 2022, 22(6), 2378.
• Hansen L, Zhang YP, Wolf D, Sechidis K, Ladegaard N, Fusaroli R. A generalizable speech emotion recognition model reveals depression and remission. Acta Psychiatrica Scandinavica, 2022, 145(2), 186-199.
• Wu J, Zhang Y, Sun S, Li Q, Zhao X. Generalized zero-shot emotion recognition from body gestures. Applied Intelligence, 2022, 52(8), 8616-8634.
• Uyanık H., Ozcelik STA, Duranay ZB, Sengur A, Acharya UR. Use of Differential Entropy for Automated Emotion Recognition in a Virtual Reality Environment with EEG Signals. Diagnostics, 2022, 12(10), 2508.
• Chowdary MK, Anitha J, Hemanth DJ. Emotion Recognition from EEG Signals Using Recurrent Neural Networks. Electronics, 2022, 11(15), 2387.
• Zuo X, Zhang C, Hämäläinen T, Gao H, Fu Y, Cong F. Cross-Subject Emotion Recognition Using Fused Entropy Features of EEG. Entropy, 2022, 24(9), 1281.
• Şengür D, Siuly S. Efficient approach for EEG-based emotion recognition. Electronics Letters, 2020 December, vol. 56, no. 25, pp. 1361-1364.
• Kyranides MN, Petridou M, Gokani HA, Hill S, Fanti KA. Reading and reacting to faces, the effect of facial mimicry in improving facial emotion recognition in individuals with antisocial behavior and psychopathic traits. Current Psychology, 1-14, 2022.
• Massaccesi C, Korb S, Willeit M, Quednow BB, Silani G. Effects of the mu-opioid receptor agonist morphine on facial mimicry and emotion recognition. Psychoneuroendocrinology, 2022, 105801.
• Khattak A, Asghar MZ, Ali M, Batool U. An efficient deep learning technique for facial emotion recognition. Multimedia Tools and Applications, 2022, 81(2), 1649-1683.
• Xiao H, Li W, Zeng G, Wu Y, Xue J, Zhang J, Guo, G. On-Road Driver Emotion Recognition Using Facial Expression. Applied Sciences, 2022, 12(2), 807.
• Menting-Henry S, Hidalgo-Lopez E, Aichhorn M, Kronbichler M, Kerschbaum H, Pletzer B. Oral contraceptives modulate the relationship between resting brain activity, amygdala connectivity and emotion recognition–a resting state fMRI study. Frontiers in Behavioral Neuroscience, 2022, 16.
• Fauvé P, Tyvaert L, Husson C, Hologne E, Gao X, Maillard L, Hingray C. Functional MRI-based study of emotional experience in patients with psychogenic non-epileptic seizures: Protocol for an observational case-control study–EMOCRISES study. Plos one, 2022, 17(1), e0262216.
• Gao L, Cai Y, Wang H, Wang G, Zhang Q, Yan X. Probing prefrontal cortex hemodynamic alterations during facial emotion recognition for major depression disorder through functional near-infrared spectroscopy. Journal of Neural Engineering, 2019, 16(2), 026026.
• Nishi R, Fukumoto T, Asakawa A. Possible effect of natural light on emotion recognition and the prefrontal cortex: A scoping review of near-infrared (NIR) spectroscopy. Advances in Clinical and Experimental Medicine: Official Organ Wroclaw Medical University. 2023 April, PMID: 37093092.
• Koelstra S ve diğerleri. DEAP: A Database for Emotion Analysis using Physiological Signals, IEEE Transactions on Affective Computing. 2012, vol. 3, no. 1, pp. 18-31.
• Zheng WL, Lu BL. Investigating Critical Frequency Bands and Channels for EEG-based Emotion Recognition with Deep Neural Networks, IEEE Transactions on Autonomous Mental Development (IEEE TAMD). 2015, 7(3): 162-175.
• Ismael AM, Alçin ÖF, Abdalla KH, Şengür A. Two-stepped majority voting for efficient EEG-based emotion classification. Brain Informatics, 2020, 7(1), 1-12.
• Ari B, Siddique K, Alçin ÖF, Aslan M, Şengür A, Mehmood RM. Wavelet ELM-AE Based Data Augmentation and Deep Learning for Efficient Emotion Recognition Using EEG Recordings. IEEE Access, 2022, vol. 10, pp. 72171-72181.
• Alakus TB, Gonen M, Turkoglu I. Database for an emotion recognition system based on EEG signals and various computer games–GAMEEMO. Biomedical Signal Processing and Control, 2020, 60, 101951.
• Joshi VM, Ghongade RB. EEG based emotion detection using fourth order spectral moment and deep learning. Biomedical Signal Processing and Control, 2021, 68, 102755.
• Marín-Morales J, Higuera-Trujillo JL, Greco A. ve diğerleri. Affective computing in virtual reality: emotion recognition from brain and heartbeat dynamics using wearable sensors. Scientific Reports, 8(1), 2018,13657.
• Suhaimi NS, Mountstephens J, Teo J. A Dataset for Emotion Recognition Using Virtual Reality and EEG (DER-VREEG): Emotional State Classification Using Low-Cost Wearable VR-EEG Headsets. Big Data and Cognitive Computing, 2022; 6(1):16.
• Chen DW, Miao R, Yang WQ, Liang Y, Chen H, Huang L, Han N. A feature extraction method based on differential entropy and linear discriminant analysis for emotion recognition. Sensors, 2019, 19(7), 1631.
• Li D, Xie L, Chai B, Wang Z. A feature‐based on potential and differential entropy information for electroencephalogram emotion recognition. Electronics Letters, 2022, 58(4), 174-177.
• Joshi VM, Ghongade RB. Optimal number of electrode selection for EEG based emotion recognition using linear formulation of differential entropy. Biomedical and Pharmacology Journal, 2020, 13(2), 645-653.
• Li Y, Wong CM, Zheng Y, Wan F, Mak PU, Pun SH, Vai MI. EEG-based emotion recognition under convolutional neural network with differential entropy feature maps. In 2019 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications, 2019 June (CIVEMSA), pp. 1-5.
• Zheng WL, Zhu JY, Lu BL. Identifying Stable Patterns over Time for Emotion Recognition from EEG. IEEE Transactions on Affective Computing. 2017, 1–1.
• Yu M. ve diğerleri. EEG-based emotion recognition in an immersive virtual reality environment: From local activity to brain network features. Biomedical Signal Processing and Control, 72 (2022): 103349.
• Delorme A, Makeig S. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 2004, 134(1):9–21.
• İnternet kaynağı, By Hugo Gambo - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=473246. Erişim Tarihi: 02.01.2023.
• Sorkhabi MM. "Emotion Detection from EEG signals with Continuous Wavelet Analyzing." American Journal of Computing Research Repository 2.4 (2014): 66-70.
• Shannon CE. A mathematical theory of communication. The Bell System Technical Journal, 27:379–423, 623–656, 1948.
• Michalowicz JV, Nichols JM, Bucholtz F. Handbook of Differential Entropy (1st ed.). Chapman and Hall/CRC, 2013.
• Lachaux JP ve diğerleri. Measuring phase synchrony in brain signals. Human brain mapping, 1999, pp.194–208.
• Bruña R, Maestú F, Pereda E. Phase locking value revisited: teaching new tricks to an old dog. Journal of neural engineering, 2018, 15(5), 056011.
• O'Shea K, Ryan N. An introduction to convolutional neural networks. arXiv preprint, 2015, arXiv:1511.08458.
• Kiranyaz S, Avci O, Abdeljaber O, Ince T, Gabbouj M, Inman DJ. 1D convolutional neural networks and applications: A survey. Mechanical systems and signal processing, 151, 2021, p. 107398.
• Yang K, Tong L, Shu J, Zhuang N, Yan B, Zeng Y. High gamma band EEG closely related to emotion: evidence from functional network. Frontiers in human neuroscience, 2020, 14, 89.
• Yang Y, Gao Q, Song Y, Song X, Mao Z, Liu J. Investigating of Deaf Emotion Cognition Pattern By EEG and Facial Expression Combination. IEEE Journal of Biomedical and Health Informatics, Feb. 2022, vol. 26, no. 2, pp. 589-599.
• Wang XW, Nie D, Lu BL. Emotional state classification from EEG data using machine learning approach, Neurocomputing, Volume 129, 2014, Pages 94-106.
• Zheng WL, Zhu JY, Peng Y, Lu BL. EEG-based emotion classification using deep belief networks. 2014 IEEE International Conference on Multimedia and Expo (ICME), 2014, pp. 1-6.
• Zhang J, Chen M, Hu S, Cao Y, Kozma R. PNN for EEG-based Emotion Recognition. 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2016, pp. 2319-2323.