FACIAL EXPRESSION RECOGNITION on PARTIAL FACE IMAGES USING DEEP TRANSFER LEARNING

Year 2022, Issue: 049, 118 - 129, 30.06.2022

İsmail Öztel , Gozde Yolcu Öztel , Veysel Harun Şahin

Abstract

Facial expression recognition has a crucial role in communication. Computerized facial expression recognition systems have been developed for many purposes. People's faces can have occlusions because of scarves, facial masks, etc. in cases such as cold weather conditions or Covid-19 pandemic conditions. In this case, facial expression recognition can be challenging for automated systems. This study classifies facial images containing only the eyebrow and eye regions over six expressions with a deep learning-based approach. For this purpose, Radboud Face Database images have been used after cropping the area that includes eye and eyebrow regions. Some popular pre-trained networks have been trained and tested using the transfer learning approach. The Vgg19 pre-trained network achieved 91.33% accuracy over the six universal facial expressions. The experiments show that automated facial expression recognition can be applied with high performance by looking at the region containing eyes and eyebrows

Keywords

Convolutional Neural Networks, Deep Learning, Facial Expression Recognition, Transfer Learning

Supporting Institution

Sakarya University Scientific Research Projects Unit

Project Number

2020-9-33-43

Thanks

This work was supported by the Sakarya University Scientific Research Projects Unit (Project Number: 2020-9-33-43).

References

• [1] Mehrabian, A., (1968), Some referents and measures of nonverbal behavior, Behavior Research Methods & Instrumentation, 1, 203–207.
• [2] Lopes, A.T., de Aguiar E., De Souza A.F., and Oliveira-Santos T., (2016), Facial Expression Recognition with Convolutional Neural Networks: Coping with Few Data and the Training Sample Order, Pattern Recognition, 67, 610-628.
• [3] Zalewski, L. and Gong, S., (2005), 2D Statistical Models of Facial Expressions for Realistic 3D Avatar Animation, IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), 2, 217–222.
• [4] Yolcu, G., Oztel, I., Kazan, S., Oz, C., Palaniappan, K., Lever, T.E. and Bunyak, F., (2017), Deep learning-based facial expression recognition for monitoring neurological disorders, IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 1652–1657.
• [5] Bartlett, M.S., Littlewort, G., Fasel, I., Chenu, J., Kanda, T., Ishiguro, H., Movellan, J.R., (2003), Towards social robots: Automatic evaluation of human-robot interaction by face detection and expression classification, Advances in Neural Information Processing Systems, 16.
• [6] Zhang, Y. and Hua, C., (2015), Driver fatigue recognition based on facial expression analysis using local binary patterns, Optik - International Journal for Light and Electron Optics, 126(23), 4501–4505.
• [7] Shaykha, I., Menkara, A., Nahas, M. and Ghantous, M., (2015), FEER: Non-intrusive facial expression and emotional recognition for driver’s vigilance monitoring, 57th International Symposium ELMAR (ELMAR), 233–237.
• [8] Kim, J.-B., Hwang, Y., Bang, W.-C., Lee, H., Kim, J.D.K. and Kim, C.-Y., (2013), Real-time realistic 3D facial expression cloning for smart TV, IEEE International Conference on Consumer Electronics (ICCE), 240–241.
• [9] Niforatos, E. and Karapanos, E., (2015), EmoSnaps: a mobile application for emotion recall from facial expressions, Personal and Ubiquitous Computing, 19(2), 425–444.
• [10] Terzis, V., Moridis, C.N. and Economides, A.A., (2013), Measuring instant emotions based on facial expressions during computer-based assessment, Personal and Ubiquitous Computing, 17(1), 43–52.
• [11] Ekman P. and Friesen W.V., (1971), Constants across cultures in the face and emotion., Journal of Personality and Social Psychology, 17(2), 124–129.
• [12] Hsu, C.-T., Hsu S.-C. and Huang, C.-L., Facial expression recognition using Hough forest, (2013), 2013 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, 1–9.
• [13] Zhong, L., Liu, Q., Yang, P., Liu, B., Huang, J. and Metaxas, D.N., (2012), Learning active facial patches for expression analysis, 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2562–2569.
• [14] Khan, F., (2020), Facial Expression Recognition using Facial Landmark Detection and Feature Extraction via Neural Networks, arXiv:1812.04510.
• [15] Saurav, S., Saini, R. and Singh, S., (2021), Facial Expression Recognition Using Dynamic Local Ternary Patterns with Kernel Extreme Learning Machine Classifier, IEEE Access, 9, 120844–120868.
• [16] Iqbal, M.T.B., Abdullah-Al-Wadud, M., Ryu, B., Makhmudkhujaev, F. and Chae, O., (2020), Facial Expression Recognition with Neighborhood-Aware Edge Directional Pattern (NEDP), IEEE Transactions on Affective Computing, 11(1), 125–137.
• [17] Dagher, I., Dahdah, E. and Al Shakik, M., (2019), Facial expression recognition using three-stage support vector machines, Visual Computing for Industry, Biomedicine, and Art, 2(1).
• [18] Nair, V., Hinton, GE., (2010) Rectified Linear Units Improve Restricted Boltzmann Machines, Proceedings of the 27th international conference on international conference on machine learning. Omnipress, USA, 807–814
• [19] Ekman, P. and Friesen, W.V., (1978), Facial action coding system: a technique for the measurement of facial movement.
• [20] Morales-Vargas, E., Reyes-García, C. A. and Peregrina-Barreto, H., (2019), On the use of action units and fuzzy explanatory models for facial expression recognition, PLoS One, 14(10).
• [21] Jin, X., and Jin, Z., (2021), MiniExpNet: A small and effective facial expression recognition network based on facial local regions, Neurocomputing, 462, 353–364.
• [22] Fan, X., Jiang, M., Shahid, A.R. and Yan, H., (2022), Hierarchical scale convolutional neural network for facial expression recognition, Cognitive Neurodynamics.
• [23] Saurav, S., Saini, A.K., Saini, R. and Singh, S., (2022), Deep learning inspired intelligent embedded system for haptic rendering of facial emotions to the blind, Neural Computing and Applications, 34, 4595-4623.
• [24] Happy, S.L., Dantcheva, A. and Bremond, F., (2021), Expression recognition with deep features extracted from holistic and part-based models, Image and Vision Computing, 105, 104038.
• [25] Jin, X., Sun, W. and Jin, Z., (2020), A discriminative deep association learning for facial expression recognition, International Journal of Machine Learning and Cybernetics, 11(4), 779–793.
• [26] Happy, S. L., Dantcheva, A. and Bremond, F., (2019), A Weakly Supervised learning technique for classifying facial expressions, Pattern Recognition Letters, 128, 162–168.
• [27] Luh, G.-C., Wu, H.-B., Yong, Y.-T., Lai, Y.-J. and Chen, Y.-H., (2019), Facial Expression Based Emotion Recognition Employing YOLOv3 Deep Neural Networks, 2019 International Conference on Machine Learning and Cybernetics (ICMLC), 1-7.
• [28] Goodfellow, I., Bengio, Y. and Courville, A., 2016, Deep learning, MIT Press.
• [29] Krizhevsky, A., Sutskever, I. and Hinton, G.E., (2007), ImageNet Classification with Deep Convolutional Neural Networks, Handbook of Approximation Algorithms and Metaheuristics, 1–1432.
• [30] Zoph, B., Vasudevan, V., Shlens, J. and Le, Q.V., (2018), Learning Transferable Architectures for Scalable Image Recognition, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8697–8710.
• [31] Zhang, X., Zhou, X., Lin, M. and Sun, J., (2018), ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6848–6856.
• [32] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V. and Rabinovich, A., (2015), Going deeper with convolutions, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1–9.
• [33] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A. and Chen, L.-C., (2018), MobileNetV2: Inverted Residuals and Linear Bottlenecks, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4510–4520.
• [34] He, K., Zhang, X., Ren, S. and Sun, J., (2016), Deep Residual Learning for Image Recognition, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778.
• [35] Huang, G., Liu, Z., van der Maaten, L. and Weinberger, K.Q., (2017), Densely Connected Convolutional Networks, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
• [36] Krizhevsky, A., Sutskever, I. and Hinton, G.E., (2012), ImageNet Classification with Deep Convolutional Neural Networks, Advances in Neural Information Processing Systems, 25.
• [37] Simonyan K. and Zisserman A., (2015), Very Deep Convolutional Networks for Large-Scale Image Recognition, International Conference on Learning Representations.
• [38] Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D. H. J., Hawk, S. T. and van Knippenberg, A., (2010), Presentation and validation of the Radboud Faces Database, Cognition & Emotion, 24(8), 1377–1388.
• [39] Face++ Cognitive Services, (2022), Face++ Web Page, https://www.faceplusplus.com, Access Date: May 23, 2022.
• [40] Oztel, I., Yolcu, G., Oz, C., Kazan, S. and Bunyak, F., (2018), iFER: facial expression recognition using automatically selected geometric eye and eyebrow features, Jorunal of Electronic Imaging, 27(2), 1.
• [41] Nayak, S., Happy, S.L., Routray, A. and Sarma, M., (2019), A Versatile Online System for Person-specific Facial Expression Recognition, TENCON 2019 - 2019 IEEE Region 10 Conference (TENCON), 2513–2518.