Videolardan Kalp Atış Hızı Kestirimi Üzerine Bir İnceleme

Year 2022, Issue: 44, 65 - 73, 31.12.2022

Mustafa Korkmaz , Saffet Vatansever

https://doi.org/10.31590/ejosat.1221945

Abstract

Kalp atış hızı; kişinin sağlığı, aktivite seviyesi, stres durumu, zindeliği ve benzeri fizyolojik durumları hakkında önemli ipuçları vermektedir. Kalp atış hızı, elektrokardiyogram (EKG) ve nabız oksimetreleriyle ölçülebilir olmakla birlikte, bu cihazlar sürekli temas gerektirdiğinden zamanla rahatsız edici olabilmektedir. Bilgisayarlı görü (computer vision) alanındaki son gelişmeler, bir kişiye elektrot veya nabız oksimetreleri takmanın mümkün veya uygun olmadığı durumlarda, videolardan kişinin kalp atış hızını tespit etmeye olanak sağlamıştır. Uzaktan fotopletismografi (rPPG), bir video kamera aracılığıyla derideki hassas renk değişikliklerini yakalayarak, yaşamsal belirtilerin tespit edilmesine imkân sağlayan bir teknolojidir. Son yıllarda yapılan çalışmalar, uzaktan kalp atış hızı tespiti için en uygun bölgenin yüz olduğunu göstermiştir. Bu çalışmada; videolar aracılığıyla kişilerin yüz bölgesinden kalp atışı hızı kestiriminin nasıl yapılabildiği, kalp atışı hızı kestirimi sürecindeki aşamaların nasıl iyileştirilebileceği ve nasıl daha yüksek doğrulukta kalp atışı hızı tespiti yapılabileceği hakkında literatürdeki mevcut yöntemler incelenerek kapsamlı bir analiz yapılmıştır.

Keywords

Kalp atış hızı, uzaktan fotopletismografi (rPPG), yüz videoları, derin öğrenme.

Heart Rate Estimation from Videos: A Review

Abstract

Heartbeat rate provides important signs about a person's health, activity level, stress level, vitality, and other related physiological conditions. Although electrocardiograms (ECG) and pulse oximeters can be exploited to measure the heart rate, these instruments might become uncomfortable with time due to the requirement for constant contact. Recent advances in computer vision have made it achievable to detect a person's heart rate from videos when it is impossible or impractical to attach electrodes or pulse oximeters to a person. Remote photoplethysmography (rPPG) is a technology that detects vital physiological information by capturing considerably precise color changes in the skin via a video camera. Recent studies have shown that the face is the most appropriate area to use for remote heart rate detection. In this study, a comprehensive analysis has been conducted by investigating the existing methods in the literature on how to estimate the heartbeat rate from a person's face through videos, how to improve the stages in the heart rate estimation process, and hence to improve the accuracy of the estimations.

Keywords

Heartbeat rate, remote photoplethysmography (rPPG), face videos, deep learning

References

• Aydemir, B. (2019). Egzersiz esnasında toplanan fotopletismografi sinyallerinden kalp atım hızı ve ventilasyon hızı değişkenlerinin ayrıştırılması [MARMARA ÜNİVERSİTESİ]. https://acikbilim.yok.gov.tr/handle/20.500.12812/282457
• Bian, M. (2019). Pattern Recognition and Computer Vision. In Fundamentals of Uncertainty Calculi with Applications to Fuzzy Inference. https://doi.org/10.1007/978-94-015-8449-4_9
• Bousefsaf, F., Maaoui, C., & Pruski, A. (2013). Continuous wavelet filtering on webcam photoplethysmographic signals to remotely assess the instantaneous heart rate. Biomedical Signal Processing and Control, 8(6), 568–574. https://doi.org/10.1016/j.bspc.2013.05.010
• Bush, I. (2016). Measuring Heart Rate from Video. https://web.stanford.edu/class/cs231a/prev_projects_2016/finalReport.pdf
• Chen, W., & McDuff, D. (2018). DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11206 LNCS, 356–373. https://doi.org/10.1007/978-3-030-01216-8_22
• Cheng, C. H., Wong, K. L., Chin, J. W., Chan, T. T., & So, R. H. Y. (2021). Deep learning methods for remote heart rate measurement: A review and future research agenda. In Sensors (Vol. 21, Issue 18). MDPI. https://doi.org/10.3390/s21186296
• De Haan, G., & Jeanne, V. (2013). Robust pulse rate from chrominance-based rPPG. IEEE Transactions on Biomedical Engineering, 60(10), 2878–2886. https://doi.org/10.1109/TBME.2013.2266196
• Deng, Y., & Kumar, A. (2020). Standoff heart rate estimation from video – a review. 6. https://doi.org/10.1117/12.2560683
• Djeldjli, D., Bousefsaf, F., Maaoui, C., & Bereksi-Reguig, F. (2019). Imaging Photoplethysmography: Signal Waveform Analysis. Proceedings of the 2019 10th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS 2019, 2, 830–834. https://doi.org/10.1109/IDAACS.2019.8924239
• Elgendi, M. (2012). On the Analysis of Fingertip Photoplethysmogram Signals. Current Cardiology Reviews, 8(1), 14–25. https://doi.org/10.2174/157340312801215782
• ElMaghraby, A., Abdalla, M., Enany, O., & Y. El Nahas, M. (2014). Detect and Analyze Face Parts Information using Viola- Jones and Geometric Approaches. International Journal of Computer Applications, 101(3), 23–28. https://doi.org/10.5120/17667-8494
• Hassan, M. A., Malik, A. S., Fofi, D., Saad, N., Karasfi, B., Ali, Y. S., & Meriaudeau, F. (2017). Heart rate estimation using facial video: A review. In Biomedical Signal Processing and Control (Vol. 38, pp. 346–360). Elsevier Ltd. https://doi.org/10.1016/j.bspc.2017.07.004
• He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-Decem, 770–778. https://doi.org/10.1109/CVPR.2016.90
• Huang, B., Lin, C.-L., Chen, W., Juang, C.-F., & Wu, X. (2021). A novel one-stage framework for visual pulse rate estimation using deep neural networks. Biomedical Signal Processing and Control, 66(June 2020), 102387. https://doi.org/10.1016/j.bspc.2020.102387
• Irani, R., Nasrollahi, K., & Moeslund, T. B. (2014). Improved pulse detection from head motions using DCT. VISAPP 2014 - Proceedings of the 9th International Conference on Computer Vision Theory and Applications, 3, 118–124. https://doi.org/10.5220/0004669001180124
• Kazemi, V., & Sullivan, J. (2014). One millisecond face alignment with an ensemble of regression trees. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1867–1874. https://doi.org/10.1109/CVPR.2014.241
• Kwon, S., Kim, J., Lee, D., & Park, K. (2015). ROI analysis for remote photoplethysmography on facial video. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, 2015-Novem, 4938–4941. https://doi.org/10.1109/EMBC.2015.7319499
• Li, X., Chen, J., Zhao, G., & Pietikäinen, M. (2014). Remote heart rate measurement from face videos under realistic situations. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 4264–4271. https://doi.org/10.1109/CVPR.2014.543
• McDuff, D., Gontarek, S., & Picard, R. W. (2014a). Improvements in remote cardiopulmonary measurement using a five band digital camera. IEEE Transactions on Biomedical Engineering, 61(10), 2593–2601. https://doi.org/10.1109/TBME.2014.2323695
• McDuff, D., Gontarek, S., & Picard, R. W. (2014b). Remote detection of photoplethysmographic systolic and diastolic peaks using a digital camera. IEEE Transactions on Biomedical Engineering, 61(12), 2948–2954. https://doi.org/10.1109/TBME.2014.2340991
• Meinzer, H. P., Deserno, T. M., Handels, H., & Tolxdorff, T. (2013). ROI Selection for Remote Photoplethysmography. Informatik Aktuell, 99–103. https://doi.org/10.1007/978-3-642-36480-8
• Niu, X., Han, H., Shan, S., & Chen, X. (2018). SynRhythm: Learning a Deep Heart Rate Estimator from General to Specific. Proceedings - International Conference on Pattern Recognition, 2018-Augus(i), 3580–3585. https://doi.org/10.1109/ICPR.2018.8546321
• Niu, X., Shan, S., Han, H., & Chen, X. (2020). RhythmNet: End-to-End Heart Rate Estimation from Face via Spatial-Temporal Representation. IEEE Transactions on Image Processing, 29, 2409–2423. https://doi.org/10.1109/TIP.2019.2947204
• Pagano, T. P., Santos, V. R., Bonfim, Y. da S., Paranhos, J. V. D., Ortega, L. L., Sá, P. H. M., Nascimento, L. F. S., Winkler, I., & Nascimento, E. G. S. (2022). Machine Learning Models and Videos of Facial Regions for Estimating Heart Rate: A Review on Patents, Datasets, and Literature. Electronics (Switzerland), 11(9). https://doi.org/10.3390/electronics11091473
• Poh, M., Mcduff, D. J., & Picard, R. W. (2010). Noncontact automated cardiac pulse measurements using video imaging and blind.pdf. Medical Optics and Biotechnology, 18(10), 795–805.
• Poh, M. Z., McDuff, D. J., & Picard, R. W. (2011). Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Transactions on Biomedical Engineering, 58(1), 7–11. https://doi.org/10.1109/TBME.2010.2086456
• Premkumar, S., & Hemanth, D. J. (2022). Intelligent Remote Photoplethysmography-Based Methods for Heart Rate Estimation from Face Videos: A Survey. In Informatics (Vol. 9, Issue 3). MDPI. https://doi.org/10.3390/informatics9030057
• Rautaray, S. S., & Agrawal, A. (2012). R Eal T Ime H and G Esture R Ecognition. 3(1), 21–31.
• Rouast, P. V., Adam, M. T. P., Chiong, R., Cornforth, D., & Lux, E. (2018). Remote heart rate measurement using low-cost RGB face video: a technical literature review. In Frontiers of Computer Science (Vol. 12, Issue 5, pp. 858–872). Higher Education Press. https://doi.org/10.1007/s11704-016-6243-6
• Sabokrou, M., Pourreza, M., Li, X., Fathy, M., & Zhao, G. (2021). Deep-HR: Fast heart rate estimation from face video under realistic conditions. Expert Systems with Applications, 186. https://doi.org/10.1016/j.eswa.2021.115596
• Shao, D., Liu, C., & Tsow, F. (2021). Noncontact Physiological Measurement Using a Camera: A Technical Review and Future Directions. ACS Sensors, 6(2), 321–334. https://doi.org/10.1021/acssensors.0c02042
• Sinhal, R., Singh, K., & Raghuwanshi, M. M. (2020). An Overview of Remote Photoplethysmography Methods for Vital Sign Monitoring. Advances in Intelligent Systems and Computing, 992, 21–31. https://doi.org/10.1007/978-981-13-8798-2_3
• Song, R., Chen, H., Cheng, J., Li, C., Liu, Y., & Chen, X. (2021). PulseGAN: Learning to Generate Realistic Pulse Waveforms in Remote Photoplethysmography. IEEE Journal of Biomedical and Health Informatics, 25(5), 1373–1384. https://doi.org/10.1109/JBHI.2021.3051176
• Spetlik, R., Franc, V., Cech, J., & Matas, J. (2018). Visual heart rate estimation with convolutional neural network. British Machine Vision Conference 2018, BMVC 2018, 1–12.
• Sun, Y., & Thakor, N. (2016). Photoplethysmography Revisited: From Contact to Noncontact, from Point to Imaging. IEEE Transactions on Biomedical Engineering, 63(3), 463–477. https://doi.org/10.1109/TBME.2015.2476337
• Swinehart, D. F. (1962). The Beer-Lambert law. Journal of Chemical Education, 39(7), 333–335. https://doi.org/10.1021/ed039p333
• Tamura, T., Maeda, Y., Sekine, M., & Yoshida, M. (2014). Wearable photoplethysmographic sensors—past and present. Electronics , 3(2), 282–302. https://doi.org/10.3390/electronics3020282
• Tang, C., Lu, J., & Liu, J. (2018). Non-contact heart rate monitoring by combining convolutional neural network skin detection and remote photoplethysmography via a low-cost camera. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2018-June, 1390–1396. https://doi.org/10.1109/CVPRW.2018.00178
• Tsou, Y. Y., Lee, Y. A., Hsu, C. T., & Chang, S. H. (2020). Siamese-rPPG network: Remote photoplethysmography signal estimation from face videos. Proceedings of the ACM Symposium on Applied Computing, 2066–2073. https://doi.org/10.1145/3341105.3373905
• Verkruysse, W., Svaasand, L. O., & Nelson, J. S. (2008). Remote plethysmographic imaging using ambient light. Optics Express, 16(26), 21434. https://doi.org/10.1364/oe.16.021434
• Wang, W. (2017). Robust And Automatic Remote Photoplethysmography (Vol. 1, Issue 2017). https://pure.tue.nl/ws/files/78340965/20171023_Wang.pdf%0Ahttps://research.tue.nl/en/publications/robust-and-automatic-remote- photoplethysmography%0Ahttps://pure.tue.nl/ws/portalfiles/portal/78340965/20171023_Wang.pdf
• Wedekind, D., Trumpp, A., Gaetjen, F., Rasche, S., Matschke, K., Malberg, H., & Zaunseder, S. (2017). Assessment of blind source separation techniques for video-based cardiac pulse extraction. Journal of Biomedical Optics, 22(3), 035002. https://doi.org/10.1117/1.jbo.22.3.035002
• YAMAN, A. U. (2018). Yüz tanıma sistemlerinin yanıltılmasına karşı bir yöntem: yüz videolarında nabız tespiti ile canlılık doğrulaması [Ankara Üniversitesi]. https://tez.yok.gov.tr/UlusalTezMerkezi/tezDetay.jsp?id=UwWffTuiMVwjS85blanc6Q&no=LlzkDJAuYHVHoM9tysjGIA
• Yang, W., Li, X., & Zhang, B. (2018). Heart Rate Estimation from Facial Videos Based on Convolutional Neural Network. Proceedings of 2018 6th IEEE International Conference on Network Infrastructure and Digital Content, IC-NIDC 2018, 45–49. https://doi.org/10.1109/ICNIDC.2018.8525602
• Yu, Z., Li, X., & Zhao, G. (2020). Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks. 30th British Machine Vision Conference 2019, BMVC 2019.
• Yu, Z., Peng, W., Li, X., Hong, X., & Zhao, G. (2019). Remote heart rate measurement from highly compressed facial videos: An end-to-end deep learning solution with video enhancement. Proceedings of the IEEE International Conference on Computer Vision, 2019-Octob, 151–160. https://doi.org/10.1109/ICCV.2019.00024
• Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016). Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks. IEEE Signal Processing Letters, 23(10), 1499–1503. https://doi.org/10.1109/LSP.2016.2603342