Integrated Cnn Based Facial Emotion Detection And Camera Based PPG Heart Rate Monitoring

Erwin Panggabean; R. Mahdelena Simanjorang; Wira Apriani; Nuraisana Nuraisana; Hartati Palentina Sipahutar; Tesalonika Pesta Siagian

doi:10.47709/cnahpc.v7i3.6299

Authors

Erwin Panggabean STMIKPelita Nusantara Medan, Sumatera Utara, Indonesia
R. Mahdelena Simanjorang STMIK Pelita Nusantara Medan, Sumatera Utara, Indonesia
Wira Apriani STMIKPelita Nusantara Medan, Sumatera Utara, Indonesia
Nuraisana STMIKPelita Nusantara Medan, Sumatera Utara, Indonesia
Hartati Palentina Sipahutar STMIK Pelita Nusantara Medan, Sumatera Utara, Indonesia
Tesalonika Pesta Siagian STMIK Pelita Nusantara Medan, Sumatera Utara, Indonesia

DOI:

https://doi.org/10.47709/cnahpc.v7i3.6299

Keywords:

Emotion Detection, Heart Rate Estimation, CNN, Deep Learning, Photoplethysmography, PPG, Facial Image Processing

Abstract

Human emotion detection and heart rate estimation are two important aspects in developing a more responsive and adaptive human-computer interaction system. This study proposes a real-time video-based system that is able to detect facial emotions and estimate the user's heart rate simultaneously. The Convolutional Neural Network (CNN) method is used to classify facial expressions into several emotion categories such as happy, sad, angry, afraid, and neutral. Meanwhile, heart rate estimation is carried out using a non-contact Photoplethysmography (PPG) approach, which utilizes variations in color intensity in the user's facial area from video recordings to calculate the pulse rate. This system is developed using a standard webcam camera without additional medical devices, allowing for practical and economical implementation. The test results show that the system is able to recognize facial expressions with good accuracy, and estimate heart rate with an average error rate that is still within the tolerance limit of non-medical applications. By integrating computer vision technology and biometric signals, this study contributes to the development of a passive, real-time, and easily accessible emotion and health monitoring system.

Downloads

Download data is not yet available.

References

Li, Y., Deng, J., Zhao, B., et al. (2020). Deep emotion classification with CNN using facial images. IEEE Access, 8, 173109–173118.

Chen, W., Liu, H., & Xu, Z. (2020). Remote heart rate measurement using PPG signals from face videos. Sensors, 20(22), 6570.

Yu, H., He, F., & Zhang, C. (2021). rPPG-based non-contact pulse measurement via deep learning and video analysis. Biomedical Signal Processing and Control, 70, 102965.

Liu, X., Qiao, Y., & Jin, M. (2023). A multimodal approach for emotion recognition using facial features and pulse signal. Expert Systems with Applications, 218, 119527.

Cheng, Chun Hong, Kwan Long Wong, Jing Wei Chin, Tsz Tai Chan, and Richard H. Y. So. 2021. “Deep Learning Methods for Remote Heart Rate Measurement: A Review and Future Research Agenda.” Sensors 21(18):1–32. doi: 10.3390/s21186296.

Edy, M. R. 2024. “Deteksi Emosi Dari Ekspresi Wajah Dengan Deep Learning.” Information Technology Education Journal 3(2):116–23.

Ekawati, Inna, Fadilla Nidya, Riyanto Putra, Malikus Sumadyo, and Retno Nugroho. 2025. “Deteksi Emosi Menggunakan Convolutional Neural Network Berdasarkan Ekspresi Wajah.” 5(1):73–82.

Evan Tanuwijaya Timotius, David Christian Kartamihardja, and Timotius Leonardo Lianoto. 2021. “Deteksi Ekspresi Wajah Manusia Menggunakan Convolution Neural Network Pada Citra Pembelajaran Daring.” Jurnal Ilmiah Betrik 13(3):224–30.

Fadilla, Muhammad Andika, Herri Setiawan, and Mustafa Ramadhan. 2023. “Implementasi Metode Convolutional Neural Network (Cnn) Pada Sistem Deteksi Emosi Dari Ekspresi Wajah Manusia Dengan Aplikasi Android Sebagai Antarmuka Pengguna.” Jurnal Ilmu Komputer Dan Sistem Informasi (JIKSI) 9(4):126–38.

Mulyani, Sri, Hilman Zarory, Aulia Ullah, and Jufrizel Jufrizel. 2024. “Prototype PPG (Photoplethysmography) Secara Real-Time Sebagai Pendeteksi Dini Gangguan Detak Jantung Dilengkapi Dengan Visual Graph Pada Android.” Journal of Telecommunication Electronics and Control Engineering (JTECE) 6(2):125–38. doi: 10.20895/jtece.v6i2.1460.

Pirzada et al. (2023) discuss remote photoplethysmography (rPPG) technology in depth, highlighting its basic principles, technical challenges, and opportunities for its use in modern biomedical engineering. The article provides a solid foundation for the development of non-contact systems for vital signs monitoring.

Zhao, Sun, and Zhang (2021) demonstrate the effectiveness of using a lightweight Convolutional Neural Network (CNN) to recognize facial emotional expressions in real time. This approach focuses on computational efficiency without compromising detection accuracy.

Zhang, Wang, and Li (2022) explore the fusion of emotion and heart rate data obtained from webcams using a combination of CNN and rPPG methods. The results show that the integration of both parameters can improve the quality of interpretation in camera-based emotion recognition systems.