Emotion and Drowsiness Detection System using Multimodal Fusion and Explainable AI

Abstract

This paper illustrates the creation of a holistic Real- Time Driver Emotion and Drowsiness Detection System that is supposed to enhance road safety through detection of a driver emotional state and the level of alertness. The application integrates the process of analysis of facial expressions, speech emotion recognition, and drowsiness detection to deliver real-time interventions and make drivers safe and aware of driving in the streets. Based on multimodal AI methods, the system both reads visual and audio information, as well as reads facial expression and speech to identify emotional states and monitors faces to recognize the signs of drowsiness. The system puts more focus on the detection of drowsiness as opposed to emotion recognition to facilitate the provision of safety alerts in time. Also, the Explainable AI (XAI) integration brings transparency in the sense that the users can see how the system makes decisions. This project emphasizes the significance of multimodal fusion, the fact is that both results of facial and speech analysis are used simultaneously to provide better and more objective results, therefore, enhancing the overall safety of drivers. The proposed system can be deployed to be in use in the real-world context with little disruption to the user experience giving the feedback in real-time and ensures safety is monitored throughout driving.