Computer vision based fatigue detection using facial parameters

Human face is a clear indicator of the fatigue and tiredness experienced by an individual. There may be many cues that can be derived through the analysis of facial parameters which clearly indicate the tiredness. Most of us feel the fatigue and tiredness but at the same time ignore it due to want of time to complete a task or necessity to complete an important work. However there can be instances when this fatigue may turn fatal. Hence an automated system that can easily predict the fatigue becomes the need of the hour. This work is focussed towards developing an automated application that can detect fatigue by analysing various facial parameters. This work uses Computer Vision and has been implemented using Python programming. The result shows good prediction accuracy when it comes to fatigue prediction using facial parameters.


Introduction
There is a common saying Face is the index of mind. True to the saying facial expressions can clearly convey the fatigue experienced by individuals. The facial fatigue can be gauged using various clues such as eyeball movements, eye blinking rate, eyes closed, yawning, head posture etc. Figure 1 shows the various clues in faces that can be used to predict facial fatigue.   [27]It is very important not to ignore fatigue as the after events may sometimes be catastrophic. Some real-time application areas where this automatic facial fatigue detection will be of great use are in detecting drowsiness level of drivers, predicting the mood of audience in a gathering, student attentiveness in class, sleep detection etc. Figure 2 shows some of these application areas of facial fatigue detection. Some of the after effects of facial fatigue are drowsiness, sleep, tiredness, stress, anger, exhaustion etc. Figure 3 lists some of these cascaded effects of facial fatigue. These after effects have the potential to cause serious events such as accidents, deaths, interpersonal discomfort etc. This clearly undermines the imminent need for an automated system to predict facial fatigue in individuals [26].  The objective of this work is to develop a facial fatigue recognition system that can observe the facial parameters listed in Figure 1 and based on some observations predict and alert if the person is experiencing fatigue or not.

Related works
As seen from the above section, it is of utmost importance to identify facial fatigue and notify the same. Several works have been carried out related to fatigue detection in human faces.G. Li et.al. [1] have developed an EEG based system that is embedded into a smartwatch and this clearly analyses the EEG signals and alerts if there is any drowsiness experienced by the individual. M. Sunagawa et al. [2] have developed a model that analyses the facial features of 50 drivers and categorizes the drowsiness into different levels ranging from weak to strong. F. You et.al.
[3] proposed a method that uses individual differentiation aspect to computing whether the person is experiencing drowsiness or not.

Authors of [4]
have developed a drowsiness detection system exclusively for drivers that alerts the drivers using smartphones when drowsiness is detected [24]. A dynamic vision sensor based approach was developed in [5] which analysed the facial features using neuromorphic sensors and predicted the drowsiness experienced [25]. Authors of [6] have performed an exhaustive survey of the various state of the art techniques and models that were developed to address this problem of drowsiness detection. In the work carried out in [7], the authors have used eye aspect ratio as a measure to predict drowsiness in individuals. The authors of [8] have used an optical correlator based approach to determine drowsiness. A wearable sensor based approach was followed in [9] to detect drowsiness in individuals while in [10] an adaptive neural network based approach was followed to predict the drowsiness levels.

Proposed System
The proposed system uses a fusion of Computer Vision concepts, OpenCV and Python programming to accomplish this automated facial fatigue detector. Figure 4 depicts the flow diagram of the proposed approach.  This is followed by computing the Eye Aspect Ratio (EAR) using the method proposed by TerezaSoukupova& Jan ´ Cech [11]. The Eye Aspect Ratio is computed using the following equation (1).
The EAR value is used to determine the drowsiness levels of individuals.

Experimental results
As discussed in the previous section, computer vision based approaches were followed to extract the facial features. The coding was performed using python programming and google colab. OpenCV was used to leverage the computer vision specific functions while performing pre-processing. Other dependencies include scipy, dlib and imutis. In this experiment, the EAR threshold was set as 55% (0.55). When the EAR value at any instant is above 0.55, it is assumed that the individual is active. However, when the EAR value falls below 0.55, it is assumed that the individual is drowsy and the system automatically triggers an alert to make the person aware of his fatigue [28]. Figure 6 shows the eyes wide open while performing the experiment which indicates that the person is active while in figure 7, the eyes are closed for a duration and an automated alert is generated for this scenario. The above experiment was also tested in challenging scenariosuch as a person wearing glasses [29]. This scenario is depicted in figure 8 when the person is awake wearing glasses while figure 9 depicts the scenario where a person's eye is closed while wearing glasses. It can be found that the experiment generated automated alert instantaneously when the person's eyes were closed even while wearing glasses [30].

Conclusion and future work
A simple and automated approach to determine the fatigue of person using facial parameters was developed. This eye specific features were extracted and the EAR was computed. Based on the EAR value, the system determined if the person has fatigue or not. The system was tested with real time videos and it has provided good accuracy and results. Future work will be directed towards developing a robust fatigue classifier that considers other factors such as yawn, facial gestures, body postures etc. along with EAR in order to make the system more efficient. .