An approach to emotion recognition in single-channel EEG signals: a mother child interaction

In this work, we perform a first approach to emotion recognition from EEG single channel signals extracted in four (4) mother-child dyads experiment in developmental psychology. Single channel EEG signals are analyzed and processed using several window sizes by performing a statistical analysis over features in the time and frequency domains. Finally, a neural network obtained an average accuracy rate of 99% of classification in two emotional states such as happiness and sadness.


Introduction
To provide an exact definition of emotions is difficult to establish [1], [2], but for convenience, we consider emotions as a neuronal response to external stimuli. The study of these stimuli is very important, because it could bring help to build a psychological profile of a person. This profile will can be usefull for the diagnosis of possible psychopathology of the individual, or will make suitable a prevention strategies with proper treatment, evaluating their behavior into a social or specific situations in the enviroment.
Different approaches have been reported for the determination of an emotional state, such as facial expression recognition, speech analysis, and interpretation of corporal gestures [3] [4][5] [6]. A different approach to the detection of emotions is through the analysis of electroencephalographic signals, making them the extent of the interaction between neurons to perform any action or work, in our case, "creating emotions"; the importance of emotion detection using these signals is that they are directly related to the brain activity and could be mapped to several areas of brain cortex. This will allows to determine or validate phisiologically the emotional state. Also, it is not necessary to perform a phonetic activity or a gestural expression, which is quite useful for the determination of lies or detecting emotions in infants.
This study seeks to determine when the emotional states of happiness, and sadness appears in a mother-child dyad with the analysis of some descriptive characteristics in time and frequency domain, after conducting an experiment to evoke emotions, using a single channel from a EEG signal, located in the x position of the standard 10-20 configuration.
The purpose of this work is to validate the psychological results adressed in [7][8] [9] by a triangulation procedure of experts. Also, extrapolate some methodological results previosly obtained in emotion recognition from audio signals and finally to establish a procedure for lack of information in EEG signals in order to find the relevant features of the signals to proceed with a multi channel procedure.
It is necessary to give a definition of what is a dyad, Kenny define the dyad as "the fundamental unit of interpersonal interaction and interpersonal relations" [10]. That is, a group of two persons in this case mother and child. The importance of this dyad relays in the high impact of the mother's behavior in the social and emotional child development [11].
The importance of the study of these dyadic interactions focuses on the fact that can help to create a complete scheme on mental development of infants. In this case the mother-child dyad was analyzed for the evaluation of the interaction mother-child into a developmental psychological study. The study evaluated the capabilty of the emotional induction over a child from the mother without verbal communication. Also, detecting emotions evoked in the mother and its consequence on the evocation of emotions in her child [7][8] [9]. Therefore, this research seeks for a classification system to estimate the emotional state of sadness and state of happiness in this dyad and validate the previous results, also develop an analysis method for EEG single channel signals.
The raw EEG signal usually has an amplitude of the order of µV and contains frequency components of up to 300Hz [12], in several studies report the optimal ranges for the detection of emotions in EEG signals in different frequency bands of the EEG signal. This bands are classified as: Delta band (0.5 to 4 Hz), Theta band (4 to 8 Hz), Alpha band (8 to 13 Hz), Beta band (13 to 30 Hz), and Gamma band (30 to 70 Hz) [12], [13]. Detecting emotions from EEG signals has been a study field in recent years [14] [22].
Pentrantonakis used the Delta, Theta, Alpha and Beta band with a HOC analysis for determinated six emotions: Happiness, Surprise, Anger, Fear, Disgust, and Sadness with a minimal rate of classification for Fear with 75% and a maximum rate for Happiness with 99.373% [14]; Lee used Theta, Alpha, Beta and Gamma band for demostrated different patterns for differents emotional state, with different combination of this bands, performing a complete analysis of correlation, coherence, and phase synchronization index of each band in different emotional states, [15]; Varun executes a more conservative analysis as to the frequency range, taking all the frequency bands previously exposed in a range of 0.5 Hz to 100 Hz, implementing a classification between happiness, neutral, sadness and fear based on multiwavelet transform, with a mean acuracy of 80% [16]; Murugappan proposes the implementation of a Surface Laplacian filter to remove noises and artifacts and used an extraction method based on multi-resolution analysis of Wavelet Transform, to determinated between Disgust, Happiness, Surprise, Fear and Neutral with different classifier as LDA and KNN, comparing the number of channels of EEG from 62 to 8, with a mean acuracy of 85% [17]; and Li proposed the classification of two emotions, sadness and happiness using Common Spatial Patterns on the Gamma Band with a mean acuracy of 93.5%. [18] This paper relates a study carried out on a small cohort of mother child "dyads" in order to detect a binary emotion of either happiness or sadness. The work revolves around using some standard statistical measures applied to a single channel EEG signal obtained from the subjects adult and child alike. Different window sizes were used to perform the statistical analyses and these were then used a features in a MLP neural network to classify outcomes into either happiness or sadness. The classifier gave a 99% success rate. In this work we propose an approach to a methodology for emotion recognition in EEG signals from one channel information. Also, presents an easily and understandable solution with preprocessing of the signal, windowing and time-frequency classical features extration and finally a neural network classifier [23] [24].
No specific EEG frequency bands were taken into account and for simplicity, statistical measurements were performed to extract relevant information from features. Signals are not only sampled from adults but also from children and emotional states were confirmed by human experts. This work is organized as follows: Section 2 presents the methodology, preprocessing, features extraction and the classifier proposed. Section 3 presents the results obtained and finally the conclusions and future work will be adressed.

Methodology
The proposed methodology, looks for the extraction of simple features from the single channel signal. The first approach to the analysis of the information relays on the classical features extracted from EEG signals as descriptors of some physical phenomenon such as energy and frequency components. More sophisticated features will be considered in future work. The relevance of the proposed methodology is based on the use of time and frequency variables for feature extraction (typically used on voice features [25]; instead of the traditional features based on frequency bands used on EEG treatment. Also, the choice of these features related to the physical phenomena allows to perform a classification task with a high rate of accuracy. The classifiers were trained separating the complete set of data in three subsets for training (60%), test (20%) and a final set for validation (20%). Several MLP architectures were tested used at least the features provided by different sets obtained from three windows size on sampling stage as detailed in section II.A.This section presents the basic scheme for filtering and addresses some descriptive features in a simple way.

Pre-processing
The signal filtering process was implemented with three IIR filters: a high pass filter of 10 th order with response Butterworth at 0.5Hz, a low pass filter of 10 th order with response Butterworth at 70Hz, and a notch filter of 10 th order with response Butterworth at 60Hz. To perform the analysis of the signal, a windowing process is implemented with a Hamming window type, which satisfies the following equation: Where L is the length of the window. An overlap of 50% was done, and three window sizes: 100ms, 40ms and 20ms; each windows sizes were essayed , in order to extract features as detalied below.

Descriptive Characteristics
For training a classifier system, it was decided to find some descriptive characteristics of the EEG signal in a temporal and frequency approach. Given the random behavior of the signal [12], some type of statistical analysis must be performed [26]. The employed features were: Where N is the length of the samples, in this case the length of the window, and x i the input signal.

RMS
The root mean square of the signal:

Mean
The mean of the signal:

ZC
The number of zero crossing of the signal: Where Y is:

MDF
The median frequency of the signal: Where P is the Power Spectrum.

MNF
The mean frequency of the signal: Where P is the Power Spectrum, and f is the frequency.
2.2.10. WL The wavelength of the signal: 2.2.11. WL2 The wavelength of the standardized signal:

Classifier
Several training options such us the training method the activation functions and neural networks architecture were evaluated. The final classifier is a well known multi layer perceptron (MLP) with an architecture of 11 inputs, 3 hidden layers, each one with 11, 11 and 10 neurons and a single output. Activation function were tansigmoidal and purelinear in output layer. Since the noisy behavior of the MLP's output, it is low pass filtered and a binarized. The low-pass filter is a moving average filter, given by the following equation: Where x f (i) is the filtered signal x(i), as this filtered signal can take different values of 0 and 1, and our classifier has only two possible states: sadness and happiness, we use the binarizared function that is show in following expression:

Experimental Protocol
The study was conducted with 8 subjects, 4 women (mothers), and 4 children (3 males and 1 female) with a mean age of 22 months, each mother complete a form of Informed consent, and each mother was informed about the purpose of this experiment. The experiment was approved by the local Ethical Commitee. The stimulus of happiness for the mother consisted into hear the happiest story of her own life and the stimulus of sadness for the mother consisted into hear the saddest story of her own life, each story was told in a previously session with a professional in charge of the experiment. The stimulus for the child in every emotional state was evoked by the presence of his mother face to face.
To perform the experiment the following protocol was followed: First, each mother was asked to make a recording in a room, the happiest moment of her life for the stimulus of happiness, and the saddest moment of her life to the stimulus sadness, then every dyad was placed face to face; the mother had headphones and listen in each case (happy or sad) the story that her previously recorded, causing the feeling of happiness or sadness, as the case, creating an evocation of emotions on her child who was staring at her. Each of these moments was videotaped, the  measures of the EEG signals were taken from single channel to minimize the level of invasiveness of the experiment and it was taken at a sampling rate of 1000 Hz and It was counted with a team of psychologists who verified the validity of this protocol that meets the ethical principles of Helsinksi [7][8] [9].

Results and Discussion
The emotion detection of sadness and happiness was performed with 3 different sizes of windows. In the Table 1 we present the averages of each of the extracted features, for happiness and sadness, with difference between the mother or child for differents length of windows.
It may be noted that each characteristic is affected by the window size, increasing its magnitude proportionally with the size of the window. It can be seen that each characteristic is  affected by the window size, but these changes are acceptably small, where it can been said after averaging all results, that could be used bigger window sizes for purposes of computational costs.
With the matrix resulting from the extraction of descriptive characteristics we proceed to train a multilayer perceptron for each characteristics individually and one for the set of characteristics, which gives us a total of 12 MLP, the same procedure was performed with 3 different sizes of window. After having each MLP trained, was calculated the percentage of correct classification, the results are in Table 2.
It may be noted that each feature individually does not provide enough information for recognition of Happiness state or Sadness state, in contrast can be seen that the percentage of classification of all these characteristics together on average has a percentage of 99.7% correct classification. Which allows infer, that the size of the window may slightly affect the success rate of the classifier, but reduce computational costs, window sizes could be around 100ms to be affected by 0.4%, which can be disregarded in this case. This could be remedied by performing an analysis of emotional states in the previous and next segment, seeking to avoid irregularities and misclassification.
Future works related to single channel emotion detection will be focused on multiresolution analysis with Dabeuchies wavelet family to features extraction to improve accuracy and provide more phenomenon related characteristics.

Conclusions
The detection between the happiness and sadness emotional state, is possible through an analysis of a single channel of EEG. The analysis of the signal using temporal and spectral features, allows the classification of the emotional states with a high rate with a nonlinear classsifer as the MLP.
Moreover, the effect of window size on the results was also evaluated. Given the high classification rates, it can be concluded that the size of the analysis window is important but stable. Allowing the selection of larger window sizes to reduce computational costs, and trying to reduce the small effect of misclassification of the signal, assesing the segments before and after the emotional states obtained by the classifier.
This analysis can be used to study the mental mappings during the mother-child dyads interactions. As future work, we will study the detection of a greater number of emotions, and different number of channels.