Wearable facial electromyography: in the face of new opportunities

The complexity and importance of the facial muscles is readily apparent. Human facial muscles serve a variety of functions, including verbal communication and mastication. Additionally, facial muscle movements play a crucial role in conveying our emotions and internal mental state [1, 2]. Owing to their important role in so many domains, even partial loss of facial muscle control may result with adverse effects on well-being. Vastly different medical conditions are associated with abnormal facial activation and the field has been the focus of attention in many different domains, ranging from plastic surgery [3, 4], facial rehabilitation [5], psychology [6, 7] and neurology [8].

As far back as the days of Douchenne [9] and Darwin [10], the study of facial muscle activation has fascinated and challenged the scientific community. To this day, fundamental questions regarding their functions and underlying mechanisms remain open and debated [1, 2, 11]. Studies in recent years are employing various computer vision approaches [12, 13], incorporating 3D imaging [14] and deep learning techniques [15] to accurately detect muscle activation and even identify emotion. These methods are non-contact and unobtrusive, but can lack in accuracy [16, 17]. Facial electromyography (fEMG) remains one of the most attractive methods in providing direct information about muscle activation, rather than providing indirect information on facial features [18]. Surface fEMG in particular provides minimal inconvenience to the subject, and can be applied in non-laboratory settings [19].

Traditional fEMG technologies have been widely used for decades in the investigations of muscle activation, and demonstrated the necessity of utilizing electromyography (EMG) measurements for studying complex behaviors such as emotional expressions [6, 20]. It has been applied to demonstrate a quantitative link between smiles and positive affect (suggesting its use in marketing [21]), to categorize emotions [22] and to differentiate between genuine and fake smiles [23], to name only a few examples. However, owing to rapid improvement in computer vision in recent years, fEMG was, for the most part, replaced by computer based visual analysis of the face [16]. Benefiting from great improvements in imaging technologies and new algorithms, most recently in the application of neural networks [24, 25], image processing approaches have become widely used and dominate the realm of facial expression analysis [12]. However, despite their simplicity and convenience, these methods lack the anatomical specificity needed in analyzing facial muscles [2]. Indeed, by using high (temporal) resolution fEMG that enables the detection of subtle modifications in facial expressions, we recently demonstrated clear discrepancies between visual analysis and fEMG [17]. These discrepancies limit the validity of video based investigations.

High-resolution EMG is a more reliable tool for analysis of facial muscle activation, especially in medical applications where muscle specificity is of importance [17]. An important additional benefit of fEMG is the ability to record data in more flexible settings, abrogating the need to have subjects in direct visual contact with a high-resolution camera (for example). This grants the opportunity to monitor subjects when they are behaving freely and naturally [19], performing very subtle facial actions [26] or even when interacting with others.

The straightforward approach to achieving high-resolution spatial fEMG is through the use of multiple electrodes. The electrodes, usually in a gel form (such as Ambu^® BlueSensor, Spes Medica or Cardinal Health™), are carefully placed on the surface of the face to achieve close proximity to a desired muscle [27]. Most commercial electrodes are compatible with pre-existing EMG recording systems, such as Biovision or Medelec Synergy VIASYS Healthcare. This approach is however very limited. Electrode placement is lengthy and subjects are extremely restricted in their motion, having their face covered with electrodes and wires. Furthermore, in the conventional use of wet electrodes, electrodes tend to dry out, which results in decrease of the signal-to-noise ratio along the recording, and may cause skin irritation and discomfort to the subject. For the highest accuracy, needle electrodes (e.g. Ambu^® Neuroline™ Concentric) were previously used. However, this method is invasive, requires special expertise and can be applied only in laboratory settings.

In an alternative approach, which we recently developed and tested, dry multi-electrode arrays enable both high temporal and spatial resolution, while minimizing discomfort and preparation time for the subject. The electrodes are printed on soft films and are used on the surface of the face to achieve wireless recordings. In figure 1 we show an example of such an array, with 16 channels (although in principle more are possible), that allows wireless real-time recording at 250 samples per second or 4000 samples per second for offline analysis. These fEMG multi-electrode arrays accommodate the contour of the face utilizing soft electronics technologies. When the electrode substrate is soft enough, the electrodes conformally attach to the skin [28] and establish good electrical contact. Dry electrodes facilitate stability over hours of recordings. More importantly, the usage of multiple electrodes enable the application of signal analysis tools, such as Independent Component Analyses (ICA), which make it possible to derive muscle source separation. Source separation is of great importance since the signal of a certain electrode does not necessarily indicate muscle activity originating from the muscle beneath the electrode, but rather the overall activity. By detecting independent patterns in the data, the ICA algorithm separates the recorded signals into their underlying origins, or components. Each source, or component, is the weighted summation of all electrodes in the array, which enables to map the components onto the electrode array [29]. In high-resolution fEMG, the ICA reliably separates the signals into their underlying muscle activation patterns and enable to project those to the face (see figure 1 below; for more details on fEMG-ICA analysis see [17, 19, 28]). Therefore, with this approach, the exact placement of the electrodes is not critically important in order to map the activation patterns to facial locations. In several recent papers, we demonstrated that with soft electrodes, fEMG can be readily achieved in natural environments and specific muscle sources can be identified [17, 19, 28]. Additionally, EMG has high temporal resolution that can capture subtle modifications in muscle activation. These performances overcome many of the challenges associated with conventional fEMG and open new opportunities in the investigation of facial muscles. We discuss below several of these opportunities.

Zoom In Zoom Out Reset image size

There is a surprisingly long list of ongoing debates and open questions in the field of facial muscle activation. In fact, some of the most basic notions are still debated, along with the emergence of new inquiries. How can facial expressions be mapped to emotions? What is the evolutionary role of emotions? Are facial expressions universal and innate? What defines genuine and fake smiles? Can facial muscle activation regulate our emotional state, instead of being affected by it? What is the role of yawning? EMG was applied in recent decades to address many of these issues. In the following section, we review how these theoretical challenges can be met with the use of advanced EMG technology.

Two of the most fundamental inquiries in the field of facial expressions relate to the extent to which facial muscle activation expresses emotional states [2], and whether those are culturally universal. Originated in Darwin's theory of the innate origins of facial expressions [10], the widely held and almost intuitive notion has long been that a discrete (small or large) set of facial expressions can be identified and used to extract different emotional states [1]. Proponents of the facial emotional signatures view ground their hypotheses mainly on experiments using recognition tasks that replicate specific expressions [30, 31], while their opponents point to inconsistencies in empirical data and methodological confounds [2, 32]. Thus far, most of the evidence supporting the emotional signatures view were based on studies that used visual inspection for classification of emotions. EMG measurements of facial activity, on the other hand, can serve as an objective tool in this heated discussion, as they reveal which muscles are activated (either from raw data or via ICA sources). Indeed, EMG has long been used to extract signatures of emotional states [6]. For instance, the zygomaticus major muscle, which pulls the corners of the mouth, was associated with positive emotions, while the corrugator supercilii muscle, which draws the brows, enhances negative emotional experience [33, 34].

Furthermore, whereas fEMG investigations were mostly carried out in artificial lab settings, new wearable EMG technologies enable investigation of emotional expressions in freely behaving humans, whether in a lab and or in a more natural setting. This may reveal patterns of authentic expressions that deviate from the common signatures, as evidenced by potential differences in real and fake smiles [17]. It would therefore be of great interest to revisit the early studies that suggested discrete facial signatures of emotions [1, 35, 36]. Moreover, carrying out experiments in natural settings opens new horizons in investigating facial expressions in clinical populations that have difficulties with conventional experimental setups, such as people diagnosed with Autism.

Additionally, wearable EMG should be of special interest for studying emotional expressions under social interactions conditions. Social communication experiments, when authentic and not tightly guided, reveal many interesting behavioral patterns [37, 38]. In social context, facial expressions do not necessarily reflect true emotional states, but play a key role in affecting the interactions by expressing intentions to others [39]. High-resolution wearable EMG is most suitable to reveal such complex facial patterns. Indeed, combining the EMG signals with support vector machine classification and unsupervised peak-density clustering, we recently showed that deceptive behavior and expressions of subjects that face each other, i.e. involved in direct interaction instead of in an artificial laboratory setup, is different than in traditional setups [40].

Owing to their ability to distinguish between positive and negative affect, fEMG was explored as a tool in marketing. Specifically, to objectively differentiate between positive and negative responses to different products [41, 42]. While low-resolution EMG can be used to differentiate between positive and negative affect, utilizing the specificity of the zygomaticus major and the corrugator supercilii muscles respectively, high-resolution EMG may provide insight to more subtle categories [43], such as those used in questionnaires applied in market research. Beyond answers such as like/dislike, questionnaires may ask subjects to report on feelings such as: Happiness, romantic, disgusted, irritated, relaxed, nostalgic and energetic, following exposure to different stimuli [44]. Recalling also the manner by which each individual interprets specific emotional category, high-resolution EMG may help distinguish between subtle differences between individuals. Facial markers are most likely subject specific, as evident in many investigations [2] and these will have to be mapped for each subject individually.

Facial EMG was also used to investigate facial muscle activation during mental and physical tasks, an almost intuitive phenomenon that has not yet been explained by neurology or psychology [45–47]. In particular, studies commonly report an increase in facial muscle activity with the increase in the effort of the task, especially in the 'frowning muscle' (corrugator supercilii muscle) [48, 49]. The origin to this increase in seemingly irrelevant facial muscles activation is not yet clear [48]. A possible interpretation, in particular to tasks that combine physical aspects, such as training of legs or arms, states that the increase can be attributed to the motor irradiation, or motor overflow, where muscle activation becomes delocalized i.e. spread in the brain's motor regions [49, 50]. Then, despite the fact that facial muscles have no biomechanical utility for the performed tasks, motor overflow causes the recruitment of muscle activation during considerable effort. Alternatively, another feasible explanation stemming from a social communications perspective, could be to reveal to others the effort exerted by an individual [51]. Facial EMG investigations during physical effort at high-resolution are now possible and may help elucidate some of the open questions in this field (e.g. origin, universality, specificity, gender differences, etc) [52].

Wearable high-resolution fEMG is also an important tool for quantifying facial muscle activity for medical diagnostics, bio-feedback and rehabilitation. In conditions such as facial palsy and synkinesis, specific muscle information is required to quantify symptoms' severity [53, 54]. Furthermore, a promising technique to expedite rehabilitation of motor palsy is EMG biofeedback, in which the patient learns to control movements by receiving real-time feedback on muscle activation [55–57] (in the first case, for example, the muscle synergy was extracted from muscle activity using non-negative matrix factorization). In facial palsy biofeedback, the patient learns to control specific muscle activation ideally without moving the rest of the facial muscles, which requires real-time information in high spatial resolution [58]. Visual analysis, even when utilizing state of the art 3D video technologies, requires large amounts of free storage and heavy computations that are not suitable for real-time feedback. Although the use of fEMG for facial rehabilitation was previously studied, its utilization is still limited [27], mainly because existing technologies restrict natural movements and are not spatially specific and prevent mirroring facial activity, which is highly important in rehabilitation.

While EMG for bio-feedback can improve rehabilitation in the clinic, its true potential is in home-based training, allowing subjects much more frequent access to efficacious sessions [59, 60]. Key technical elements in enabling such home-based training are: (1) technology suitable for self-use; and (2) reliable and automated data analysis in real time. Conventional fEMG systems are too technically demanding to allow such self-use. As wearable fEMG can accommodate both high quality medical data (muscle specificity) and ease of use, it has the potential to open new horizons in facial muscle rehabilitation. Furthermore, with recent advances in digital signal analysis machine learning approaches [61], automated real time data analysis is becoming feasible.

Facial EMG was also considered in many additional domains including as an objective marker for the expression of pain [62], and as a window to the brain in neurological evaluation. For instance, fEMG is used for measuring automatic facial motor mimicry for evaluating emotional dysfunctions [63], as well as oral and facial dysfunctions in speech and language pathologies [64]. Recent advances in EMG technology are expected to increase the usage of EMG for clinical purposes.

As the application of wearable EMG is relatively simplistic and avoids cumbersome equipment and setups, it allows researchers to collect multiple physiological measures with more ease. This is especially beneficial for investigations looking to explore the potential impact one physiological measure has over the other. For example, in the field of cosmetic surgery, they explored the link between the zygomaticus major (smiling) muscle and the facial vein, and reported that the activation of the former resulted in an obstruction of flow in the latter [65]. Recently we reported on a mechanical relationship between muscle activation and blood flow in the forearm and the face [66]. As many physiological mechanisms have unclear origins [67], capturing and understanding these phenomena requires entirely different approaches to allow simultaneous measurements at high sensitivity. Measuring different parameters (e.g. heart rate, blood pressure) is a significant challenge, let alone measuring them together on humans while allowing subjects to function naturally so their regular physiological parameters can be monitored. Such capabilities and investigation approaches will contribute to better understand facial muscle activation.

Facial muscle activation externalizes a vast range of physiological and psychological conditions: They report on physical fatigue, mental effort, emotional state, pain, and even deception. However, large gender and cultural differences exist and have to be accounted for in future studies. Challengingly, the manner by which the information provided by facial muscles can be interpreted and used for clinical and non-clinical applications remains at variance. Surely, with the availability of better technologies, including high-resolution and wireless systems, more thorough and far-reaching investigations will be possible.

Facial EMG should not be considered as a stand-alone tool. Its combination with modern signal analysis and algorithms can enable research to deduce accurate conclusions on the activation of facial muscles. Additionally, due to the fact that facial muscle activation is part of more complex physiological processes, involving the nervous system, blood circulation, and sweating for example, its combination with other physiological measurements has the potential to uncover previously unexplored links between different mechanisms.

To summarize, the range of applications that can benefit from improved mapping of facial muscle activation ranges from psychological investigations, cognitive neuroscience, speechless voice recognition, and sports. In all of these domains, past investigations were restricted to artificial environments, limiting their scope and possibly even their validity. High-density fEMG from freely behaving humans aligns with an important recent trend that aims to study subjects in more realistic and natural conditions than was previously permitted. Subjects may interact with one another, and move freely in the lab or even outside the lab. The investigation of facial muscles under these conditions may help lift some of the ambiguities related to facial expressions and may even help develop better therapeutic approaches.

Many of the ideas presented in this paper build on discussions with present and past students and colleagues including: Paul Funk, Dvir Ben Dov, Liron Amihai, Itay Ketko, Rawan Ibrahim, Lilah Inzelberg, Liraz Gat, Dino Levy, Yaara Yeshurun, Galit Yovel, Mickey Scheinowitz, Anat Mireleman, Miriam Kuntz, Stefan Lautenbacher, Hava Siegelmann and Orlando Guntinas-Lichius.

The paper includes no data.

YH declares a financial interest in X-trodes Ltd which developed the screen-printed electrode technology used in this paper. YH has no other relevant financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.