The Emotional Expression Response of a Patient Based on their Facial Expression
—Focus on Music Stimuli ()
1. Introduction
Transition from impaired consciousness to a vegetative state creates a complex atmosphere where family caregivers have conflicting feelings, including a sense of emotional burden of care and expectations for recovery. Family visits may decrease as patients pass the acute stage and their overall condition becomes chronic, probably owing to the lack of a significant change or response from these patients. Therefore, it may be crucial for caregivers to feel that the patient is recognizing and responding to external stimuli.
We focused on facial expression analysis as an objective and quantitative method for evaluating the response of patients with unresponsive wakefulness syndrome (UWS) to external stimuli. Ribeiro et al. [1] observed changes in vital signs (blood pressure, heart rate, respiratory rate, and oxygen saturation) and facial expression after music stimuli in 26 patients with severe brain injury. However, in that study, most changes in facial expressions were subjectively evaluated. Facial analysis software has been developed to allow automatic and quantitative assessment of facial expression. FaceReader® (Noldus, Wageningen, Netherlands; version 6) reads movements of the facial muscles, eyebrows, and mouth angles; subsequently, it integrates them to objectively evaluate the emotional response. A previous study reported 90% consistency between facial expression classification through FaceReader® and manual facial expression classification based on the Facial Action Coding System (FACS) [2] [3] [4]. In particular, emotions can be evaluated more accurately by using a facial analysis software than by visual evaluation performed by humans, without resorting to the facial expression [5]. Diagnostic imaging of individuals with disorders of consciousness (DOC) caused by severe brain injury has revealed that auditory stimuli yield considerable activity in the brain’s auditory area [6]. A recent study noted that listening to one’s preferred music is particularly stimulating and enhances the hearing-recognition-related response of patients with disorders of consciousness [7]. Therefore, daily-life auditory stimuli are considered suitable and minimally invasive approaches of yielding static facial images necessary for facial analysis.
This study focused on “emotion” as a change in the valence value (comfort-discomfort index, which is described later) based on the classification proposed by Banham-Bridges [8]. In healthy individuals, changes in facial expression can be attributed to changes in emotions underlying conscious or unconscious responses to external stimuli (facial muscle reactions). However, it remains challenging to clearly distinguish between conscious and unconscious responses to external stimuli through facial expression changes in patients with UWS. Consequently, we defined emotional expression as a change in the valence value from the baseline based on valence values integrating changes from each facial expression muscle. Moreover, we defined facial expression motion as the degree of change in facial muscle movement, such as the action unit integral value, which is unrelated to emotional expression.
This study aimed to quantitatively measure facial expression changes in response to auditory stimuli in patients with UWS and evaluate the presence or absence of emotional responses. We hypothesized that auditory stimulation would yield higher valence integral, action unit integral, average heart rate, and data loss values (due to difficulty in capturing facial expressions because of active movement during auditory stimulation). These hypotheses were based on the study by de Jong et al. [9] where a mother-narrated story was presented on tape to a patient in a persistent vegetative state. Concomitantly, the regional cerebral blood flow was measured using positron emission tomography. The activation of the rostral anterior cingulate, right middle temporal, and right premotor cortices is indicative of appropriate cortical involvement in processing emotional attributes of sound or speech. Gutiérrez et al. [10] used heart rate variability to assess the responses to auditory stimulation with emotional content in patients in persistent vegetative or minimally conscious states; they found a changing pattern induced by auditory stimulation in three patients (decreased heart rate, increased heart rate variability, and decreased/increased power in the low/high frequencies, respectively), consistent with increased cardiovagal stimulation, and auditory stimuli could increase body movement.
2. Methods
Ethical considerations
This study was approved by the ethics committee of the participating facility. Consent to participate in the research was provided by the families of the patients; however, an explanation of the research was given at the bedside where feasible. The date and time of filming were determined with consideration of the patient’s condition and in consultation with the ward manager to ensure no interference with treatment. The family could withdraw from the study at any time, even after starting the filming, when the patient’s state became unstable, the family thought that the patient was being burdened, or the family members themselves felt burdened. When a patient’s eyes were closed for a prolonged period or their body movement was extreme during filming, it would be stopped and rescheduled where possible.
Participants
This study enrolled patients diagnosed with DOC due to brain injury with permission for participation from their doctor. Other inclusion criteria included functional hearing capacity as well as no history of arrhythmias or heart disease. We excluded patients with prolonged eye closure, those undergoing oral or nasal ventilation, or those with pre-injury hearing abnormalities. Data were collected between April 2017 and January 2018 in a private room or separate space at the hospital or in the patient’s home in Osaka, Japan.
Study intervention
Severe brain injury impairs the auditory brainstem’s response to sounds from the inner ear’s cochlear nerve to the midbrain; notably, the response remains until brain death, regardless of whether a person is asleep or awake [11]. Auditory stimulation was chosen based on the findings of our first research phase, where auditory stimulation was identified as a good stimulation method with minimal invasiveness.
There were two patterns of auditory stimulation used, music and voice stimuli, which are frequently used as interventions for restoring consciousness in patients with UWS. The stimuli were selected according to the circumstances and preferences of the family. Subsequently, patients’ facial expression changes were examined. Headphones were used to block other sounds and promote active listening. Regarding voice stimuli by the family, the primary caregiver who was closest to the patient recorded a 5-min recording of a letter and played it for the patient using headphones. For music stimuli, the primary caregiver played music familiar to the patient for 5 min using headphones. Based on the amount of time the patient could be supine, filming was performed for 5 min before and during auditory stimulation, with the session lasting for 10 min; this was repeated thrice a week. The volume was adjusted to 50 - 60 dB or less based on environmental standards. The timing for conducting the procedure was chosen in consultation with the head of the ward or the family. The procedure was performed during waking hours.
Study procedure
Head nurses at the hospital or home-visit nursing station, as well as doctors responsible for outpatient procedures, were asked to identify home-based and hospitalized patients based on the study’s objectives. Subsequently, the study was explained to the participants and proxies at the paticipant’s bed side. After consent was obtained, the date and time of the procedure were set. We collected the patients’ information, including data on age, sex, diagnosis, time since diagnosis, current medical records, and medical history. Moreover, the level of consciousness was evaluated based on the Tohoku Ryogo Center’s Prolonged Disorders of Consciousness Scale (Kohnan score) (Table 1).
Heart rate data were collected and analyzed using remote photoplethysmography, which is a non-invasive technique included in the analysis technology of FaceReader® that measures small changes in color under the skin associated with changes in blood volume and oxygen saturation in blood vessels resulting from heartbeats. The consistency of the patient’s daily electrocardiogram monitoring data (cardiogram or pulse oximeter) was checked through visual examinations.
Regarding the facial expression analysis, feature points extracted on a
![]()
Table 1. Tohoku Ryogo center’s prolonged disorders of consciousness scale*.
*Kohnan Score. Categorization of patients according to the Persistent Consciousness Disorder Scale Score. Worst Total score is 70 points. Best score is 0 points. 70 - 65 points: Extreme cases (Completely Vegetative Symptoms), 64 - 55 points: Severe cases (Complete), 54 - 40 points: moderate cases (Incomplete), 39 - 25 points: Mild (Transitory Symptoms), below 24 points: Slight (Recovering cases)
frame-by-frame basis for videos were analyzed using FaceReader®, which reads and quantifies human expressions and can record time-series changes in emotions with the FACS. The FACS was developed to objectively identify and classify human facial expression and emotional changes based on the movement of facial expression muscles [12]. It quantitatively describes the “smallest action unit of anatomically independent and visually identifiable facial expression motion.” Subsequently, it analyzes the action unit integral values after classifying them into 44 different action units based on set conversion rules. FaceReader® extracted FACS-specified facial feature points from still images or videos of the patients’ faces before filming and identified 491 facial movement points (opening and closing of the mouth and both eyes, as well as raising and lowering of both eyebrows). Among the 44 action units, five levels of changes in the representative action unit could be identified. The FACS can define the action units involved in the formation of facial expressions, including happiness, anger, sadness, and pleasure, with each emotion being represented by a specific action-unit combination (Table 2).
For example, happiness can be expressed as action unit 6 + 7 + 12 or action unit 12. Additionally, 3D facial expression modeling based on these action-unit combinations can be used to quantify the intensity of arousal and basic emotions (neutral, happy, sad, angry, surprise, scared, and disgust) on a scale of −1 to 1 through comparisons with the established expression database of 10,000 people from different countries.
Valence indicates pleasure as positive and discomfort as negative. Basic emotions calculated through action unit combinations are all summed as negative, except for surprise and happiness, which are considered positive. Patients’ facial expressions were recorded at the patients’ home or hospital room using a video camera (HC-W570M; Panasonic, Kadoma, Japan) fixed to clearly show the patient’s face under direct light. During recording, the patient was wakeful in the supine position without any stimulation. The patients’ safety and provision of adequate care were emphasized, and filming was stopped when they were compromised. The total
![]()
Table 2. Each emotion expressed using a specific AU combination.
Ekman & Friesen, 1978. AU, action unit.
recording duration from preparation to completion was <1 h.
Regarding the patient’s environmental factors, any major changes in temperature, humidity, sound, and luminosity on the day or during filming were recorded as stimuli using a luminance meter (SK-10LX; Sato Keiryoki MFG. Co., Ltd., Tokyo, Japan), a sound level meter (Mother Tool TM-102 Inc.), and a combined thermometer and hygrometer (SK-110TRH II Series Type 1, Sato Keiryoki MFG. Co., Ltd) installed at the patient’s bedside.
Data analysis
Every 1/15-s video footage (18,000 frames per session) was converted to a valence value ranging between −1 and 1. Moreover, action-unit integral values were determined on a five-point scale.
The time from the start of measurement to 5 min after measurement was represented on the x-axis, while valence and action-unit values were plotted on the y-axis for each timepoint. The line graphs were plotted at 5-min intervals to calculate the integral value of the area from the baseline. Missing values were calculated using the following formula:
Corrected integral value = actual integral value × 9000/(9000 − number of missing values).
Values obtained during the 5-min duration before and during the auditory stimulation were classified into before-stimulation and with-stimulation groups, respectively. Between-group comparisons were made for 1) valence integral values to verify the degree of emotional expression, 2) action-unit integral values to verify the movement of the muscles on the face and their degree of movement, 3) average heart rate values to verify the degree of response using a physiological indicator, and 4) missing values resulting from body movement or full-face movement (behavioral response) to verify the incidence of behavioral responses to auditory stimulation.
Regarding valence and action-unit integral values, only sessions with missing values of <10% of the 9000 missed frames were analyzed. Variables underwent normality testing using the Shapiro-Wilk test with subsequent analysis using the Wilcoxon signed-rank rest. All statistical analyses were performed using SPSS Statistics (IBM, Armok, NY).
3. Results
Patient characteristics are summarized in Table 3. This study included three men and three women with DOC due to brain injury. Two participants were hospital inpatients (less than 1 year from onset), while the other four resided at home. One patient’s family requested both voice and music stimuli (Patient 1). Six patients underwent 21 sessions. The average age of the patients was 45.2 ± 20.1 years (range: 21 - 76 years). Two patients were filmed at home and four at the hospital. Auditory stimuli were provided thrice to each patient. Ten sessions with missing values of <10% of the total 9000 frames are indicated using an asterisk. In each session, there was no change in the Kohnan score.
Assessments of reaction to auditory stimulation
Expression of emotional response (Table 4)
Data plotting and integration values were analyzed for ten effective sessions and significant between-group differences were observed in the degree of emotional expression (Figure 1). There was no significant between-group difference in the valence integral values (before-auditory stimulation: 36.4 ± 16.1; with-auditory stimulation: 52.8 ± 44.2; p = 0.49). Areas representing valence integral values per session are shown in gray in Figure 2. Auditory stimulation increased emotional responses in 5 (Patient-Session: 1-1, 1-3, 6-1, 6-2, and 6-3) of the 10 sessions (Patient-Session: 1-1, 1-3, 3-1, 3-2, 3-3, 4-3, 5-2, 6-1, 6-2, and 6-3). In the with-auditory stimulation group, Patients 3 and 6 showed decreased and increased emotional responses in all three sessions, respectively. The five sessions with increased valence integral levels were Patient-Session 1-1, 1-3, 6-1, 6-2, and 6-3. Patient 1 was a 42-year-old woman who had been diagnosed with brain injury 24 years earlier and lived at the home where she underwent in-home filming of voice stimulation sessions. Patient 6 was a 21-year-old man who usually lived at home but underwent filming of music stimulation sessions at a respite inpatient care center. He had been diagnosed 5 years previously with DOC caused by acute subdural hematoma after a traffic accident. The families of both patients reported reactions such as crying-like noises, producing a sound similar to a voice calling out, and staring at faces. However, in other times, they did not completely respond during excretion and treatment or when they were thought to be uncomfortable. Furthermore, among the five sessions with reduced valence integral values (Patient-Session: 3-1, 3-2, 3-3, 4-3, and 5-2), Patient 3 often responded with a voice calling out a name; however, movement and response usually stopped when the television was turned on. Although Patient 4 showed a change in facial expression depicting pain in response to barely painful stimuli, the patient appeared to be almost entirely unresponsive. Patient 5 exhibited a
![]()
Table 3. Patient characteristics and stimulation methods.
Patients were categorized according to Persistent Consciousness Disorder Scale Score (worst, 70 points; best 0 points). 70 - 65 points: extreme (completely vegetative symptoms), 64 - 55 points: severe (complete), 54 - 40 points: moderate (incomplete), 39 - 25 points: mild (transitory symptoms), <24 points: slight (recovering patients). *Indicates a session where the number of missing values was less than 10% of all 9000 frames.
![]()
Table 4. Areas representing the valence integral values (n = 18,000).
![]()
Figure 1. Integration values before and during auditory stimulation.
reaction to voice stimulus involving staring in the direction of the voice without movement. Patients with decreased emotional expression responses showed a static response, rather than a dynamic response, to stimuli. Additionally, in these patients, facial expression analysis could not determine emotion expression or perception.
Regarding the mean action-unit integral values, there was a significant between-group difference for action unit 10 only (p = 0.03). (Table 5) Although emotion assessment could not be conducted using action-unit 10 only, the lip-raising movement of facial muscles was less in the with-auditory stimulation group than in the before-auditory stimulation group. A visual image shows one of the nasolabial folds disappearing. The missing action-unit values for the five sessions that showed increased emotional expression due to auditory stimulation were <10%; moreover, there were action-unit 1, 4, and 7 responses, which correspond to upper-face movements (Figure 3). In the five sessions that showed an emotional expression response, in sessions 1, 2, and 3, Patient 6 showed significant variations in facial expression behavior in action units 1, 4, and 7, which
![]()
Figure 2. Areas representing valence integral values per sessions.
![]()
Table 5. Comparison of AU values before and during auditory stimulation across ten sessions.
AU, action unit.
![]()
Figure 3. Action units for sessions where an emotional response to the music stimulation increased. Examination of action unit integral values in the sessions with an increased emotional expression response to music stimulation indicated responses of action units 1, 4, and 7, all of which correspond to the movement of the upper face.
![]()
Figure 4. Physiological reactions. Patients whose average heart rate increased or decreased in all three sessions are surrounded by a dotted line. Effective cases are marked with an asterisk.
correspond to eyelid and upper-face movements.
Physiological reactions
The average heart rates in the before-auditory and with-auditory stimulation groups were 70.9 ± 6.9 and 69.4 ± 6.8 bpm, respectively, with no significant between-group difference (p = 0.33). As shown in Figure 4, there were fluctuating heart rate changes over time. The valence integral values and heart rate variations varied across the sessions. Moreover, Patients 1, 5, and 6 showed similar variation patterns for the average heart rate in both groups for all three sessions, with all sessions involving music stimuli.
4. Discussion
Real-time detection of responses to external stimuli and DOC fragmentation benefits both patients and their caregivers in case of a decline in quality of life resulting from impeded identification of a patient’s negative response to an intervention, as well as the inability of the family and caregivers to determine when the patient is feeling happy or comfortable.
This study conducted a facial expression analysis of patients with UWS at the bedside. There was no significant post-stimulation change in the valence integral levels; moreover, among the ten valid sessions, valence integral values for auditory stimulation increased in half the sessions and did not increase in the other half. This observed heterogeneity in patients responding to stimuli is consistent with that reported in previous studies.
Only action unit 10 showed a significant between-group difference among the average action-unit integral values in ten sessions. Although this was insufficient for evaluating emotion, the interpreted visual finding indicated greater disappearance of the nasolabial folds in the with-stimulation group than in the before-stimulation group. This finding suggests that the patient was responding to the stimulus.
There was no significant between-group difference in the average heart rate or valance integral values. However, two patients experienced a decrease in the average heart rate in all three sessions involving music stimuli, while one patient experienced an increase in the average heart rate in all sessions. This finding suggests that the three patients who underwent music stimulation experienced similar changes in each session. In patients with UWS, music stimulation caused unique changes in the cerebral blood flow of each person. This could be associated with the reproducibility of heart rate changes (similar heart rate change for each stimulus).
Patient 6 exhibited an increase in valence integral values, as well as a lower average heart rate, during auditory stimulation in all three sessions. The DOC in Patient 6 was caused by a subdural hematoma due to a traffic accident. Patient 6 was the youngest (21 years old) and the only one in their twenties. Emotional responses may be easier to elicit in younger patients.
Our findings show the characteristic emotional reactions that the families of each patient noticed on several occasions, consistent with the findings of previous research. Our findings show, subjectively and objectively, that patients with unresponsive arousal syndrome have a unique response. Specifically, this was due to gaining important insights from a nursing perspective. Our findings show that not only the subjective primary caregiver but also others may be able to confirm the emotional expression of the patients, which is a new finding.
5. Conclusion
In conclusion, we observed significant between-group differences in the average heart rate values and valence integral values, which represent pleasant or unpleasant emotions. Valence integral values increased in half of the sessions, which suggest that some patients showed an emotional expression response to auditory stimulation. Furthermore, AU integral values revealed upper-face, eyebrow, and eyelid movement during auditory stimulation, which could be used for discriminating between emotional expressions in clinical settings. Moreover, regarding the response of patients with UWS to external stimuli, music stimuli were considered more suitable than voice stimuli since the average heart rate fluctuations were consistent for nine music stimulation sessions (three sessions per case) and differed for the 12 voice stimulation sessions. Future clinical applications are anticipated after validation of these findings using other modalities.
6. Limitations
This study has several limitations. First, the sample size (n = 6) was small owing to altering physical conditions, as well as difficulty in gaining cooperation from recently diagnosed or home-based patients. Further multicenter studies in different cities and environments are needed to confirm the present findings. Second, data were collected based on a bedside survey using headphones. Therefore, although auditory stimuli were controlled to some degree, the possible influence of non-auditory stimuli could not be excluded. Third, this study did not employ a control group, and the results were only partially verified based on those of a previous study on healthy individuals. Despite these limitations, our results are significant given the potential impact of studies that advance behavioral assessments for patients with DOC, which may have wide implications in nursing and caregiving settings.
Support
This study was supported by the Japam Spciety for the Promotion of Science KAKENHI Grant in-Aid-for Young Scientists (B) [Grant number 16K20781, 2016].