The Effect of Facial Expressions on the Process of Learning, and Memory of Face Images among Healthy Participants and Individuals with Traumatic Brain Injury (TBI): Examination Using Eye Movements

Natalie Lugasi¹, Yaron Sachar², Eli Vakil^1*

¹Department of Psychology and Leslie and Susan Gonda (Goldschmied), Multidisciplinary Brain Research Center, Bar-Ilan University, Ramat-Gan, Israel.
²Brain Injury Rehabilitation, Loewenstein Hospital, Raanana, Israel.

DOI: 10.4236/psych.2023.148077                

Abstract

Objective: Context-Dependent Effect (CDE) is a process by which restoring at test the original learning context enhances memory. We examined the CDE of facial expressions through behavioral and eye movement measures. Participants and Method: Twenty-three healthy individuals and 23 patients with moderate-to-severe TBI individuals participated in the current study. Participants were exposed to pictures of faces with neutral expression and were asked to remember them for a subsequent memory test. In the testing session, they were asked to determine whether or not the faces presented to them had appeared before, under two conditions: 1) where the context remains constant (facial expression remained neutral—the Repeat condition); 2) where the context changes (facial expression changed to angry or happy expression— the Re-pair condition). Results: While memory of the individuals with TBI was poorer than that of the control group, both groups exhibited CDE, that is consistency of facial expressions between study and test facilitated face recognition. In both groups, there is a relationship between Dwell Time on the faces and face recognition. This effect appeared in the study and test phases. Conclusions: This research supports previous studies showing evidence for CDE in the TBI group and extends our comprehension of the relationship between eye movements, memory, and context of facial expression.

Keywords

Facial Expression, Memory, Eye Tracking, TBI

1. Introduction

In the literature, there are several distinctions between the different types of memory, including the difference between semantic and episodic memory (Hayes, Ryan, Schnyer, & Nadel, 2004) , explicit versus implicit memory (Schacter, 1987) and the event’s content versus the context in which it occurred, as evidenced by brain imaging studies (Schmitter-Edgecombe & Seelye, 2012) .

The important role of context in memory processes has been recognized for many years. A context-dependent effect (CDE) is said to occur when memory performance is improved by the presence of a contextual stimulus. CDE have been widely documented in the human memory literature in various cognitive tasks (Peterson & Kramer, 2001; Vakil, McDonald, Allen, & Vardi-Shapiro, 2019a) . This effect is considered as an implicit memory measure of context (Vakil, 2005) . Context can consist of perceptual information, spatial location, and temporal order, emotional or social details (Davachi & DuBrow, 2015; Schmitter-Edgecombe & Seelye, 2012; Vakil, Raz, & Levy, 2007) . Brain imaging studies found that memory for content is associated with structures in the medial temporal lobes (MTL), i.e., the hippocampus and surrounding cortices, namely entorhinal, perirhinal, parahippocampal, and temporopolar cortices. In contrast, memory for source (Source memory) is associated with frontal lobes functioning. These results suggest that memory for content and context involved different brain areas (Cabeza, Locantore, & Anderson, 2003; Schmitter-Edgecombe & Seelye, 2012) . In addition, during the learning, the amygdala and orbitofrontal cortex are key structures that interpreted what the stimuli are, including facial expression (Rolls, 2019) , while the hippocampus is more connected to context memory, information on space, action, and “where” events occur, as well as from “what” ventral processing streams (Rolls, 2018) . Other models of declarative memory claimed that the hippocampus has a key role in recollection of complex events, by binding item and context information. The perirhinal cortex is involved in item identification and familiarity of items and has connections with the ventral stream. The parahippocampal cortex is involved in the coding of object location and spatial context and has connections with the dorsal stream. Both cortices project to the entorhinal cortex from which the fibers congregate in the hippocampus (Diana, Yonelinas, & Ranganath, 2007) .

One of the elements that can be examined is the effect of facial expression (i.e., context) on the memory of facial identity (i.e., target) (Vakil, Aviv, Mishael, Schwizer, Ashkenazi, & Sacher, 2019b) . Several studies have examined the brain regions associated with facial perception. In a study using the positron emission tomography (PET) technique, participants were involved in a matching facial emotions task and a control task or face identity. It was found that emotions activated the area of the right anterior cingulate and the bilateral inferior frontal gyri. In addition, there was an increase in activation of the fusiform cortex, regardless of whether the face included emotion (George, Ketter, Gill, Haxby, Ungerleider, Herscovitch, & Post, 1993) . In addition, a study conducted recently examined the functions of the uncinate fasciculus (UF) (in the process of facial expression, using high-angular resolution diffusion-weighted imaging, constrained spherical deconvolution tractography, a control tract (the corticospinal tract, and quantified via fractional anisotropy). Results showed that encoding of facial expression of emotion relates to the right hemisphere UF microstructure (Coad, Postans, Hodgetts, Muhlert, Graham, & Lawrence, 2020) . In addition, a UF subtract selectively predicted face-name learning (Metoki, Alm, Wang, Ngo, & Olson, 2017) . A review of haemodynamic and electrical neuroimaging studies indicated that emotional face perception is a complex process that includes an interactive network with activity distributed in time and space. The study also demonstrated that facial expression and facial identity are mutually affecting processes (Vuilleumier & Pourtois, 2007) .

Facial recognition contains two separate components—facial identity and facial expression. In an article on facial processing, Yankouskaya, Booth, and Humphreys (2012) pointed to three different approaches regarding the relationship between facial identity recognition and recognition of facial expression. The first approach argues that these are two parallel and independent processes (Bruce & Young, 1986) . The second approach argues that recognition of facial identity depends on facial expression, but not vice versa (Haxby, Hoffman, & Gobbini, 2000) . The third approach argues that these two systems interact with each other (Yankouskaya et al., 2012) . fMRI study supported the second approach and found a dissociation between facial identity processing and emotional expression during the coding phase, as the identity was correlated with reduced signals in the fusiform cortex and posterior superior temporal sulcus (STS), while facial expression was correlated with reduced signals in a more anterior region of STS (Winston, Henson, Fine-Goulden, & Dolan, 2004) .

Therefore, different approaches exist as to whether there is or is not a dissociation between facial identity and facial expression (the context) and consequently, whether changes in facial expression between study and test phases will affect the memory of facial identity. In a recent study, participants were presented with facial images and were asked to learn them for a subsequent test. The study included three groups: participants who studied faces with a neutral, angry, or happy expression, while at test, some of the faces were presented with the same expression as in the study, and some of the expressions were changed. It was found that maintaining the same facial expression in the study and the test was associated with better performance than changing the facial expression (Vakil et al., 2019a) . These findings suggest that there is a CDE and support the approach that there is an interaction between facial identity recognition and facial expression recognition, as argued by Yankouskaya et al. (2012) . This study also showed that the CDE was stronger when the facial expression at the study phase was anger, which suggests that this emotion is strongly bound to facial perception.

The ability to detect emotions and facial expressions correctly is a crucial to functioning in society, but certain injuries might cause a deficit in this ability. Traumatic Brain Injury (TBI) leads to several problems, the most common of which are diffused axonal injury, lesions, or degeneration in the frontal and temporal lobes and hippocampal atrophy (Wright, Wong, Obermeit, Woo, Schmitter-Edgecombe, & Fuster, 2014) . Also, TBI can cause bruising, cerebral edema, ischemia, and hemorrhage. MRI often reveals an abnormality in the frontal, mesial-temporal and lateral lobes, enlarged ventricles, and diffuse axonal injury. As a result, people with TBI most often show dysfunction in attention-related tasks, memory, and executive functions, along with motor deficiencies in case motor pathways are impaired (Trojano, Moretta, & Estraneo, 2009; Vakil, 2005) . The deficits in memory function among individuals who have sustained TBI are usually caused by axonal damage in the hippocampus region as well as to the hippocampus itself and associated structures. The latter regions relate to the orbital and ventromedial limbic frontal cortex that are also involved in the memory process (Barbas & Blatt, 1995) .

Studies have shown that people with TBI had difficulty in perceiving and identifying emotions that were displayed in various modes, such as photos (Mancuso, Magnani, Cantagallo, Rossi, Capitani, Galletti, & Robertson, 2015; Bornhofen & McDonald, 2008) , dynamic audiovisual displays, conversational tone alone, moving facial displays, and still photographs (McDonald & Saunders, 2005) . As mentioned previously, facial expression can be used as context of the face in a memory task. CDE still occurs in TBI populations, even though participants with TBI are impaired relative to controls on multiple measures of directed tasks of contextual recognition memory (Vakil, 2005; Vakil, Jaffe, Eluze, Groswasser, & Aberbuch, 1996; Vakil, Sherf, Hoffman, & Stern, 1998; Vakil & Tweedy, 1994) . Given the above, it seems that indirect measurement of context memory among individuals following TBI is more resistant than direct memory measurement of contextual information.

Eye-tracking is a powerful method for evaluating a variety of areas including memory, and its advantage is that this method does not require verbal reporting or internal judgment. This feature allows monitoring and analyzing behavioral correlates of encoding and recognition memory processes in special populations, such as children, adults, and people with severe amnesia, as well as animals (Holm, Eriksson, & Andersson, 2008) .

This method also has added value when examining different parameters among a healthy population, as they are not necessarily aware of where their gaze was directed at any given moment. In doing so, it has been found that more than one second before the explicit memory is reported, there was longer observation time toward familiar stimuli versus new stimuli (Hannula, Ryan, Tranel, & Cohen, 2007) . Also, participants were found to have 25 fixations toward a familiar stimulus before declaring it familiar (Holm et al., 2008) . Therefore, many measures have been developed to characterize the types of changes that may occur in eye movements that reflect different processes and memory states. Key measures mentioned in the literature are fixations (number of fixations and their duration), saccades, number of areas where fixations occurred, number of transitions between areas, pupil size, and total dwell time (DT) time spent in an Area of Interest (AOI) (Hannula, Althoff, Warren, Riggs, Cohen, & Ryan, 2010) .

Regarding the number of fixations, participants were found to make more fixations in the study phase toward stimuli that were subsequently identified as familiar (Heisz, Pottruff, & Shore, 2013; Kafkas & Montaldi, 2011; Snow et al., 2011) . Regarding eye movements in the retrieval phase, it should be noted that most studies that examined the retrieval phase compared eye movements toward new items versus toward familiar items, without addressing whether the items were correctly identified. Meanwhile, these studies found that participants looked more closely at stimuli presented in the study phase as opposed to new stimuli (Chanon & Hopfinger, 2008; Chen & Lee, 2015; Ryan, Hannula, & Cohen, 2007) . Contrary to the above findings, note that there are studies indicating that participants sampled fewer regions and spent less time looking at old stimuli compared to new stimuli (Heisz & Shore, 2008; Smith & Squire, 2008).

The studies mentioned above support the validity of eye movement measures as associated with memory processes in various populations. However, few studies have examined measures of eye movement in a population with TBI. A study conducted recently with TBI participants tested the CDE effect of distinctive, trial-unique hats using behavioral and eye movement measures. The results of the eye movements of the TBI group revealed that even though they spent less time focusing on the targets in the study and test phases than the control participants, the relations between DT on target and recognition in the various conditions are similar in both groups (Vakil et al., 2019b) . Another study tested the effect of eye movement reactivation (the tendency to return eye movements to the visual area where the stimuli were first located) on visual memory among individuals with TBI and healthy controls. In this study, the participants were exposed to a stimulus during the test phase under two conditions: free viewing and fixed eyes. The study showed that unlike the control group, individuals with TBI barely reactivated eye movements effectively so they did not benefit from free viewing and did not reactivate their eyes to assist in retrieval (Deitcher, Sachar, & Vakil, 2020) .

The current study replicated and combined two previous studies conducted by Vakil and colleagues. As reported earlier, Vakil et al. (2019a) found CDE of facial expression among healthy participants. CDE was also found by Vakil et al. (2019b) among healthy participants and individuals with TBI by using photographs of male faces wearing hats. In both studies, eye movement measures were used. The present study makes several additional contributions. First, this study examined the contextual effect of facial expressions among participants with TBI, not only among healthy participants as previously mentioned. Second, as noted above, a small number of studies using eye movement measures have been performed among TBI participants in general, and in an examination of an emotional context in particular. Unlike the study conducted by Vakil et al. (2019b) using hats as context, this study focuses on detecting patterns of facial expressions that may be related to facial identity recognition, as well as examining the pattern of eye movements that distinguishes between remembered and forgotten stimuli.

We used happy and angry expressions because these two expressions are clearly recognized emotions and are well dissociable (Rosenberg, McDonald, Dethier, Kessels, & Westbrook, 2014) . Consistent with previous studies that found evidence for context effect among TBI when measured indirectly, we hypothesized that CDE would also be found in the current study. Furthermore, we expected to find a difference in eye movement patterns, so that DT towards faces that were recognized correctly will be longer in the study and test phases in the two groups, as found in previous studies (Chanon & Hopfinger, 2008; Chen & Lee, 2015; Hannula et al., 2007; Ryan et al., 2007) . Finally, as found in the study by Vakil et al. (2019a) , we expected to find a correlation between DT toward the different parts of the face and memory (d’ measure).

2. Method

2.1. Participants

The control group consisted of 23 participants (13 males and 10 females), whose ages ranged from 21 to 39 years (M = 23.99, SD = 3.69) and their education years ranged from 12 to 13 years (M = 12.04, SD = 0.21). Inclusion criteria were age range 19 to 40 and at least 12 years of education. All of them were undergraduate students at Bar-Ilan University (Israel) (students took part in the experiment to fulfill academic requirements). Exclusion criteria were no histories of neurologic or psychiatric disorders, based on self-reports. The patients, recruited from the Loewenstein Rehabilitation Hospital, were diagnosed with moderate-to-severe TBI (mild TBI injuries were excluded) and were hospitalized while participating in the study. The experimental group of patients with TBI consisted of 20 participants (17 males and 3 females), whose ages ranged from 19 to 39 years (M = 23.30, SD = 4.63) and their education ranged from 10 to 16 years (M = 12.05, SD = 1.05). Neither group differed in age, t(41) = 0.54, p > 0.05 and in education year t(41) = −0.29, p = 0.98. The groups were different in the gender proportion, χ²(42) = 4.11, p < 0.05. As mentioned above, the group with TBI included mostly male participants. The Glasgow Coma Scale of patients with TBI ranged from 3 to 10 (M = 5.30, SD = 2.32), and time after onset (in days) ranged from 14 to 245 (M = 88.65, SD = 60.21). The patients were classified as having sustained moderate-to-severe TBI, according to the Glasgow Coma Scale score (ranging from 3 - 8 for severe TBI, and 9 - 12 for moderate TBI) and length of loss of consciousness (for severe TBI, more than 24 hours, and for moderate TBI, more than 30 minutes and less than 24 hours). The study was approved, as required, by the Institutional Review Board of Bar-Ilan University and by the Helsinki Commission of Loewenstein Rehabilitation Hospital. Consent to take part in the study was obtained from all participants.

Note that two individuals from the control group were excluded from the behavioral analysis of sensitivity (d’) due to extreme values on this measure (more than two SDs from the mean in a negative direction). Two more students and one patient with TBI were excluded from analysis of the response bias (C) for the same reason (more than three SDs from the mean in a negative direction). In addition, two participants from the TBI group were excluded from the eye movement analysis in the study phase and one more from the test phase due to recording problems. Finally, three participants with TBI were excluded from the analysis due to high Glasgow Coma Scale scores (13 - 15).

2.2. Instruments

Eye Tracker: The stimuli were presented on a laptop with a 15.6-inch screen. A camera with an infrared source was located at the front of the laptop screen, 60 cm away from the participant, below the participant’s eye level. The temporal parameter presentations of the stimuli were presented on E-PRIME 2.0 software, which also schedules appearance of the stimuli with computer-recorded eye movements. Senso Motoric Instrument (SMI) RED-M remote eye-tracker recorded the eye movements. This system has a sampling rate of 120 Hz and high accuracy of 0.5˚. At the beginning of the experiment, a 9-point calibration cycle was presented, providing a spatial resolution of 0.1˚.

Visual stimuli: In the current study, we used a total of 118 different color facial images of Caucasian males and females with neutral, happy, or angry facial expressions. Fifty-three faces were selected from Karolinska Directed Emotional Faces (Lundqvist, Flykt, & Öhman, 1998) and thirty-one faces from the Radboud Faces Database (Langner et al., 2010) . The photos were selected based on a high inter-rater agreement.

In order to ensure that attention was directed entirely at the face, the model’s hair, ears, and neck were covered with a black oval frame. This method of framing was used in a previous study (Vakil et al., 2019a) . The selected faces had the required emotions (see Appendix 1, a neutral, happy, and angry face, from left to right), and did not feature distracting facial details like beards.

2.3. Memory Instruments

In order to measure the cognitive abilities of the controls and participants with TBI we used several memory tasks: the Rey-Osterrieth Complex Figure (ROCF), Corsi block-tapping test and the Digit Span subtest from the Wechsler Adult Intelligence Scale-Revised (WAIS-R; Wechsler, 1981 ). As expected, the results showed that both groups differed in their memory ability across all tests (see Table 1).

2.4. Task and Procedure

The participants were tested individually. Upon arrival, participants were informed that they would participate in a facial recognition experiment and were instructed to observe and try to remember the faces that appear on the screen for a future recognition test, regardless of their facial expression (the same instructions appeared on the computer screen later). They were also informed that their

Table 1. Group difference on memory tests.

* p < 0.05, ** p < 0.01, *** p < 0.001.

eye movements would be monitored. Calibration was conducted at the beginning of the task using a standard 5-point grid for both eyes. A 4-point grid was used for validation after each calibration trial. If the accuracy exceeded 0.8˚, calibration and validation were repeated.

In the study phase, 42 color photos of faces (21 men and 21 women) were presented to participants consecutively, for 5 seconds. The expression of all faces was neutral. During the test phase, 84 color photos of faces were presented to participants, half of which had appeared in the study phase, and half were new. The stimuli were presented in random order in both the learning and test phases. Participants were asked to press a key on the keyboard to signify whether the image was old (L) or new (A). Unlike the study phase, at the test phase, the images appeared on the screen until there was a response. One-third of the old images (14 images) remained with the same neutral facial expression as they were in study (the Repeat condition), and two-thirds changed to another facial expression, the “Re-pair” condition (14 images changed to angry expressions and 14 to happy expressions). Regarding the new images, one-third of them had a neutral expression, one-third a happy expression and one-third an angry expression.

3. Results

3.1. Behavioral Results

In order to control for response bias, i.e., reinstatement involving a simple criterion shift, with more powerful response when faces bore the expression seen at encoding, we used the sensitivity measure d' (d' = z(H) − z(FA)) (Snodgrass & Corwin, 1988) as the dependent measure. This measure was calculated by combining the Hits Rate (H) (the number of correct “Yes” responses for the 14 old faces in each condition) and the FA rate (the number of correct “Yes” responses for the 14 new faces in each condition). In order to test the CDE, we compared the Repeat condition’s performance with average performance on the two Repair conditions.

Mixed Analyses of Variance were conducted in order to analyze the effect of Group (control versus TBI) and Context (Repeat vs. Repair-averaged). The former is a between-subjects factor, and the latter is a within-subjects factor. It should be noted that based on previous studies conducted in this field (e.g., Vakil et al., 2019a, 2019b; Zwickel & Muller, 2010 ) we expect a minimum effect size of 0.1.

3.2. Recognition (Sensitivity d’)

Consistent with our hypothesis, CDE was found, so that the sensitivity measure was significantly higher under the Repeat condition compared to the Re-pair condition, F(1, 39) = 7.01, p < 0.05, ${\text{η}}_p^2$ = 0.15. Group effect was not significant, so neither group performed differently on the sensitivity measure, in total F(1, 39) = 2.47, p = 0.12, ${\text{η}}_p^2$ = 0.06. It should be noted that although the main effect of group did not reach significance, the effect size is moderate. Examination of the means shows that overall, the sensitivity measure of the control group tends to be positive (M = 0.09, SD = 0.25), while among the TBI group this measure was negative (M = −0.46, SD = 0.25). The interaction between context and group was not found to be significant F(1, 39) = 0.14, p = 0.71, ${\text{η}}_p^2$ = 0.004, so that both groups exhibited CDE to the same extent (see Figure 1).

3.3. Recognition (Response Bias C)

In addition to the sensitivity d’, we also calculated the measure C, C = 0.5 [z(H) + z(FA)], as a criterion or response bias. When C is negative, it reflects a liberal response bias (tendency to say “yes”). When C is positive, it reflects a conservative response bias (tendency to say “no”) (Snodgrass & Corwin, 1988) . In general, a main effect was found for the group, so that control participants on average tended to be more liberal compared to the TBI group, F(1,37) = 8.50, p < 0.01,

Figure 1. Mean (SE) of sensitivity (d') of the two groups (control and TBI) under Repeat and Re-pair-averaged condition. *p < 0.05.

${\text{η}}_p^2$ = 0.19. CDE was not found, so there was no overall difference between the Repeat and the Re-pair conditions, F(1, 37) = 2.71, p = 0.11, ${\text{η}}_p^2$ = 0.07. It should be noted that although the main effect of context did not reach significance, the effect size is moderate. Examination of the means reveals that the C measure of the Repeat and the Re-pair conditions is negative, but in the Repeat condition the participants tended to be more liberal (M = −0.34, SD = 0.50) than in the Re-pair condition (M = −0.26, SD = 0.03). These main effects should be interpreted cautiously because of the significant interaction between group and context effects, F(1, 37) = 4.49, p < 0.05, ${\text{η}}_p^2$ = 0.11. Although there were no differences between the Repeat and the Re-pair conditions among TBI, the control group tended to be significantly more liberal in the Repeat condition as compared to the Re-pair condition (see Figure 2).

3.4. Eye Movement Results

The eye movement measure used in this study is DT, which is defined as the sum of durations of all fixations and saccades that hit the AOI. We analyzed three AOIs: the eyes (each eye is marked separately), the nose, and the mouth (see Appendix 2). These areas were selected because previous studies have shown that people tend to look at these areas most frequently during facial recognition (Barton, Radcliffe, Cherkasova, Edelman, & Intriligator, 2006) . It should be noted that based on previous studies conducted in this field (e.g., Vakil et al., 2019b ), we expect a minimum effect size of 0.09.

3.5. Study Phase

Mixed-design ANOVAs were performed in order to test the differences in DT towards faces that were remembered (Hits) between the context conditions (Repeat and Re-pair, within-subjects variable) among the two groups (control and TBI, between-subject variable) and the three AOIs (eyes, mouth, and nose). We

Figure 2. Mean (SE) of response bias (C) of the two groups (control and TBI) under Repeat and Re-pair-averaged condition. *p < 0.01.

found a significant main effect for group, so that DT towards correctly recognized faces was higher among the control group than the TBI group, F(1, 37) = 15.27, p < 0.001, ${\text{η}}_p^2$ = 0.29. Main effect of context was also found, so DT towards faces in the Repeat condition was higher than toward faces in the Re-pair condition, F(1, 37) = 4.08, p < 0.05, ${\text{η}}_p^2$ = 0.10. Main effect of AOI was close to significance, F(2, 74) = 2.90, p = 0.06, ${\text{η}}_p^2$ = 0.07. Examination of the data revealed that participants tended to look more at the nose. The interactions between context and group, F(1, 37) = 0.21, p = 0.65, ${\text{η}}_p^2$ = 0.006 did not reach significance, and likewise for the AOI and group interaction, F(2,74) = 2.51, p = 0.09, ${\text{η}}_p^2$ = 0.06. The latter interaction is close to significance and the effect size is moderate, so we looked more deeply into the interaction and found that only in the control group was there a tendency for longer DT towards the nose. Finally, the triple interaction between context, AOI and group was marginally significant, F(2,74) = 2.61, p = 0.08, ${\text{η}}_p^2$ = 0.07. In order to detect the source of the triple interaction, we examined the DT separately for the three AOIs in the Repeat condition (see Figure 3). We found that in the control group, there was longer DT for recognized faces, F(1, 37) = 10.46, p < 0.01. Furthermore, while the control group spent more time looking at the nose, there was no significant difference in the TBI group between the three AOIs. In the Re-pair condition, we found the same main effect in which the control group looked at the recognized faces longer than the TBI group F(1, 37) = 11.10, p < 0.01, but there were no differences between the groups regarding the three AOIs (see Figure 4).

We also performed a comparison of the DT for recognized faces (Hits) versus forgotten faces (Miss). From this analysis, it was found that participants looked longer at recognized faces (Hits, M = 14.93, SD = 0.68) than at faces that were forgotten (Miss, M = 11.11, SD = 0.77), F(1, 37) = 6.43, p < 0.05, ${\text{η}}_p^2$ = 0.15. In addition, we found significant interaction between the recognition result (Hits vs. Miss) and the group, F(1, 37) = 10.81, p < 0.01, ${\text{η}}_p^2$ = 0.23, as this difference

Figure 3. Mean of DT for the 3 AOI of the two groups (control and TBI) under Repeat condition in the learning phase. *p < 0.01.

Figure 4. Mean of DT for the 3 AOI of the two groups (control and TBI) under Re-pair condition in the learning phase. *p < 0.001.

was found only among the control group (Hits, M = 17.80, SD = 1.00; Miss, M = 9.02, SD = 1.23) but not among individuals with TBI (Hits, M = 12.06, SD = 1.01; Miss, M = 13.19, SD = 1.33). The additional interactions did not reach significance.

3.6. Test Phase

Mixed-design ANOVA’s were performed to examine the differences between the Repeat and the Re-pair conditions (within-subjects variable) and between the two groups (control and TBI, between-subjects variable) in the DT on faces that were remembered and those that were forgotten (Hit and Miss condition, within-subjects variable). Comparison of Hits and Miss conditions revealed that participants looked more at recognized faces compared to forgotten faces F(1, 35) = 9.08, p < 0.01, ${\text{η}}_p^2$ = 0.21, with no significant interactions between the recognition result (Hits vs. Miss) and the group, F(1, 35) = 2.12, p = 0.15, ${\text{η}}_p^2$ = 0.06 or between the recognition result and the context, F(1, 35) = 1.01, p = 0.32, ${\text{η}}_p^2$ = 0.03. The former interaction indicates that the difference between DT towards faces that were remembered correctly and faces that were forgotten was greater in the control group compared to the TBI group. We found a significant main effect of context, F(1, 35) = 5.57, p < 0.05, ${\text{η}}_p^2$ = 0.14, as the DT in the Re-pair condition was longer (M = 13.28; SD = 0.45) than in the Repeat condition (M = 12.60; SD = 0.38). Finally, we did not find a significant main effect of group, F(1, 35) = 0.78, p = 0.38, ${\text{η}}_p^2$ = 0.02, and no significant interactions between context and group, F(1, 35) = 0.08, p = 0.78, ${\text{η}}_p^2$ = 0.002 or between context, the recognition result and group, F(1, 35) = 0.18, p = 0.67, ${\text{η}}_p^2$ = 0.01 (see Figure 5).

Finally, we compared the DT for recognized faces (Hits) versus the faces that were mistakenly recognized (FA) between the two groups. This analysis will allow us to examine whether the results reported above in which DT toward stimuli that were remembered (“Hit”) is longer than stimuli that were forgotten

Figure 5. Mean of DT for the 3 AOI in total of the two groups (control and TBI) under Hit and Miss memory result in the test phase. *p < 0.01.

(“Miss”), is due to the association with recognition and not due to the stimulus selection (choosing “yes”). It was found that the DT for recognized faces (Hits) was higher compared to faces that were recognized mistakenly, (FA) F(1, 35) = 61.38, p < 0.001, ${\text{η}}_p^2$ = 0.64, Hits, M = 14.80, SD = 0.68, FA, M = 8.61, SD = 0.63. Neither the group effect, F(1, 35) = 0.62, p = 0.44, ${\text{η}}_p^2$ = 0.02, nor the interaction between the recognition result and group, F(1, 35) = 0.11, p = 0.74, ${\text{η}}_p^2$ = 0.003 reached significance.

4. Discussion

The findings of this study replicated previous findings by Vakil et al. (2019a) with healthy participants, demonstrating the effect of facial expressions (i.e., context) on the memory of facial identity (i.e., target, information to be remembered). CDE was found since recognition of facial identity was better under the Repeat condition, i.e., when the face was presented with the original neutral expression, compared to faces in which the facial expression was changed to happy or angry (i.e., Re-paired). Interestingly, in the present study, it was also found that the group with TBI showed CDE to the same extent as the control group. These findings are consistent with previous findings demonstrating that while individuals with TBI have impaired memory when tested explicitly, memory for contextual memory when tested implicitly (i.e., CDE) is preserved (Vakil, 2005; Vakil et al., 1996, 1998; Vakil, Blachstein, & Hoofien, 1991) . These results also suggested that although individuals with TBI had difficulty in perceiving and identifying emotions that were displayed in various modes (Mancuso, Magnani, Cantagallo, Rossi, Capitani, Galletti, & Robertson, 2015; Bornhofen & McDonald, 2008; McDonald & Saunders, 2005) , the ability to implicitly identify emotions is still preserved, as evidenced by the CDE effect that was found among TBI participants. Analysis of response bias as measured by C index revealed that participants tended to have a more liberal response bias when the facial expression remained constant from study to test (i.e., Repeat condition) than when facial expression was changed from study to test (i.e., Re-pair condition). However, this effect was found only within the control group. Among the TBI group, there was no difference between the conditions. Thus, just like the control group, the group with TBI tended to have better recognition in the Repeat than the Repair condition (measured by d’), but unlike the controls, did not show a higher rate of C (liberal response bias) under the Repeat than the Re-pair condition. A possible explanation for this finding is that the group with TBI, being aware of their memory deficits, adopted a more conservative response bias, which was not affected by the familiarity of the contextual information (facial expression).

One of the aims of the current study was to test whether there is a dissociation or association between representation of the identity, the consistent elements of the face and the changeable elements of the face (such as facial expression). Consistent with the previous study by Vakil et al. (2019a) , the current study’s finding found a CDE using facial expression, contradicting Bruce and Young (1986) and Haxby et al. (2000) cited above, who claimed that the representation of identity must be relatively independent of the representation of the changeable aspects of a face. If this approach was correct and the face is represented separately from the facial expression, then in the current study we would not have found a difference in recognition when the facial expression was the same at the study or changed in the test. Therefore, these results support the approach that suggests that there is an interaction between identity recognition and facial recognition (Yankouskaya et al., 2012) .

Eye movement results of the current study support previous results that showed that there is an association between eye movements (DT during study) and recognition memory performance during a subsequent test (see Table 2 & Table 3). This result was found among the control group and the TBI group. Contrary to our hypothesis, we did not find a CDE effect, as there was no difference in the correlation between the DT towards the AOI and memory performance (d') under the Repeat and Re-pair conditions. Further analysis reveals that in the study phase, the control group, but not TBI patients, looked longer at the faces in the Repeat condition (especially at the nose) and towards faces that were recognized than those that were forgotten. These results are consistent with previous reports showing that during face recognition, participants tend to focus first on the nose (Hsiao & Cottrell, 2008) . The results provide additional support for findings of the connection between familiarity and eye movements, so that longer DT towards the stimuli is associated with higher performance in recognition of those stimuli (Heisz et al., 2013; Kafkas & Montaldi, 2011; Snow et al., 2011) . The same result was found regarding the test phase, i.e., longer DT was also associated with higher recognition performance, thus supporting earlier studies in which participants looked longer at stimuli presented in the study phase as opposed to new stimuli (Chanon & Hopfinger, 2008; Chen & Lee, 2015; Ryan et al., 2007) .

The current study has several limitations. The main one is the sample size that includes about 20 participants in each group due to the limitations in research

Table 2. Correlation between DT and d’ in the study phase (in total and separately for the 3 AOI) in the repeat and the re-pair conditions.

*p < 0.05, **p < 0.01.

Table 3. Correlation between DT and memory in the test phase (in total and separately for the 3 AOI) in the Repeat and the Re-pair conditions.

*p < 0.05, **p < 0.01.

on clinical population. As can be noted in the results section, some of the analyses were close to significance and in some cases the effect size was moderate. We believe that increasing the sample size could lead to some of the effects reaching significance, and we encourage researchers to continue with this line of research. The second limitation concerns the study set which in the study phase includes only faces with neutral expression. According to research that memory might work differently according to facial emotion (i.e., Vakil et al., 2019a; Zwickel & Muller, 2010 ), and the results of the current study that show evidence for CDE using neutral face expression, we propose further studies that would examine the CDE effect of facial expressions with various emotions (anger, joy, etc.) among a TBI population, as well as using eye movement measures.

5. Conclusion

The findings of the current study contribute to our knowledge about context effect among individuals with TBI who suffer from several dysfunctions including memory, social interaction problems and difficulty in understanding facial expressions (i.e., Deitcher et al., 2020; Suh et al., 2006; Vakil, 2005 ). The current study indicated that the group with TBI was affected similarly to the control group from the change in facial expressions (i.e., CDE), so even though they may have explicit difficulties identifying facial expressions when tested implicitly, this ability is preserved. Eye movements were another measure tested in the current study, in the study and test phases. It seems that longer DT is associated with better recognition in the study phase and the test phase as well.

Funding

This work was supported by the Israeli Ministry of Defense, Rehabilitation Department, under Grant number [203003-846].

Appendix 1

Faces with facial expressions: neutral, happy, and angry face, from left to right, respectively.

Appendix 2

Three AOIs were chosen: the eyes (each eye is marked separately), the nose, and the mouth.

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1]	Barbas, H., & Blatt, G. J. (1995). Topographically Specific Hippocampal Projections Target Functionally Distinct Prefrontal Areas in the Rhesus Monkey. Hippocampus, 5, 511-533. https://doi.org/10.1002/hipo.450050604
[2]	Barton, J. J. S., Radcliffe, N., Cherkasova, M. V., Edelman, J., & Intriligator, J. M. (2006). Information Processing during Face Recognition: The Effects of Familiarity, Inversion, and Morphing on Scanning Fixations. Perception, 35, 1089-1105. https://doi.org/10.1068/p5547
[3]	Bornhofen, C., & Mcdonald, S. (2008). Emotion Perception Deficits Following Traumatic Brain Injury: A Review of the Evidence and Rationale for Intervention. Journal of the International Neuropsychological Society, 14, 511-525. https://doi.org/10.1017/S1355617708080703
[4]	Bruce, V., & Young, A. (1986). Understanding Face Recognition. British Journal of Psychology, 77, 305-327. https://doi.org/10.1111/j.2044-8295.1986.tb02199.x
[5]	Cabeza, R., Locantore, J. K., & Anderson, N. D. (2003). Lateralization of Prefrontal Activity during Episodic Memory Retrieval: Evidence for the Production-Monitoring Hypothesis. Journal of Cognitive Neuroscience, 15, 249-259. https://doi.org/10.1162/089892903321208187
[6]	Chanon, V. W., & Hopfinger, J. B. (2008). Memory’s Grip on Attention: The Influence of Item Memory on the Allocation of Attention. Visual Cognition, 16, 325-340. https://doi.org/10.1080/13506280701459026
[7]	Chen, H. C., & Lee, Y. S. (2015). The Eye Movement Measure of Memory and Its Relationship with Explicit Measures. Consciousness and Cognition, 33, 354-363. https://doi.org/10.1016/j.concog.2015.02.003
[8]	Coad, B. M., Postans, M., Hodgetts, C. J., Muhlert, N., Graham, K. S., Lawrence, A. D. (2020). Structural Connections Support Emotional Connections: Uncinate Fasciculus Microstructure Is Related to the Ability to Decode Facial Emotion Expressions. Neuropsychologia, 145, Article ID: 106562. https://doi.org/10.1016/j.neuropsychologia.2017.11.006
[9]	Davachi, L., & DuBrow, S. (2015). How the Hippocampus Preserves Order: The Role of Prediction and Context. Trends in Cognitive Sciences, 19, 92-99. https://doi.org/10.1016/j.tics.2014.12.004
[10]	Deitcher, Y., Sachar, Y., & Vakil, E. (2020). Effect of Eye Movement Reactivation on Visual Memory among Individuals with Moderate-to-Severe Traumatic Brain Injury (TBI). Journal of Clinical and Experimental Neuropsychology, 42, 208-221. https://doi.org/10.1080/13803395.2019.1704223
[11]	Diana, R. A., Yonelinas, A. P., & Ranganath, C. (2007). Imaging Recollection and Familiarity in the Medial Temporal Lobe: A Three-Component Model. Trends in Cognitive Sciences, 11, 379-386. https://doi.org/10.1016/j.tics.2007.08.001
[12]	George, M. S., Ketter, T. A., Gill, D. S., Haxby, J. V., Ungerleider, L. G., Herscovitch, P., & Post, R. M. (1993). Brain Regions Involved in Recognizing Facial Emotion or Identity: An Oxygen-15 PET Study. The Journal of Neuropsychiatry and Clinical Neurosciences, 5, 384-394. https://doi.org/10.1176/jnp.5.4.384
[13]	Hannula, D. E., Althoff, R. R., Warren, D. E., Riggs, L., Cohen, N. J., & Ryan, J. D. (2010). Worth a Glance: Using Eye Movements to Investigate the Cognitive Neuroscience of Memory. Frontiers in Human Neuroscience, 4, Article 166. https://doi.org/10.3389/fnhum.2010.00166
[14]	Hannula, D. E., Ryan, J. D., Tranel, D. & Cohen, N. J. (2007). Rapid Onset Relational Memory Effects Are Evident in Eye Movement Behavior, but Not in Hippocampal Amnesia. Journal of Cognitive Neuroscience, 19, 1690-1705. https://doi.org/10.1162/jocn.2007.19.10.1690
[15]	Haxby, J. V., Hoffman, E. A., Gobbini, M. I. et al. (2000). The Distributed Human Neural System for Face Perception. Trends in Cognitive Sciences, 4, 223-233. https://doi.org/10.1016/S1364-6613(00)01482-0
[16]	Hayes, S. M., Ryan, L., Schnyer, D. M., & Nadel, L. (2004). An fMRI Study of Episodic Memory: Retrieval of Object, Spatial, and Temporal Information. Behavioral Neuroscience, 118, 885-896. https://doi.org/10.1037/0735-7044.118.5.885
[17]	Heisz, J. J., & Shore, D. I. (2008). More Efficient Scanning for Familiar Faces. Journal of Vision, 8, 1-10. https://doi.org/10.1167/8.1.9
[18]	Heisz, J. J., Pottruff, M. M., & Shore, D. I. (2013). Females Scan More than Males: A Potential Mechanism for Sex Differences in Recognition Memory. Psychological Science, 24, 1157-1163. https://doi.org/10.1177/0956797612468281
[19]	Holm, L., Eriksson, J., & Andersson, L. (2008). Looking As If You Know: Systematic Object Inspection Precedes Object Recognition. Journal of Vision, 8, 1-7. https://doi.org/10.1167/8.4.14
[20]	Hsiao, J. H., & Cottrell, G. (2008). Two Fixations Suffice in Face Recognition. Psychological Science, 19, 998-1006. https://doi.org/10.1111/j.1467-9280.2008.02191.x
[21]	Kafkas, A., & Montaldi, D. (2011). Recognition Memory Strength is Predicted by Pupillary Responses at Encoding While Fixation Patterns Distinguish Recollection from Familiarity. The Quarterly Journal of Experimental Psychology, 64, 1971-1989. https://doi.org/10.1080/17470218.2011.588335
[22]	Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D. H. J., Hawk, S. T., & van Knippenberg, A. (2010). Presentation and Validation of the Radboud Faces Database. Cognition & Emotion, 24, 1377-1388. https://doi.org/10.1080/02699930903485076
[23]	Lundqvist, D., Flykt, A., & Öhman, A. (1998). The Karolinska Directed Emotional Faces KDEF. CD ROM from Department of Clinical Neuroscience, Psychology Section, Karolinska Institutet.
[24]	Mancuso, M., Magnani, N., Cantagallo, A., Rossi, G., Capitani, D., Galletti, V., Cardamone, G., & Robertson, I. H. (2015). Emotion Recognition Impairment in Traumatic Brain Injury Compared with Schizophrenia Spectrum: Similar Deficits with Different Origins. The Journal of Nervous and Mental Disease, 203, 87-95. https://doi.org/10.1097/NMD.0000000000000245
[25]	McDonald, S., & Saunders. J. C. (2005). Differential Impairment in Recognition of Emotion across Different Media in People with Severe Traumatic Brain Injury. Journal of the International Neuropsychology Society, 11, 392-399. https://doi.org/10.1017/S1355617705050447
[26]	Metoki, A., Alm, K. H., Wang, Y., Ngo, C. T., & Olson, I. R. (2017). Never Forget a Name: White Matter Connectivity Predicts Person Memory. Brain Structure and Function, 222, 4187-4201. https://doi.org/10.1007/s00429-017-1458-3
[27]	Peterson, M. S., & Kramer, A. F. (2001). Attentional Guidance of the Eyes by Contextual Information and Abrupt Onsets. Perception and Psychophysics, 63, 1239-1249. https://doi.org/10.3758/BF03194537
[28]	Rolls, E. T. (2018). The Storage and Recall of Memories in the Hippocampo-Cortical System. Cell Tissue Research, 373, 577-604. https://doi.org/10.1007/s00441-017-2744-3
[29]	Rolls, E. T. (2019). The Orbitofrontal Cortex and Emotion in Health and Disease, Including Depression. Neuropsychologia, 128, 14-43. https://doi.org/10.1016/j.neuropsychologia.2017.09.021
[30]	Rosenberg, H., McDonald, S., Dethier, M., Kessels, R. P. C., & Westbrook, R. F. (2014). Facial Emotion Recognition Deficits following Moderate-Severe Traumatic Brain Injury (TBI): Re-Examining the Valence Effect and the Role of Emotion Intensity. Journal of the International Neuropsychological Society, 20, 994-1003. https://doi.org/10.1017/S1355617714000940
[31]	Ryan, J. D., Hannula, D. E., & Cohen, N. J. (2007). The Obligatory Effects of Memory on Eye Movements. Memory, 15, 508-525. https://doi.org/10.1080/09658210701391022
[32]	Schacter, D. L. (1987). Implicit Memory: History and Current Status. Journal of Experimental Psychology: Learning, Memory and Cognition, 13, 501-518. https://doi.org/10.1037/0278-7393.13.3.501
[33]	Schmitter-Edgecombe, M., & Seelye, A. M. (2012). Recovery of Content and Temporal Order Memory for Performed Activities Following Moderate to Severe Traumatic Brain Injury. Journal of Clinical and Experimental Neuropsychology, 34, 256-268. https://doi.org/10.1080/13803395.2011.633497
[34]	Snodgrass, J. G., & Corwin, J. (1988). Pragmatics of Measuring Recognition Memory: Applications to Dementia and Amnesia. Journal of Experimental Psychology: General, 117, 34-50. https://doi.org/10.1037/0096-3445.117.1.34
[35]	Snow, J., Ingeholm, J., Levy, I., Caravella, R., Case, L., Wallace, G., & Martin, A. (2011). Impaired Visual Scanning and Memory for Faces in High-Functioning Autism Spectrum Disorders: It’s Not Just Eyes. Journal of the International Neuropsychological Society, 17, 1021-1029. https://doi.org/10.1017/S1355617711000981
[36]	Suh, M., Basu, S., Kolster, R., Sarkar, R., McCandliss, B., & Ghajar, J. (2006). Increased Oculomotor Deficits during Target Blanking as an Indicator of Mild Traumatic Brain Injury. Neuroscience Letters, 410, 203-207. https://doi.org/10.1016/j.neulet.2006.10.001
[37]	Trojano, L., Moretta, P., & Estraneo, A. (2009). Communicating Using the Eyes without Remembering It: Cognitive Rehabilitation in a Severely Brain-Injured Patient with Amnesia, Tetraplegia and Anarthria. Journal of Rehabilitation Medicine, 41, 393-396. https://doi.org/10.2340/16501977-0344
[38]	Vakil, E. (2005). The Effect of Moderate to Severe Traumatic Brain Injury (TBI) on Different Aspects of Memory: A Selective Review. Journal of Clinical and Experimental Neuropsychology, 27, 977-1021. https://doi.org/10.1080/13803390490919245
[39]	Vakil, E., & Tweedy, J. R. (1994). Memory for Temporal Order and Spatial Position Information: Tests of the automatic-Effortful Distinction. Neuropsychiatry, Neuropsychology, and Behavioral Neurology, 7, 281-288.
[40]	Vakil, E., Aviv, O., Mishael, M., Schwizer Ashkenazi, S., & Sacher, Y. (2019b). Direct and Indirect Measures of Context in Patients with Mild-to-Severe Traumatic Brain Injury (TBI): The Additive Contribution of Eye Tracking. Journal of Clinical and Experimental Neuropsychology, 6, 644-652. https://doi.org/10.1080/13803395.2019.1604946
[41]	Vakil, E., Blachstein, H., & Hoofien, D. (1991). Automatic Temporal Order Judgment: The Effect of Intentionality of Retrieval on Closed-Head-Injured Patients. Journal of Clinical and Experimental Neuropsychology, 13, 291-298. https://doi.org/10.1080/01688639108401044
[42]	Vakil, E., Jaffe, R., Eluze, S., Groswasser, Z., & Aberbuck, S. (1996). Word Recall versus Reading Speed: Evidence of Preserved Priming in Head-Injured Patients. Brain and Cognition, 31, 75-89. https://doi.org/10.1006/brcg.1996.0026
[43]	Vakil, E., McDonald, S., Allen, S. K., & Vardi-Shapiro, N. (2019a). Facial Expressions Yielding Context-Dependent Effect: The Additive Contribution of Eye Movements. Acta Psychologica, 192, 138-145. https://doi.org/10.1016/j.actpsy.2018.11.008
[44]	Vakil, E., Raz, T., & Levy, D. A. (2007). Multifactorial Context Effects on Visual Recognition Memory. Quarterly Journal of Experimental Psychology, 60, 916-923. https://doi.org/10.1080/17470210701357568
[45]	Vakil, E., Sherf, R., Hoffman, M., & Stern, M. (1998). Direct and Indirect Memory Measures of Temporal Order and Spatial Location: Control versus Closed-Head Injury Participants. Neuropsychiatry, Neuropsychology, and Behavioral Neurology, 11, 212-217. https://doi.org/10.1037/0894-4105.11.4.545
[46]	Vuilleumier, P., & Pourtois, G. (2017). Distributed and Interactive Brain Mechanisms during Emotion Face Perception: Evidence from Functional Neuroimaging. Neuropsychologia, 45, 174-194. https://doi.org/10.1016/j.neuropsychologia.2006.06.003
[47]	Wechsler, D. A. (1981). Wechsler Adult Intelligence Scale—Revised Manual. The Psychological Corporation.
[48]	Winston, J. S., Henson, R. N. A., Fine-Goulden, M. R., & Dolan, R. J. (2004). fMRI-Adaptation Reveals Dissociable Neural Representations of Identity and Expression in Face Perception. Journal of Neurophysiology, 92, 1830-1839. https://doi.org/10.1152/jn.00155.2004
[49]	Wright, M. J., Wong, A. L., Obermeit, L. C., Woo, E., Schmitter-Edgecombe, M., & Fuster. J. M. (2014). Memory for Performed and Observed Activities Following Traumatic Brain Injury. Journal of Clinical and Experimental Neuropsychology, 36, 268-277. https://doi.org/10.1080/13803395.2014.884543
[50]	Yankouskaya, A., Booth, D. A., & Humphreys, G. (2012). Interactions between Facial Emotion and Identity in Face Processing: Evidence Based on Redundancy Gains. Attention, Perception, & Psychophysics, 74, 1692-1711. https://doi.org/10.3758/s13414-012-0345-5
[51]	Zwickel, J., & Müller, H. J. (2010). Observing Fearful Faces Leads to Visuo-Spatial Perspective Taking. Cognition, 117, 101-105. https://doi.org/10.1016/j.cognition.2010.07.004