Subliminal faces with different valence: Unconscious mismatch detection indicates interactions between unconscious processing ()
1. INTRODUCTION
Many studies have shown that unconscious information processes could affect subsequent processing of different information [1-5]. In the meantime, other studies have demonstrated that consciousness modulates unconscious processes [6-11]. The question arises whether interactions between unconscious processes can occur.
One indication for a possible unconscious influence on unconscious processing was found in two studies [12,13]. Here, the unconscious stimulus (erotic pictures and impending threat) elicited unconscious spatial attention resulting in behavioral response changes. Obviously, the processing of the unconscious stimulus and the triggering of spatial attention toward a specific unconscious stimulus are different processes. Therefore, these results might demonstrate that unconscious processes influence other unconscious processes. In addition, unconscious processes were generally thought to only involve posterior brain areas as hypothesized in the global neuronal workspace theory [14,15]. From a theoretical point, fMRI studies demonstrating that unconscious processes reach prefrontal areas [16,17] are at conflict with the global neuronal workspace theory and suggest that unconscious processes could reach a high level of information processing [17].
In order to investigate the possibility of interactions between unconscious processes, we rely on evidence that subliminally presented facial expressions are processed differently in dependence of their valence [18-24]. In particular, we assume that, in comparison to two subliminal facial expressions of same valence, two subliminal facial expressions of different valence would elicit unconscious mismatch detection and a resulting conflict which are likely localized in frontal brain areas. If the effect exists, it would indicate that our cognitive system tries to integrate the two subliminal facial expressions, and moreover that unconscious processes reach higher processing levels and interact with each other. Although sequential presentation of stimuli is often used in studies about the influence between conscious and unconscious processing, the concurrent presentation of subliminally emotional faces could be regarded as processing one face under the influence of the other face just as invisible stimuli could be modulated by concurrent management of attentional load [25]. This is an exploratory research, therefore, both ERP and fMRI techniques are used to see whether the assumed effect is stable.
Two experimental conditions were designed: 1) two simultaneously and subliminally presented faces of different unconscious valence (fearful versus happy face, DUV) and 2) two simultaneously and subliminally presented faces with the same unconscious valence (fearful or happy faces, SUV). DUV probably affects the N2 ERP component more differently than the SUV condition, since this component is sensitive to both subliminal face processing [18,20] and conscious mismatch detection [26]. This N2 effect might be the result of activation differences in the frontal gyrus, because this structure is linked to conscious mismatch processing and unconscious incongruent priming [16,17,27,28].
2. EXPERIMENT 1: ERP STUDY
2.1. Method
2.1.1. Participants
Twenty five participants from Southwest University in China volunteered for this EEG study (12 women, 13 men; aged 18 - 25 years [mean age 22.6 years]). All participants were paid, gave written informed consent, were right-handed, had no history of current or past neurological or psychiatric illness, and had normal or corrected-to-normal vision.
2.1.2. Stimuli
A sample consisting of 40 images of happy facial expression (20 females, 20 males) and 40 images of fearful facial expression (20 females, 20 males) was collected from the Chinese Facial Affective Picture System [29]. The mean valence for fearful and happy faces was 2.73 (SD, 0.44) and 6.28 (SD, 0.61), respectively. The mean arousal for fearful and happy faces was 6.31 (SD, 1.22) and 5.46 (SD, 1.17), respectively. The scale range for valence and arousal was 1 - 9. During the experiment, the face images were displayed centrally on a uniform grey background (RGB = 192, 192, 192) and each face subtended approximately 4.3 (height) × 3.8 (width) degrees of visual angle.
2.1.3. Procedure
The stimulus presentation sequence of an example trial is displayed in Figure 1. In the experiment, participants were asked to monitor a number and indicate if it was
Figure 1. Schematic illustration of the ERP task. Participants were required to identify the numbers as odd or even, having been told that the stimuli prior to the numbers were distractions that they should ignore.
odd or even. First, a fixation cross appeared in the center of the screen for 1000 ms. Subsequently two paired face images were presented side-by-side on the screen for 16 ms (one screen refresh), followed by a backward mask of two scrambled faces for 400 ms. Following a blank screen of 300 ms, a number was displayed for 1500 ms. Participants were asked to indicate if the number was odd or even by pressing 1 or 2 with their right index and middle fingers respectively, as quickly and accurately as possible. The stimuli prior to the number were distractors that should be ignored. The responses were counterbalanced across subjects. At last, a blank screen appeared for 2000 ms. During the experiment, the participants were seated in a quiet room facing a screen placed approximately in 100 cm distance from the eyes and were asked to sit still and blink as little as possible.
There were two conditions. The two faces had either different unconscious valences (DUV) or the same unconscious valence (SUV). To avoid any difference in low-level features, stimuli between DUV and SUV were matched (Figure 2). Thus, unconscious effects between these two conditions cannot be attributed to physical stimulus differences. Face pictures in a trial were of the same gender, and the ratio of odd to even numbers was one to one within each condition. First, the subjects were trained in five practice trials. Subsequently, two experimental blocks were run. Each one consisted of 80 trials, resulting in 80 trials per condition (DUV and SUV). The different conditions in each block were displayed randomly. Between the blocks, the subjects could take an appropriate rest.
After the ERP experiment, the participants were asked to report whether they saw something apart from the numbers, and then a forced-choice discrimination task was performed in order to test the participants’ ability to recognize the masked faces. After a fixation cross, displayed for 1000 ms, a face image was presented for 16 ms, followed by a backward mask of a scrambled face
Figure 2. Examples of the pairing stimuli. Top: SUV condition. Bottom: DUV condition. The four faces for each condition were same, except for their match.
for 400 ms. Then a pair of faces was presented. Participants were asked to determine which one was presented before. Before performing this task, participants were informed that the probability of the two faces being the test face was equal, and that only the accuracy not the speed of the response was important.
2.1.4. ERP Recording and Analysis
Brain electrical activity was recorded from 64 Ag/AgCl electrodes mounted in an elastic cap (Brain Products), electrodes at the left and right mastoids served as reference. The electrooculogram (EOG) was recorded from electrodes placed above and below the right eye (vertical), and at the right side of the right eye and the left side of the left eye (horizontal). All interelectrode impedances were kept below 5 kΩ. The EEG and EOG were amplified using a 0.05 - 80 Hz bandpass and continuously sampled at 500 Hz/channel for offline analysis. Oculor artifacts (blinks and eye movements) were rejected offline. Trials with EOG artifacts (mean EOG voltage exceeding ±80 mV) and those contaminated with artifacts due to amplifier clipping, bursts of electromyographic activity, or peak-to-peak deflection exceeding ±80 mV were excluded from averaging.
The EEG of the main experiment was epochized from −100 to 450 ms relative to the onset of the face stimulus. The first 100 ms served as baseline. The targeted N2 ERP component peaked around 220 to 230 ms with a maximum at left frontal-central electrodes (F1, F3, FC1, FC3, FC5, C1, C3, C5). In order to define statistical differences at this component between the SUV and DUV condition mean amplitude values averaged in this timewindow across these electrodes were submitted to a paired t-test.
2.2. Results
2.2.1. Visibility Test
The participants reported not seeing the masked faces. In the measure of stimulus visibility, 23 of 25 participants scored at chance level (binomial test, p > 0.05), suggesting that these individuals were unable to perceive masked faces. Also at the group level, discrimination performance did not deviate from chance level (mean percentage correct = 48.96%, SD = 0.08, t(22) = −0.61, p = 0.55). The two participants that scored above chance level were excluded from further behavioral and ERP analysis.
2.2.2. Number Judgment
The reaction times (RTs) to the numbers for SUV and DUV were 596.6 ± 146.1 and 603.1 ± 154.0 ms, respectively. The effect of RTs was not significant, t(22) = 1.42, p = 0.17. In addition, the degrees of accuracy for SUV and DUV were 98.6% ± 1.8% and 98.3% ± 2.3%, respectively. The effect was also insignificant, with t(22) = −0.79, p = 0.43.
2.2.3. Electrophysiological Scalp Data
The N2 component was significantly influenced by the stimulus type, t(22) = −2.27, p < 0.05, as the N2 amplitude for DUV trials was less negative than for SUV trials (Figure 3), indicating processing differences between identical and different subliminally perceived facial expressions.
3. EXPERIMENT 2: fMRI STUDY
3.1. Method
3.1.1. Participants
As paid volunteers, eighteen new adults that did not contribute data to the ERP study (7 women, 11 men) aged 19 - 28 years (mean age, 23.6 years) from Southwest University in China participated in this experiment. All participants gave written informed consent, were righthanded, had no history of current or past neurological or psychiatric illness, and had normal or corrected-to-normal vision.
3.1.2. Stimuli
Same as those used in ERP study.
Figure 3. ERP averaged across eight electrodes (F1, F3, FC1, FC3, FC5, C1, C3, C5), neativity is plotted upwards. The N2 component peaked around 220 to 230 ms. Topographical voltage map of the N2 effect (DUV-SUV) in the averaged time window from 220 to 230 ms with the left-fronto-central maximum.
3.1.3. Procedure
The experimental task, stimulus delivery, and recording of behavioral responses were carried out with E-prime 2.0 Software (Psychology Software Tools, Inc. http://www.pstnet.com). The face images were displayed centrally on a uniform grey background (RGB = 192, 192, 192) and each face subtended a visual angle of approximately 4.3 (height) × 3.8 (width) degrees. Stimuli were displayed on a back-projection screen placed at the back of the scanner bore, which was viewed by the participants via a mirror attached to the head-coil. Manual responses were recorded using an MRI-compatible button box.
In the experiment, participants were required to monitor a number for vigilance maintenance and indicate whether the number was odd or even by pressing 1 or 2, respectively, in the button box. The stimulus-responseassignment was counterbalanced across participants. The trial sequence was as follows (Figure 4): a forward mask of two scrambled faces was first displayed for 384 ms, and then two paired face images were presented side-byside on the screen for 16 ms (one screen refresh), followed by a backward mask for 384 ms. For a better BOLD signal, paired face images of the same conditions and the following masks were repeated five times. The last mask was displayed for 400 ms in a trial, followed by the target number for 1600 ms. Note that no image was displayed twice in a trial. Between trials, a central fixation cross was displayed for a jittered inter-trial interval of 2 - 6 s.
The two paired face images were presented in the DUV and the SUV condition. Stimuli between DUV and SUV were matched as described in the ERP study (Figure 2). In a single trial of each condition, only one type
Figure 4. Schematic illustration of the fMRI task. For better blood oxygenation level-dependent (BOLD) signal, the paired face images of one type (DUV and SUV) were successively repeated five times. However, there was no face image displayed twice in a single trial.
of paired face images was displayed and no face image was displayed twice. In the DUV condition, the happy and fearful faces interchanged their positions successively between five subliminal displays to maximize the mismatch which had two sources: one from the concurrent mismatch, another from the successive mismatch. In the SUV condition, happy or fearful face pairs were displayed five times. In addition, the face images in a trial were of the same gender, and the ratio of odd to even numbers was one to one within each condition. The participants were asked to make the odd/even judgment as quickly and accurately as possible. The participants were also informed that the stimuli prior to the number should be ignored. There were two runs, each contained 20 trials of DUV and 20 trials of SUV (10 trials of happy or fearful face, respectively). The different conditions in each run were displayed randomly.
In order to test the participants’ ability to see the masked faces, the participants were asked after fMRI scanning whether or not they saw something apart from the numbers, and then a forced-choice discrimination task was performed while still lying in the scanner. Stimuli and trial times were similar to those used in the formal fMRI experiment. However, one face instead of the paired faces was presented, and the same face was repeated five times in a trial. After the presentation of the last mask, a pair of faces, of which one was the same face presented before, was presented left and right of the fixation. Participants were asked to determine which one was presented previously. The two faces remained on the screen until the participants made a response. There were 60 trials in the discrimination task. Before performing this task, participants were informed that the probability of the two faces being the test face was equal, and that only the accuracy not the speed of the response was important.
3.1.4. fMRI Data Acquisition
Imaging data were acquired with a 3-T Siemens Trio Scanner (Siemens Medical Systems, Erlangen, Germany) using a 12-channel birdcage headcoil. Two functional scans were acquired using an echo planar imaging (EPI) sequence (TR = 2000 ms, TE = 30 ms, flip angle: 90˚; field of view: 220 × 220 mm2; matrix size: 64 × 64). Each functional volume consisted of 32 axial slices of 3 mm thickness with 1 mm gap between slices. Each scan lasted 6 min and consisted of 180 volumes. Two dummy scans were performed prior to the image acquisition to eliminate signals arising from progressive saturation.
3.1.5. fMRI Data Analysis
All pre-processing and statistical analyses were carried out using SPM8 (http://www.fil.ion.ucl.ac.uk/spm/software/spm8/). Prior to pre-processing, the first three volumes of each run were discarded. For each participant, functional images were spatially aligned and slice-time corrected to the first volume of the first run, and then normalized to the Montreal Neurological Institute (MNI) template brain. The normalized functional images (resampled at 3 mm3) were spatially smoothed with a Gaussian kernel of full widthhalf-maximum of 8 mm3. A 128 s temporal high pass filter was applied to remove low-frequency scanner artifacts. Using a first-order autoregressive model (AR-1), temporal autocorrelations were estimated using restricted maximum likelihood estimates of variance components, and maximum likelihood estimates of the activations were formed using the previously resulting non-sphericity [30].
For statistical analysis, we constructed models of two separate regressors coding for onsets and durations (2 s) of DUV and SUV. The regressors were convolved with SPM8’s canonical hemodynamic response function (HRF) and the models were regressed against the observed BOLD data. For each participant, we extracted mean activation estimates (% signal change) for DUV and SUV trials from their individual MFG which activated in whole-brain analysis, using Marsbar software (http://marsbar.sourceforge.net/), and presented the mean activation estimates across all subjects (Figure 5).
All 18 participants were included in these analyses except for one run of one participant because the behavioral results were overwritten. The fMRI analysis included (DUV-SUV) and (SUV-DUV). Both comparesons were thresholded at p < 0.05 (small volume corrected and cluster size > 5 voxels). In addition, the contrasts of (SUV-DUV) was also evaluated at a more liberal threshold (p< 0.01 uncorrected and cluster size > 0 voxels) to see to what extent there would be no activation.
3.2. Results
3.2.1. Visibility Test
The participants reported not seeing the masked faces. In the measure of stimulus visibility, all 18 participants
Figure 5. Left: Comparison of the results of the DUV with SUV. The image shows activation in the left middle frontal gyrus (x = −21, y = 35, and z = −14) at a threshold of p < 0.05 (small volume corrected, cluster size > 5 voxels). Right: The mean and SE of % signal change for SUV and DUV.
scored at chance level (binomial test, p > 0.05), suggesting that these individuals were unable to perceive the masked faces. Also at the group level, discrimination performance did not deviate from chance level (mean percentage correct = 51.00%, SD = 0.06, t(17) = 0.74, p = 0.47).
3.2.2. Number Judgment
The reaction times (RTs) to the numbers for SUV and DUV were 787.2 ± 144.9 and 787.6 ± 135.5 ms, respectively. The effect of facial valence (same vs. different) was not significant, t(17) = −0.06, p = 0.95. In addition, the accuracy rates for SUV and DUV were 96.4% ± 7.2% and 96.3% ± 7.0%, respectively. The difference was also insignificant, with t(17) = 0.12, p = 0.91.
3.2.3. fMRI Results
Comparing the DUV with SUV trials, the fMRI results with whole-brain analysis only showed activation in the left middle frontal gyrus (MFG) (x = −21, y = 35, and z = −14; 21 voxels, and t = 5.26) at a threshold of p < 0.05 (small volume corrected, cluster size > 5 voxels) (Figure 5). The result was further visualized using xjView toolbox (http://www.alivelearn.net/xjview). Thus, the hypothesis of interactions between unconscious processes was supported. Furthermore, comparing SUV with DUV, no activation was observed even at an uncorrected threshold of p < 0.01 (cluster size > 0 voxels).
4. DISCUSSION
In this study, we investigated whether concurrent unconscious processes interact. Both ERP and fMRI data indicated that subliminally presented faces of different valence are differently processed than faces of the same valence. In the first ERP experiment, DUV elicited a smaller N2 as compared to SUV. In the second fMRI experiment, the left MFG was activated under DUV condition in comparison to SUV condition. We assume that in comparison to two subliminal facial expressions of the same valence, two subliminal facial expressions of different valence are subconsciously perceived as a mismatch and elicit a conflict. Previous studies demonstrated that supraliminally presented bilateral face stimuli lead to interhemispheric cooperation [31,32]. Therefore the effect in our studies would indicate that at a certain processing level the brain tries to integrate the expression analyses of the two faces, which likely supports the idea of interactions between unconscious processes. In the following, the ERP and fMRI results as well as theoretical implications are discussed.
4.1. Discussion about ERP and fMRI Results
Because stimuli between DUV and SUV were matched in both ERP and fMRI studies, the results cannot be attributed to differences in low-level features. Instead, we assume that the results reflect unconscious detection of a mismatch. In the ERP study, the N2 component correlated with unconscious mismatch processing and had a left anterior maximum. The N2 was smaller under DUV than under SUV conditions. In contrast to the posterior N2, which is related to the degree of attention required for processing stimuli in the visual modality [33], the anterior N2 was considered to be associated with mismatch or cognitive control [26,34]. For example, the N2 was enhanced for incongruent flankers in Flanker tasks [35] and for no-go trials in go/no-go tasks [34,36]. In comparison with strong evidence that the N2 associated with cognitive control is located in anterior cingulate area [37,38], there are studies that did not find a novelty/mismatch effect in the anterior cingulate area but in the frontal cortex using visual oddball paradigms, which are known to elicit robust N2 components [39,40]. Considering the activation in MFG in the fMRI experiment, it is reasonable to assume that the N2 in the ERP study reflects unconscious mismatch detection rather than cognitive control. Interestingly, in our study the N2 was small-ler under the DUV than under the SUV condition which is at odds with studies investigating conscious stimulus processing. Thus, one possible assumption could be that ERP effects to subthreshold processing differ from ERP effects to supraliminal perception. This is quite speculative, since there is to the best of our knowledge no study that actually measured ERPs to unconsciously perceived primes but only to conscious targets. Comparing two identical studies by Kiefer and colleagues, in which unconscious semantic priming was investigated [41], ERP effects in the N400 component were reversed, when unconscious prime processing was modulated by top-down influences [42]. Future studies should aim at investigating this issue explicitly.
In the fMRI experiment, the left MFG was stronger activated under the DUV condition in comparison to the SUV condition. The MFG has been associated with mismatch processing across a wide range of experimental paradigms. For example, MFG activation was related to auditory mismatch processing when comparing blocks that contained rare duration-deviant tones with blocks of a series of standard tones [27]. In a study with a coherent object in a scenic context and an incoherent object in a scenic context, bilateral MFG was activated during the incoherent condition [28]. Moreover, transcranial magnetic stimulation of the left frontal cortex significantly reduced neuronal and behavioral repetition priming effects (which resembles our SUV condition) [43]. In addition, mid-dorsolateral prefrontal cortex was activated during unconscious priming when comparing the incongruent priming condition with the congruent priming condition [16]. This prefrontal activation was assumed to be related to increasing cognitive demands or to a high level of cognitive conflict [16]. Furthermore, it was found that correlations between neuronal and behavioral priming were specific for faces in the left middle frontal gyrus [44]. Therefore, we assume that the left MFG is involved in unconscious mismatch processing, maybe trying to integrate the two conflicting subliminal facial expressions.
In the DUV condition of the fMRI experiment, the happy and fearful faces interchanged their positions successively between five subliminal displays to maximize the mismatch. Therefore, the mismatch process in DUV condition has two sources: one from the concurrent mismatch, another from the successive mismatch. The additional successive mismatch in fMRI experiment compared to ERP experiment could be related to the reversed result pattern between the two studies. This is a caveat that needs further corroboration.
To summarize, the ERP and fMRI results showed that perceiving two facial expressions of different valence subliminally elicited different unconscious processes as compared to perceiving two facial expressions of the same valence, which might reflect the interaction of unconscious processes due to unconscious mismatch detection.
4.2. Theoretical Implications
The existence of interactions between unconscious processes could amend the theory of global neuronal workspace. In a study about the access threshold to consciousness, Del Cul and colleagues proposed that 1) masked and unmasked stimuli initially elicit identical visual activations, and 2) that the amount of masking is negatively correlated to the depth of processing (i.e. unconscious processes are restricted to occipital and parietal brain areas) [15]. However, there is an exceptional study that showed that subliminal words activated the anterior cortex relative to subliminal pseudowords [45]. This study is in line with our present findings, which demonstrate that unconscious processing is not limited to the posterior brain area.
In addition, the theory of global neuronal workspace proposes two computational spaces: 1) the global workspace, which is composed of distributed and interconnected neurons with long-range axons, and 2) specialized and modular perceptual, motor, memory, evaluative, and attentional processors [46]. Dehaene et al. suggested that automatic processing should activate specialized processors without requiring coordination by global workspace neurons [46]. However, some works indicate that unconscious processes are not necessarily automatic but influenced by top-down factors, like task sets for instance [42,47,48]. Accordingly, one could assume that global workspace neurons are also active during some unconscious processes. This would allow not only for sequential activation of modular processors but also for more complex unconscious cognitive processes like for example the demonstrated unconscious mismatch detection. Just thinking of the existence of interactions between the unconscious perceptual, motor, memory, evaluative, and attentional processors, this could lead to abundant interactions between them, and many phenomena could be well explained. For instance, it could be that persons with obsessive-compulsive disorders cannot consciously control the unconscious memory/evaluative influences on unconscious motor processes. Such an interpretation is corroborated by work demonstrating that motor preparatory processes occur before we have the conscious will to act [49].
5. CONCLUSION
To conclude, this study extends previous results on relationship between conscious and unconscious cognitive processing to unconscious interactions between unconscious cognitive processes. However, we are just at the beginning to understand how different unconscious processes influence each other. Future research needs to further verify and specify its phenomenal and theoretical use.
6. ACKNOWLEDGEMENTS
This research was supported by the National Natural Science Foundation of China (31170983), the Fundamental Research Funds for the Central Universities (NSKD11002) and the Fundamental Research Funds of China West Normal University.
NOTES