Detection and Recuperation of Mental Fatigue

Abstract

Mental fatigue is a complex state that results from prolonged cognitive activity. Symptoms of mental fatigue can include change in mood, motivation, and temporary deterioration of various cognitive functions involved in goal-directed behavior. Extensive research has been done to develop methods for recognizing physiological and psychophysiological signs of mental fatigue. This has allowed the development of many AI-based models to classify different levels of fatigue, using data extracted from eye-tracking device, EEG, or ECG. In this paper, we present an experimental protocol which aims to both generate/measure mental fatigue and provide effective strategies for recuperation via VR sessions paired with EEG and eye tracking devices. This paper first provides a comprehensive state-of-the-art of mental fatigue predictive factors, measurement methods, and recuperation strategies. Then the paper presents an experimental protocol resulting from the state-of-the-art to 1) generate and measure mental fatigue and 2) evaluate the effectiveness of virtual therapy for fatigue recuperation, using a virtual reality (VR) simulated environment. In our work, we successfully generated mental fatigue through completion of cognitive tasks in a virtual simulated environment. Participants showed significant decline in pupil diameter and theta/alpha score during the various cognitive tasks. We trained an RBF SVM classifier from Electroencephalogram (EEG) data classifying mental fatigue with 95% accuracy on the test set. Finally, our results show that the time allocated for virtual therapy did not improve pupil diameter in post-relaxation period. Further research on the impact of relaxation therapy on relaxation therapy should allocate time closer to the standard recovery time of 60 min.

Share and Cite:

Assaf, A. , Abdessalem, H. and Frasson, C. (2023) Detection and Recuperation of Mental Fatigue. Journal of Behavioral and Brain Science, 13, 15-31. doi: 10.4236/jbbs.2023.132002.

1. Introduction

Mental fatigue is a complex state mainly resulting in mood variation, change in motivation, and a temporary decline of cognitive functions [1] [2] . The consequences of such a state can negatively impact workplace performance and, sometimes, be a potential danger for oneself or/and others. Thus, it is not surprising that mental fatigue is the most frequent cause of accident in the workplace [3] .

In recent years, many intelligent systems have been developed to detect mental fatigue, using various physiological and psychophysiological features [4] . However, most of works published on the subjects are limited to the detection of mental fatigue without addressing options to alleviate symptoms. In theory, sleep and/or sufficient resting time are essential to recover cognitive functions affected by prolonged hours of work [5] . However, daily responsibilities and duties do not always allow enough time to rest and fully recover attentional resources. Thus, more attention should be paid to the development of recuperation strategies after inducing fatigue. Hence, they can be used to temporarily alleviate some symptoms of mental fatigue, until proper resting time is possible for a more complete recovery.

Previous work in the field of mental fatigue generation and detection mostly uses classic laboratory settings where users are asked to engage in various cognitive exercises or simulations on a computer screen. These settings are rarely representative of real-world contexts in which mental fatigue arises and thus challenges integrity of the data collected. In our work, we not only intend to 1) generate and measure mental fatigue but also 2) test the effect of virtual relaxation therapy on the recuperation of fatigue symptoms. Moreover, we intend to use virtual reality for (1) and (2) to best reproduce contexts and emotions in which mental fatigue arises and recuperation can occur. Virtual Reality (VR) session paired with Electroencephalogram (EEG) recording has been used in several experiments to assess emotional response from a virtual simulated environment [6] [7] . Moreover, this framework has also been used for relaxation purposes: immersing the participants into a relaxing environment to reduce stress and anxiety [8] .

The goal of our experiment is to use VR session paired with EEG and Eye tracking to 1) generate and measure mental fatigue and 2) investigate the benefits of relaxation therapy. This paper will first present related work in mental fatigue generation/measurement and recuperation. The following sections present predictive factors of mental fatigue, physiological and psycho-physiological measures, recuperation strategy and the methods and material for our experiment protocol which have been motivated by the findings and key concepts of the state-of-the-art detailed in the next sections. We will present our results in terms of the chosen fatigue indicators: pupil diameter, theta/alpha ratio, task engagement index. Finally, we will present the machine learning model obtained to classify mental fatigue as well as the effect of 10 - 15 min virtual therapy session on pupil diameter.

2. Related Work

Kamińska et al. investigated the use of EEG signals to classify a subject’s mental stress level using virtual reality environment. Participants were immersed in two alternating VR interactive simulations: stress inducing and relaxing. The stress inducing environment consisted of the Stroop test, while relaxing environment consisted of interactive relaxing scene based on scenarios created for psychotherapy treatment. During the session, brain wave activity was continuously monitored using EEG, and participants were asked to fill a questionnaire to assess their mood and level of stress, before and after the session. The experimenters used a convolutional neural network (CNN) to classify the level of stress of the participants and matched the subjective stress assessment of the participants with 96.42% accuracy [9] .

Like many other related studies, Kamińska et al. established their labels based on subjective assessment of fatigue via the questionnaire. However, subjective feedback questionnaires are time-consuming and unreliable for real-time fatigue detection. Ren Ziwu and colleagues [10] developed a Radical Basis Function (RBF) Neural Network to detect fatigue in driving simulation using EEG signals. Instead of using questionnaire, they used eye closure, a well-known fatigue indicator to label fatigue and alert segments. Ren Ziwu et al. achieved 92.71% mean accuracy on their RBF neural network.

While most of the recent literature on VR-based emotion induction and recognition is mostly focused on emotions such as stress, anxiety, and fear [11] [12] , the use of VR for inducing mental fatigue has not been as extensively explored. However, we believe that VR environment can provide realistic experience to induce mental fatigue. When it comes to reproducing natural environments and real-world circumstances, many studies support that VR allow users evoke emotions in a more natural approach [13] [14] .

3. Predictive Factors of Mental Fatigue

Predictive factors are the set of tasks demands exerted on an individual, that might influence whether they become mentally fatigued by the task. For instance, the time spent on a task, the number of cognitive resources required, and the level of engagement are important factors able to predict if a task is mentally fatiguing or not. Several factors can be at the origin of mental fatigue: 1) time on task, 2) workload, and 3) task engagement. Moreover, different levels of predictive factors can lead to different types of fatigue. This section presents the predictive factors (1) (2) (3) of mental fatigue and the different types of fatigue that can result from them.

Time on Task (TOT): The effect of task duration on mental fatigue and performance is known as the time-on-task effect (TOT). In general, mental fatigue increases as the time spent on a task increases. However, it should be noted that the relation between TOT and performance is not linear: during the first blocks of a task, an improvement in performance can be observed as a result of learning or automatization of performance [15] . Nonetheless, this peak in performance is generally followed by a decrease in performance caused by mental fatigue, which makes task duration an important task-related prediction factor.

Workload (WL): The mental workload can be defined as the number of cognitive resources or/and energy required to execute a cognitive task, requiring attention, memory, alertness or decision making [16] . In general, working under high levels of mental workload over prolonged periods results in an individual’s depletion of cognitive resources and energy and, eventually, mental fatigue [17] . Thus, mental workload is also an important task-related factor in predicting mental fatigue.

Task Engagement (TE): The level of attention, involvement, and interest one dedicates to a particular task is one of the many factors affected by mental fatigue. Hence, mental fatigue can result in an unwillingness for further efforts, abandoning behavior, where one becomes disengaged with the current task [2] . Consequently, task engagement decreases with TOT effects (mental fatigue).

Passive vs. Active Fatigue: An important theory proposed by Desmond and Hancock’s (2001) suggested there are two types of fatigue: active and passive. In driving studies, active fatigue is characterized by elevated stress and results from a continuous and prolonged demanding interaction vehicle control requiring constant perceptual and motor adjustments. On the other hand, passive fatigue is characterized by task disengagement and is the result of prolonged hours of little to no perceptual-motor response or interaction with vehicle control [18] [19] . Thus, active fatigue appears to occur in higher workload conditions while passive fatigue occurs in lower workload conditions. Research by Saxby and Matthews (2008) later confirmed that the passive fatigue induced by low workload condition resulted in a significantly greater task disengagement over time compared to the active fatigue and control group. Moreover, performance impairment was greater in the passive fatigue condition [19] .

4. Physiological and Psycho-Physiological Measures

4.1. Pupil Diameter in Mental Fatigue and Workload

Pupil diameter (PD) is another eye measurement which can be used to detect workload and mental fatigue. As mental fatigue increases the pupil diameter decreases with respect to baseline measurements [20] . While pupil diameter is also sensitive to changes in arousal and fatigue, it is also sensitive to changes in mental workload. Gonca Gokce Menekse Dalveren and colleagues conducted a study to measure changes in mental workload in surgical residents during surgery where participants were subjected to a computer-based simulation of surgical task. This study found that pupil diameter grows in direct proportion with mental workload [21] . Bastian Pfleging and colleagues also used pupil diameter to develop a model able to predict mental workload under different task load demands. They were able to accurately predict workload under varying mental demands and lightning condition with 75% accuracy [22] . Thus, during the transition state between high mental workload and mental fatigue, pupil diameter is expected to decrease.

4.2. Electroencephalogram (EEG) Power Spectral Density

The electroencephalogram is a non-invasive way to measure the electrical activity originating from the brain from a set of electrodes spaced on the scalp. EEG is a common way to assess and monitor mental workload and fatigue because of the fluctuation in EEG waveforms, delta (0.5 - 4 Hz), theta (4 - 8 Hz), alpha (8 - 12 Hz), beta (12 - 18 Hz), in various cortical areas [23] . Figure 1 shows a scalp topography of electroencephalogram activity before (beginning) and after (end) driving simulation. Blue indicates a decrease in power spectra and red an increase in power spectra for the specified waveband, from Zhao et al. 2012 [24] .

Figure 1. Brain activity before and after driving simulation (from Zhao et al. 2012 [24] ).

Transition state between increasing workload and mental fatigue generally results in an overall increase in delta, theta and alpha frequency band, and a decrease in beta frequency band as we can see on the figure.

4.2.1. Task Engagement Index

Pope and colleagues (1995) at the NASA developed an engagement task index based on EEG frequency bands applied in a closed-loop system to modulate task allocation. This index is defined by the ratio of frequency bands Beta/(Alpha + Theta) [25] . This ratio reflects allocation of attention, information-gathering, and visual processing [26] .

4.2.2. Workload with EEG

The ratio of theta power in frontal areas over alpha power in parietal area is a well-known ratio index to measure changes in workload. Thus, this index is based on the principle that during increases in task demands, theta power increases in frontal regions while alpha power decreases in parietal regions [27] . Thus, increases in the theta frontal to alpha parietal ratio indicate an increase in task load perception. However, from transition states from high mental workload to increasing mental fatigue levels, this ratio decreases as alpha power starts increasing [28] . Thus, this ratio can be used to monitor changes in workload and mental fatigue perception.

5. Recuperation of Mental Fatigue

5.1. Recovery

Justine R. Magnuson and colleagues (2021) investigated the development and recovery of task-induced mental fatigue. They collected EEG signals and subjective assessment of participants while performing a 60 min N-back test followed by a 60 min post-task resting time. The authors observed that mental fatigue was induced after 30 - 45 min of the N-back test, from both objective and subjective measures. They also observed a 60 min recovery for some alpha and theta bands to baseline levels during resting time. However, complete recovery was not achieved after 60 min resting time [29] .

5.2. Virtual Travel Relaxation

VR-based relaxation therapy has received a lot of attention over the last few years. Among them, the virtual train therapy [30] is a VR relaxation therapy in which participants are immersed in a moving train and travel virtually looking through the windows to the landscape (Figure 3). The virtual train immersive therapy has shown to be effective in promoting relaxation, apathy, sleep, and reducing negative emotion such as stress and anxiety [30] . Hence, the cognitive benefits of a short virtual train therapy motivate the interest of exploring its effects on mental fatigue symptoms.

6. Methods and Materials

The protocol we present for this paper has two parts: the first is generation of mental fatigue and the second is recuperation of mental fatigue which will be detailed in the following Sections 6.1 and 6.2. We aimed to successfully generate mental fatigue in the first part and test the effectiveness of virtual relaxation therapy in the second part while collecting EEG and eye tracking data from participants. EEG data from the first part of the experiment were transformed and fed to multiple machine learning models to select the one which best fits mental fatigue EEG measures, and eye tracking data will be used to isolate the pupil diameter to derive the labels (more details in 6.3). In addition, we expect to track the fluctuation of workload and task engagement to analyze predictive patterns and possible formulas for mental fatigue. Workload and task engagement will be computed with the formulas presented in 4.2.1 and 4.2.2.

6.1. Generation of Mental Fatigue

We generate mental fatigue among participants through a simulated VR environment of a work office where participants are prompted to perform various set of cognitive tasks. There are two types of cognitive tasks set: one with distractors and one without distractors. Distractors are fake answers or wrong hints that will be displayed in the user’s virtual visual field to distract them and provoke fatigue. During these tasks, fatigue is measured through 3 indicators: pupil dilation, workload, and task engagement. The exercises chosen to generate mental fatigue are aligned with the active fatigue framework discussed in Section 3: we generate mental fatigue through cognitive tasks which aim to exert a high cognitive load on the participants.

At first, participants will perform in both set of tasks consecutively (25 min). Then the relaxation period will begin (10 min). Finally, the participant will answer different questions from the set of cognitive tasks without distractors (10 min).

The set of cognitive tasks with distractors consists of mental arithmetic tasks, anagram tasks and backward digits span (BDS) tasks. The user must perform these tasks within the time allocated for each. For each cognitive tasks, fake answers in the form of “hints” are displayed to distract the users. The arithmetic tasks (Figure 2) consist of a series of addition and subtraction of decimal numbers presented in the virtual environment that must be solved by the user. An example of a mental arithmetic task is an equation of the form “24.54 − 12.89 + 2.13 + 11.72 − 7.08 − 3.23” in which the user must use the virtual keyboard to summit the correct answer to this equation, 15.19 in this example.

Figure 2. Mental arithmetic task (first task of the cognitive task with distractors) before the appearance of distractor answers on the screen.

In the anagram task, the user is presented with a set of letters, and must rearrange these letters to form the appropriate dictionary word. For example, a user is presented with the letters “R”, “E”, “D”, “R”, “U”, “M”, and must use his virtual keyboard to type the correct word “MURDER”.

In the backward digits span, the user is presented with an ordered sequence of numbers that appears on the user’s virtual environment for a short period of time. The user is asked to memorize the sequence during that period. A few moments after the sequence is removed from the user’s screen, the participant is asked to recall the sequence in the reverse order of presentation using their virtual keyboard. For instance, if the user is presented with the sequence “9-7-4-2-5-9-3”, he must memorize the sequence during the allowed time window and enter the reverse sequence order, “3-9-5-2-4-7-9”.

Then another set of cognitive tasks without distractor is presented to the participant. The goal here is to show that the eventual fatigue has been previously generated and is no more due to the distractors. This set has different types of exercises of a 5 min duration to measure cognitive and memory performance. These tests continue to generate mental fatigue as they require concentration, attention, memory, and other cognitive resources. This set is composed of an attention, a naming exercise and three different memory tests to evaluate contextual/visual memory, working memory and short-term memory. These exercises will allow cognitive performance comparison between post-fatigue and post-recuperation states.

6.2. Relaxation Therapy “Travelling Therapy”

After the mental fatigue generation events, participants are exposed to the “Travelling Therapy” VR environment for 10 - 15 minutes to reduce their negative emotions, mental fatigue and increase their concentration [30] . This environment (Figure 3) projects the users into a virtual train (360-degree environment), where they are sitting and can turn their head to look through the windows or observe events inside the train. The windows reveal a natural or relaxing landscape which can consist of forests with animals, mountains, snow mountains, simple roads, lakes, etc. In the train, the user can also see persons and/or pets interacting with each other. It is known that exposure to natural elements and landscape aid in the recovery of attentional fatigue and experiments realized with this virtual train has proved a reduction of negative emotion and an increase of memory and cognitive function [30] . We aim to analyze if exposure to this virtual natural landscape can produce similar effects on mental fatigue recovery.

6.3. Participants, Data Preprocessing and Labels

31 participants (15 female and 16 male) aged between 19 and 29 years old were invited to a room at Beam Me Up office, partner of the project, 5925 Monkland Ave, H4A1G7, Montréal, to complete the different steps of the experiment and a few real-time and offline outcome measures.

Figure 3. The virtual train (relaxing environment).

During the experiment, some electrodes loss their signal due to displacement caused by movement of the head and weak signal caused by hair density on the scalp. We selected EEG channels in a way that for the majority of participant, selected, at least 1 of 2 electrodes defining a scalp region remained intact. Regions are defined as follows:

1) (fp1, fp2) prefrontal

2) (f4, f8) frontal right

3) (f7, f3) frontal left

4) (t4, t6) temporal right

5) (t3, t5) temporal left

6) (c4, c3) central

7) (p3, p4) parietal

8) (o1, o2) occipital

Selection of participant: Participant missing 1 or more regions from the selected scalp region were discarded from the dataset. Thus, we had to discard the central electrodes (c4, c5) from the dataset for all participants. Moreover, we had to discard 7 participants (4 female and 3 male) due to a too large number of failed electrodes during the experiment. Finally, 3 more participants (1 female, 2 male) were additionally discarded from the dataset due to failure of the EEG headset during the experiment or because they were unable to finish the experiment due to discomfort/headache. The final dataset was thus composed of 12398 EEG power spectral density segments of prefrontal, frontal right, frontal left, temporal right, temporal left, parietal, and occipital electrodes from the 21 participants (10 female and 11 male).

The EEG signal was band-pass filtered with a fourth-order Butterworth filter (high-pass filter cut-off frequency: 1 Hz, low-pass filter cut-off frequency: 30 Hz). The signal was then passed onto a wavelet denoising filter to remove signal noise, then the power spectral density was computed from the result using the Welch method. Outliers were removed using the interquartile range independently for each participant dataset. From the result of the Welch calculation to retrieve power spectral density, the absolute power of theta, alpha, beta and delta at each electrode was calculated. Relative powers (W/Hz) were used to compute workload and task engagement using the formulas presented in

Relative power of θ = (power of θ)/(power of θ + power of α + power of β + power of ∆)

Relative power of α = (power of α)/(power of θ + power of α + power of β + power of ∆)

Relative power of β = (power of β)/(power of θ + power of α + power of β + power of ∆)

Relative power of ∆ = (power of ∆)/(power of θ + power of α + power of β + power of ∆)

Band power of two electrodes elements of the same region were averaged when both electrode data was available. If one of the two electrodes was identified as “railed”, only the available was taken to describe the target region.

Labels have been established based on variation of the pupil diameter, which is a well-known physiological indicator of mental fatigue and mental workload: pupil increases with respect to baseline when subjects are under high mental charge and decreases when fatigue rises. Baseline range is calculated at the first 60 epoch for each participant. 10 epoch moving segments by 1 epoch increments are assigned to the label 0 if the mean of the segment falls between the baseline range, 1 if the mean falls above baseline range, and 2 if the mean falls below the baseline range.

7. Results and Discussion

The experimental results are presented in the following subsection. The first section will be concerned with analysis of the pupil size with respect to the method chosen to classify mental fatigue. The second part will be concerned with the fatigue indicators, workload and task engagement, progression throughout the experiment. The third part will address selection of the best machine learning model to classify mental fatigue. Finally, the last part will be concerned with the analysis of fatigue indicator following the relaxation period.

7.1. Pupil Size Analysis

Labels were assigned with respect to pupil size variation with respect to baseline, to identify segments of increasing workload and fatigue. Assigning fatigue labels according to eye measurements is a methodology that was employed by Ren Ziwu and colleagues [10] to classify mental fatigue using an RBF neural network. The correlation between pupil diameter and workload/fatigue progression was verified against a well-known EEG indicator of workload and fatigue: Theta F/Alpha P (theta bands of frontal electrodes over alpha bands of parietal electrodes). Thus, increases in the theta frontal to alpha parietal ratio indicate an increase in workload. An increase of this index followed by a decrease indicates transition states from high mental workload to increasing mental fatigue levels [28] . Figure 4 shows the measure of such a ratio during cognitive exercise for participant #2. We see, in dashed yellow, an increase of both indexes followed by a decrease in dashed red which indicates transition states from high mental workload to increasing mental fatigue levels.

Figure 4. Normalized values of the smoothed ThetaF/AlphaP ratio and pupil diameter.

The correlation value between the smoothed and normalized pupil diameter and thetaF/alphaP ratio over all participants is 0.54, which is a strong relation for the nature of the data (see Figure 4). The distribution of the labels during the first half and the second half of the cognitive tasks part of the experiment shows a significant decrease in absence of fatigue segments and slow fatigue progression proportion. Moreover, we noted a significant increase of rapid fatigue segment proportion during the second half of cognitive tasks compared to the first half. To compare the different fatigue segments, we used the Wilcoxon non-parametric test. This test was chosen (instead of paired t-test) because the distribution of fatigue segments did not follow the normality assumption from parametric tests (Table 1). Results show a significant decrease in non-fatigue and slow fatigue signs, and significant increase in rapid fatigue progression signs.

Table 1. Wilcoxon test of the proportion of mental fatigue signs comparing the first half of the experiment and the second half.

7.2. Workload and Task Engagement

The workload and task engagement EEG score was computed from the formulas presented in 4.2.1 and 4.2.2. To provide evidence of their evolution we compare the measures of the first half of tasks with the measures of the second half. Hence, we observe a significant decrease in workload score during the second half of the cognitive tasks compared to the first half (p < 0.05) (Figure 5 and Figure 6). However, we did not observe a significant decrease (p < 0.05) in task engagement for the majority of participant. This phenomenon can be explained by the nature of the fatigue generated by the cognitive tasks: active fatigue. Active fatigue (fatigue induced by high workload condition) results in a slower task disengagement overtime than passive fatigue (fatigue induced by low workload condition) [19] . Thus, a longer period of cognitive tasks would have been needed to see a change in task engagement. Figure 5 shows that the mean ThetaF/AlphaP index decreases in the second half as a result of fatigue. No significant change in task engagement.

Figure 5. Mean score indices of thetaF/alphaP (workload) and beta/(alpha + theta) (task engagement) during the first half and second half of the fatigue generation tasks.

Figure 6. ThetaF/AlphaP score of participant #1 during the mental fatigue generation tasks.

Another phenomenon that might explain the stability of the task engagement observed in this experiment is the diversity of the exercise presented to the participants. Thus, it is possible that participants did not disengage in the presented task simply because the rapid rotation of different exercises did not allow them to get bored by them, but rather intrigued. This motivational influence on task-engagement was also observed by Jesper F. Hopstaken and colleagues, who observed that change in reward during mentally fatigue task was able to restore task-engagement levels and performance score, while participants still reported to be highly fatigued [31] . Thus, change in task engagement is likely to be influenced by a complex relation of trade-off/reward system rather than depletion of a finite reserve of cognitive resources.

Figure 6 shows the progression of workload. This index is known to decrease as mental fatigue increases. In dashed yellow, an increase of theta/alpha score followed by a decrease, shown in dashed red, indicates transition states from high mental workload to increasing mental fatigue levels.

7.3. RBF SVM for Mental Fatigue Classification

Our aim was to classify states of mental fatigue segment during a period where participants were asked to perform in various cognitive tasks. All 8 electrodes cerebral regions with their transformation (presented in 6.3), except for central electrode region, were used in the feature matrix for classification. The balanced accuracy as opposed to standard accuracy was chosen as one of the parameters to evaluate model performance, as the proportion of labels across the dataset was unbalanced. Various machine learning (ML) algorithm candidates were trained and tested in order to select the ML algorithm which best fits our data with respect to evaluation metric “balanced accuracy” and “f1” measures (Table 2). Among the 8 different classifiers candidates, RBF SVM showed a better performance with respect to balance accuracy and f1 evaluation metrics.

After parameter tuning achieved through grid search algorithm, the balanced accuracy of the RBF SVM on the test data was 95%.

Table 2. Evaluation metric of different machine learning models on the EEG dataset before parameter tuning.

7.4. Relaxation Period

Our aim was to verify the effect of a short (10 - 15 min) VR relaxation therapy on the participant fatigue levels. During the experiment, the set of cognitive tasks without distractors were presented before and after the relaxation therapy to compare fatigue levels through pupil dilation.

A Wilcoxon test performed on mean pupil diameter across participants, for pre and post relaxation period, favored rejection of the alternative hypothesis: a difference in mean pupil diameter across participants eye segments for pre and post relaxation period (Figure 7). We used the Wilcoxon test for the same reasons indicated in Section 7.1.

Figure 7. Mean pupil diameter across all participants during pre-relaxation period and post-relaxation period.

While these results do not invalidate the use of virtual therapy to alleviate mental fatigue symptoms, they indicate that time allocated for relaxation in this experiment might have not been sufficient to observe benefits in this recuperation technique. Thus, since standard recovery after mentally fatiguing tasks can take as long as 60 min post-task [29] , further research would be needed to compare the recuperation of fatigue symptoms with the aid of VR relaxation therapy and standard no activity recovering.

8. Conclusion

Mental fatigue is a complex multi-faceted state resulting from a change of emotions and cognitive capacity. Although it is impossible to fully recreate real-world contexts and emotions from which mental fatigue and recuperation can arise, VR session enables us to achieve laboratory setting that are closer to the ones observed in real world situations. Our experiment combines VR sessions with EEG and eye tracking to generate, measure and recuperate mental fatigue in a way that best capture the context in which the measured signals occur in the real world. Our results showed significant decrease in theta/alpha ratio score and pupil size across participants during the completion of cognitive tasks, indicating increasing mental fatigue. While we did not find significant changes in task engagement, we believe that this phenomenon is the results of the fast rotation of different cognitive tasks, which did not allow enough time for participant to get bored by them. Using EEG for feature data and pupil diameter to derive labels, we were able to develop an RBF SVM classifier able to detect signs of mental fatigue with 95% balanced accuracy. This important finding (ratio score and pupil size) provides a way to detect and measure the apparition of fatigue.

Thus, we believe that using virtual reality in the context of mental fatigue generation and recuperation will allow us to extract data from an environment that is better at mimicking real-life situations. Finally, more investigation needs to be done on the benefits of relaxing virtual environments such as the virtual train on mental fatigue symptoms, mainly in terms of the relaxing exposure time needed to observe significant effects on mental fatigue. Further research on the impact of relaxation therapy on mental fatigue should allocate time closer (but shorter) to the standard recovery time of 60 min.

Acknowledgements

We acknowledge NSERC-CRD (National Science and Engineering Research Council Cooperative Research Development), Prompt, and BMU (Beam Me Up) for funding this work.

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References

[1] Ishii, A., Tanaka, M. and Watanabe, Y. (2014) Neural Mechanisms of Mental Fatigue. Reviews in the Neuroscience, 25, 469-479.
[2] Hopstaken, J.F., van der Linden, D., Bakker, A.B. and Kompier, M.A.J. (2015) The Window of My Eyes: Task Disengagement and Mental Fatigue Covary with Pupil Dynamics. Biological Psychology, 110, 100-106.
https://doi.org/10.1016/j.biopsycho.2015.06.013
[3] Hopstaken, J.F., van der Linden, D., Bakker, A.B. and Kompier, M.A.J. (2015) A Multifaceted Investigation of the Link between Mental Fatigue and Task Disengagement. Psychophysiology, 52, 305-315.
https://doi.org/10.1111/psyp.12339
[4] Parekh, V., Shah, D. and Shah, M. (2020) Fatigue Detection Using Artificial Intelligence Framework. Augmented Human Research, 5, Article No. 5.
https://doi.org/10.1007/s41133-019-0023-4
[5] Åkerstedt, T., Nilsson, P.M., Kecklund, G., Sonnentag, S., Perrewé, P.L. and Ganster, D.C. (2009) Sleep and Recovery. In: Sonnentag, S., Perrewé, P.L. and Ganster, D.C., Eds., Current Perspectives on Job-Stress Recovery (Research in Occupational Stress and Well Being, Vol. 7), Emerald Group Publishing Limited, Bingley, 205-247.
https://doi.org/10.1108/S1479-3555(2009)0000007009
[6] Horvat, M., Dobrinic, M., Novosel, M. and Jercic, P. (2018) Assessing Emotional Responses Induced in Virtual Reality Using a Consumer EEG Headset: A Preliminary Report. Proceedings of the 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), Opatija, 21-25 May 2018, 1006-1010.
https://doi.org/10.23919/MIPRO.2018.8400184
[7] Tarrant, J., Viczko, J. and Cope, H. (2018) Virtual Reality for Anxiety Reduction demonstrated by Quantitative EEG: A Pilot Study. Frontiers in Psychology, 9, 1280.
https://doi.org/10.3389/fpsyg.2018.01280
[8] Amores, J., Richer, R., Zhao, N., Maes, P. and Eskofier, B. (2018) Promoting Relaxation Using Virtual Reality, Olfactory Interfaces and Wearable EEG. Proceedings of the 2018 IEEE 15th International Conference on Wearable and Implantable Body Sensor Networks (BSN), Las Vegas, 4-7 March 2018, 98-101.
https://doi.org/10.1109/BSN.2018.8329668
[9] Kamińska, D., Smólka, K. and Zwoliński, G. (2021) Detection of Mental Stress through EEG Signal in Virtual Reality Environment. Electronics, 10, 2840.
https://doi.org/10.3390/electronics10222840
[10] Ren, Z.W., Li, R.H., Chen, B., et al. (2021) EEG-Based Driving Fatigue Detection Using a Two-Level Learning Hierarchy Radial Basis Function. Frontiers in Neurorobotics, 15, Article ID: 618408.
https://doi.org/10.3389/fnbot.2021.618408
[11] Chirico, A., Cipresso, P., Yaden, D.B., Biassoni, F., Riva, G. and Gaggioli, A. (2017) Effectiveness of Immersive Videos in Inducing Awe: An Experimental Study. Scientific Reports, 7, Article No. 1218.
https://doi.org/10.1038/s41598-017-01242-0
[12] Weiß, S., Klassen, N. and Heuten, W. (2021) Effects of Image Realism on the Stress Response in Virtual Reality. Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology, Osaka, 8-10 December 2021, 1-10.
https://doi.org/10.1145/3489849.3489885
[13] Marín-Morales, J., Higuera-Trujillo, J., Greco, A., Guixeres, J., Llinares, C., Scilingo, E. and Valenza, G. (2018) Affective Computing in Virtual Reality: Emotion Recognition from Brain and Heartbeat Dynamics Using Wearable Sensors. Scientific Reports, 8, Article No. 13657.
https://doi.org/10.1038/s41598-018-32063-4
[14] Xu, T., Yin, R., Shu, L. and Xu, X. (2019) Emotion Recognition Using Frontal EEG in VR Affective Scenes. Proceedings of the 2019 IEEE MTT-S International Microwave Biomedical Conference (IMBioC), Nanjing, 6-8 May 2019, 1-4.
https://doi.org/10.1109/IMBIOC.2019.8777843
[15] Csathó, á., Linden, D.V., Hernádi, I., Buzás, P. and Kalmár, G. (2012) Effects of Mental Fatigue on the Capacity Limits of Visual Attention. Journal of Cognitive Psychology, 24, 511-524.
https://doi.org/10.1080/20445911.2012.658039
[16] Chaouachi, M., Jraidi, I. and Frasson, C. (2011) Modeling Mental Workload Using EEG Features for Intelligent Systems. 19th International Conference, UMAP 2011, Girona, 11-15 July 2011, 50-61.
https://doi.org/10.1007/978-3-642-22362-4_5
[17] Fan, J.L. and Smith, A.P. (2017) The Impact of Workload and Fatigue on Performance. 1st International Symposium, H-WORKLOAD 2017, Dublin, 28-30 June 2017, 90-105.
https://doi.org/10.1007/978-3-319-61061-0_6
[18] Desmond, P.A. and Hancock, P.A. (2001) Active and Passive Fatigue States. In: Hancock, P.A. and Desmond, P.A., Eds., Stress, Workload, and Fatigue, CRC Press, Boca Raton, 455-465.
https://doi.org/10.1201/b12791-3.1
[19] Saxby, D.J., Matthews, G., Hitchcock, E.M., Warm, J.S., Funke, G.J. and Gantzer, T. (2008) Effect of Active and Passive Fatigue on Performance Using a Driving Simulator. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 52, 1751-1755.
https://doi.org/10.1177/154193120805202113
[20] Bafna, T., Bækgaard, P. and Hansen, J.P. (2021) Mental Fatigue Prediction during Eye-Typing. PLOS ONE, 16, e0246739.
[21] Dalveren, G.G.M., Cagiltay, N.E., Ozcelik, E. and Maras, H. (2018) Insights from Pupil Size to Mental Workload of Surgical Residents: Feasibility of an Educational Computer-Based Surgical Simulation Environment (ECE) Considering the Hand Condition. Surgical Innovation, 25, 616-624.
https://doi.org/10.1177/1553350618800078
[22] Pfleging, B., Fekety, D.K., Schmidt, A. and Kun, A.L.. (2016) A Model Relating Pupil Diameter to Mental Workload and Lighting Conditions. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI’16), Association for Computing Machinery, New York, 5776-5788.
https://doi.org/10.1145/2858036.2858117
[23] Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D. and Babiloni, F. (2014) Measuring Neurophysiological Signals in Aircraft Pilots and Car Drivers for the Assessment of Mental Workload, Fatigue and Drowsiness. Neuroscience and Biobehavioral Review, 44, 58-75.
https://doi.org/10.1016/j.neubiorev.2012.10.003
[24] Zhao, C., Zhao, M., Liu, J.P. and Zheng, C.X. (2012) Electroencephalogram and Electrocardiograph Assessment of Mental Fatigue in a Driving Simulator. Accident Analysis and Prevention, 45, 83-90.
https://doi.org/10.1016/j.aap.2011.11.019
[25] Pope, A., Bogart, E.H. and Bartolome, D.S. (1995) Biocybernetic System Evaluates Indices of Operator Engagement in Automated Task. Biological Psychology, 40, 187-195.
https://doi.org/10.1016/0301-0511(95)05116-3
[26] Laurent, F., Valderrama, M., Besserve, M., Guillard, M., Lachaux, J.-P., Martinerie, J. and Florence, G. (2012) Multimodal Information Improves the Rapid Detection of Mental Fatigue. Biomedical Signal Processing and Control, 8, 400-408.
https://doi.org/10.1016/j.bspc.2013.01.007
[27] Rojas, R.F., Valderrama, M., Besserve, M., Guillard, M., Lachaux, J.-P., Martinerie, J. and Florence, G. (2020) Electroencephalographic Workload Indicators during Teleoperation of an Unmanned Aerial Vehicle Shepherding a Swarm of Unmanned Ground Vehicles in Contested Environments. Frontiers in Neuroscience, 14, 40.
https://doi.org/10.3389/fnins.2020.00040
[28] Cao, T., Wan, F., Wong, C.M., da Cruz, J.N. and Hu, Y. (2014) Objective Evaluation of Fatigue by EEG Spectral Analysis in Steady-State Visual Evoked Potential-Based Brain-Computer Interfaces. BioMedical Engineering OnLine, 13, Article No. 28.
https://doi.org/10.1186/1475-925X-13-28
[29] Magnuson, J.R., Doesburg, S.M. and McNeil, C.J. (2021) Development and Recovery Time of Mental Fatigue and Its Impact on Motor Function. Biological Psychology, 161, Article ID: 108076.
https://doi.org/10.1016/j.biopsycho.2021.108076
[30] Ben Abdessalem, H., Byrns, A., Cuesta, M., Manera, V., Robert, P., Bruneau, M.A., Belleville, S. and Frasson, C. (2020) Application of Virtual Travel for Alzheimer’s Disease. SENSORNET 2020, the 9th International Conference on Sensor Networks, Valletta, 28-29 February 2020, 52-60.
https://doi.org/10.5220/0008976700520060
[31] Hopstaken, J.F., van der Linden, D., Bakker, A.B., et al. (2016) Shifts in Attention during Mental Fatigue: Evidence from Subjective, Behavioral, Physiological, and Eye-Tracking Data. Journal of Experimental Psychology: Human Perception and Performance, 42, 878-889.
https://doi.org/10.1037/xhp0000189

Copyright © 2024 by authors and Scientific Research Publishing Inc.

Creative Commons License

This work and the related PDF file are licensed under a Creative Commons Attribution 4.0 International License.