Assessing Intercultural Competence Using Situational Judgement Tests: Reports from an EMI Course in Japan ()
1. Introduction
Intercultural communication is the study of how people from diverse cultural and linguistic backgrounds behave, communicate, and perceive the world around them (Lauring, 2011). In higher education settings, educators teaching intercultural communication aim for their students to achieve a level of competence after they complete their courses. This competence relates to various skills, attitudes, and knowledge. While there are various definitions in the literature, in this study, I define intercultural competence as “the ability to communicate effectively and appropriately in intercultural situations based on one’s intercultural knowledge, skills, and attitudes” (Deardorff, 2004: p. 194).
As a result of internationalisation efforts, intercultural development and understanding have become a feature of Japanese higher education contexts. That is, students are expected to develop a second language (often English) as well as their ability to communicate interculturally. The Ministry of Education, Culture, Sports, Science and Technology (MEXT) has outlined specific benchmarks for students to achieve while at university and funded various support mechanisms, to develop globally competent graduates (MEXT, 2014; Bradford, 2019; Allen, 2019). Furthermore, employers and educators stress the importance of intercultural competence among graduates (Sercu, 2004). Employers need graduates who can appropriately interact with others who have different linguistic and cultural backgrounds (Sercu, 2004; Allen, 2021). In addition, educators want to equip students with the skills to function within multicultural and globally connected societies (Sercu, 2004).
Due to the government’s promotion of intercultural communication education, such intercultural communication courses are offered in addition to language courses in Japanese universities (Bradford, 2013; MEXT, 2014). These courses are typically taught as English medium of instruction (EMI) and introduce the study of intercultural communication and develop students’ knowledge, skills, and attitudes (Deardorff, 2004; Bradford, 2013, 2016; Allen, 2021). EMI courses in Japan are those that use English as a lingua franca to teach academic content to students. Such courses are offered to attract inbound international students as well as increase the internationalisation of the institution (Walkinshaw et al., 2017; Shibata, 2021). Typically, in these courses, domestic and international students learn together using English as the mode of instruction. Previous research has acknowledged both positive (e.g., developing skills for worldwide employability) and negative (e.g., language proficiency and ideologies) aspects of these courses in various contexts (e.g., Shibata, 2021; Zhang & Pladevall-Ballester, 2022).
From an intercultural perspective, generally, “the teaching of [intercultural communication] tends to focus on developing students’ awareness of cultural differences, their place within a globalized and diverse society, and how to interact effectively and meaningfully in these spaces” (Allen, 2021: p. 214). While this is the case, there are negative aspects to intercultural EMI courses. For example, these courses are often taught by L1 English speakers who are not trained in intercultural communication education (Shibata, 1998). In addition, MEXT has not adequately indicated how such courses should be implemented and taught as well as how students’ intercultural competence should be measured (Fritz & Sandu, 2020).
Thus, this study aims to explore participants’ responses to situational judgement tests and discuss the efficacy of such an assessment in the Japanese context. Specifically, the study examines responses to three videos depicting intercultural workplace critical incidents. These videos portray issues in punctuality, effectively sharing ideas in a group environment, and having an uncomfortable conversation with someone about a colleague’s behaviour (further explored in Section 3). In the next section, I discuss the literature surrounding intercultural competence and development and the various ways these skills have been assessed.
2. Literature Review
In this section, literature that has focused on defining and describing intercultural competence is discussed (Section 2.1). In addition, studies that have analysed the various forms of intercultural assessment used to assess intercultural competence are explored (Section 2.2).
2.1. Intercultural Competence & Development
The concept of intercultural competence is difficult to define because many scholars have put forth several descriptions and models over the years (Rathje, 2007). These definitions and models have emphasised different aspects of intercultural competence such as language knowledge and performance, behaviour, skills, awareness, and attitudes. For example, from the early 1960s, intercultural competence was determined by one’s ability to use a second (or third) language (Sercu, 2004). Thus, different terms were also used to describe intercultural competence such as intercultural communicative competence, cultural awareness, sensitivity, empathy, and many others (Sercu, 2004; Deardorff, 2006).
While there has been a discrepancy in defining the concept, incorporating scholars’ views on intercultural competency, I adopt Deardorff’s (2006) definition, which she defines as “… the ability to develop targeted knowledge, skills and attitudes that lead to visible behaviour and communication that are both effective and appropriate in intercultural interactions” (pp. 247-248). While the definition is broad, it does outline clear elements of intercultural competence that can be tested, particularly in educational contexts (Fantini, 2012; Tran et al., 2019). These elements are described in Figure 1.
In addition to the various definitions and elements put forward by scholars to define intercultural competency, several other models have also been suggested as ways of describing one’s intercultural development (e.g., Byram, 1997; Dervin, 2010). However, Deardorff (2004, 2006) proposed a process model of intercultural development, which encompasses various elements of competency and movement between individual and interactional processes (shown in Figure 2). Furthermore, the model stresses the ongoing development of one’s intercultural competency and the developing of appropriate and effective communication behaviours (Deardorff, 2006).
While this section has outlined the models of intercultural development, previous studies have also documented how learning activities that assist in the acquisition of intercultural skills, knowledge, and behaviours should be implemented in the classroom. For example, Gudykunst et al. (1991) proposed a master syllabus for effective intercultural teaching in the classroom. This syllabus outlined the various cognitive, affective, and behavioural components required for such instruction. While it is a good starting point, more recent scholars have suggested some changes. Namely, syllabi should be developed through a context-sensitive approach (Díaz & Moore, 2018; Moore & Díaz, 2019; Allen, 2021), which includes up-to-date methodologies, authentic resources, and an exploration of alternative views of conventional theories (Allen, 2021; R’boul, 2021). This also includes authentic and meaningful assessments, which are discussed in
the next section.
2.2. Assessing Intercultural Competence
Intercultural communication courses aim to develop students’ acquisition of skills to operate in multicultural worlds. Assessing intercultural competence is necessary for two reasons: 1) students want to see whether they are making suitable progress and how to improve their skills, and 2) teachers seek ways to better recognise whether their students have successfully acquired these skills (Sercu, 2004). As noted by Sercu (2004),
Educators too are looking for ways of assessment which can provide them with feedback on whether their learners have benefited from their intercultural teaching, and which can prove to society that learners have reached the educational goals set in the curricula (p. 73).
To determine competence, scholars have suggested several methods to assess the knowledge, skills, and attitudes acquired through intercultural development. Often in classrooms, such assessments are used as teaching activities (Sercu, 2004). For example, methods include questionnaires with a variety of open, closed, and Likert-type formats (e.g., Griffith et al., 2016); interview protocols (e.g., Kulich, 2015); observation of communication behaviours during various activities (e.g., groupwork) (e.g., Burdett, 2014); ethnographic activities such as investigating linguistic landscapes (e.g., Hatoss, 2019); implicit association tests (e.g., Bazgan & Norel, 2013); simulation-based tests such as observing students during role plays; and situational judgement tests, such as those described in this study (e.g., Tran et al., 2019). While previous studies have examined situational judgement tests, research that focuses on such activities in an EMI context is limited.
Despite these various assessment methods, scholars have suggested that a single instrument may not be able to adequately judge all aspects of intercultural competency (Sercu, 2004; Fantini, 2012; Griffith et al., 2016). More importantly, recent research has highlighted the need to include actual interaction or ethnographic approaches as a form of intercultural competency assessment (Kulich, 2015) and authentic and context-sensitive forms (Allen, 2021). Hence, the use of critical incident videos offers students these aspects of intercultural assessment; particularly, analysing instances of authentic intercultural interactions and asking questions that stimulate reflection on the developmental aspects of skills, knowledge, and attitudes.
As a result of the discussions in this section, the following research questions are proposed:
1) What do students report when asked to describe, reflect, and analyse cross-cultural scenarios that depict workplace critical incidents?
2) Considering these reports, do situational judgement tests accurately assess students’ intercultural competence?
To answer these research questions, the methodology is discussed in the next section.
3. Methodology
3.1. The Course and Assessments
The current study took place in a private university in western Japan. At this university, students are enrolled in a faculty which focuses on language learning and areas of applied linguistics1. Students are highly motivated to develop their language skills and aspire to work in international companies upon their graduation. A mandatory component of this faculty is for students to complete a one-year study abroad program. These programs are fully coordinated by faculty members at the university. Students select their destination and complete language and undergraduate level study (undergraduate units are pre-selected based on the five areas of focus in the faculty). Once students return from their study abroad, they specialise in one of the five areas of focus at the home faculty, which are: area studies, intercultural communication, translation and interpreting, language communication education, and language analysis.
As a part of the intercultural communication program, students enrolled in an introductory intercultural communication course in 2020-2021. Students in this course completed 15 weeks of study. The course is designed to: 1) introduce and explore intercultural communication concepts; 2) raise awareness of the complex nature of interactions across cultures; 3) and develop affirmative behaviours towards those who have different linguistic and cultural backgrounds. In other words, the course aims to develop students’ intercultural knowledge, skills, and attitudes. The course uses a textbook to guide the concepts covered each week (Introducing Language and Intercultural Communication, Jane Jackson, 2014) and supplementary readings from The Conversation website. The textbook was used as it covers the main topics of intercultural communication and includes authentic material from a variety of contexts to exemplify the theory (e.g., intercultural case studies). The course was designed to be active and engaging, inviting students to reflect on their own intercultural experiences with the content covered in each class. Such content included verbal and non-verbal behaviour, intercultural transitions, identity, ethnocentrism, and global citizenship and intercultural competence.
During the semester, students completed several activities in class including “think, pair, share” sessions, and group discussions. These were used to encourage active learning to confirm students’ understanding of the content in the classes. Furthermore, YouTube videos and other visual aids were used to help students understand the content of this EMI course and reduce the linguistic burden on students. These discussions were part of the overall assessment in the course. Students completed mid-term and final concept quizzes and a group linguistic landscape project that involved experiencing intercultural spaces in their cultural milieu and documenting this experience. This further provided students with an opportunity for authentic experiential learning.
3.2. Participants
In the current study, 48 participants completed the reflective survey that asked them to consider their IC learning and study abroad experiences and complete the situational judgement test. Of the 48 participants, 24 were male and 24 were female. Participants were not all Japanese, some participants identified as mixed nationality (e.g., Japanese-Thai) and some were from other parts of Southeast Asia. Participants also indicated that they spoke languages other than Japanese and English such as Tagalog, Spanish, French, Korean, and Chinese, and they participated in study abroad programs (both online and in-person) hosted in countries such as New Zealand, Australia, Germany, the United Kingdom, the United States, and Korea. Participants in the study were aged between 21 and 22 years and were purposively selected (Baker, 2011; Creswell & Creswell, 2018) as they enrolled in an intercultural communication course. Furthermore, these participants were purposively selected as they were developing their intercultural communication skills, knowledge, and attitudes, and could complete the situational judgement test from a point of knowledge. While students complete language assessments pre and post-study abroad (e.g., IELTS, TOEFL, TSST), their intercultural communicative competence was not measured during their studies.
3.3. Reflective Survey, Situational Judgement Test & Analysis
The data for this study comes from a reflective survey, given to students during the last week of class. This reflective survey had four parts using a mix of open, closed, and Likert scale items: 1) demographic information, 2) study abroad experiences, 3) post-study abroad experiences studying intercultural communication, and 4) situational judgement tests focused on workplace critical intercultural incidents. Results from section three of the survey showed how students conceptualise their own intercultural learning (see Allen, 2021). The surveys were distributed in English, as the course was designed and facilitated as EMI. However, this may have impacted how students expressed themselves.
In section four, students were asked to watch three videos that depicted workplace intercultural incidents and responded to four questions. Table 1 outlines the videos and the issues that arise in each context.
The videos were designed as resources for instructors to use to teach intercultural competence by the Colbourne Institute for Inclusive Leadership (Norquest College). The videos and activities listed on the website were designed to help immigrants settle in Alberta, Canada. However, only the videos were used in the current study. The following reasons outline why the videos are a useful tool in the classroom for teachers and students, and what they potentially offer for researchers:
· The videos provide students with firsthand experiences of workplace intercultural conflict. While some students have experiences of intercultural conflict, this places the student outside of the interaction, to allow them to observe and reflect on the situation.
![]()
Table 1. Overview of critical workplace incident videos.
· Although scripted, the scenarios are an authentic activity or form of assessment to observe and examine students’ reflections on an intercultural critical incident in the workplace.
· From a researcher’s perspective, the assessment provides insights into students’ knowledge, skills, and attitudes about specific forms of workplace critical incidents.
Once the students watched the video during the survey, they were prompted with the four following questions aimed at interrogating the three elements of intercultural competency: knowledge, skills, and attitudes (Deardorff, 2004, 2006, 2009). The following list outlines the open-ended questions and what they aimed to elicit in terms of students’ intercultural competency.
1) What intercultural communication concepts do you notice? (Testing for knowledge and skills);
2) What similarities and differences do you notice compared to your own linguistic/cultural background? (Testing for knowledge and skills);
3) Was the situation handled/managed appropriately? What verbal and non-verbal cues help you? (Testing for knowledge, skills, and attitudes);
4) How would you handle this type of situation in the future? (Testing for skills and attitudes.)
Due to the aim of the study and the nature of the questions above, qualitative research methods were used to collect and analyse participants’ reports focused on intercultural competency. Previous studies have also used qualitative methods to investigate intercultural competency in various ways such as interviews and surveys (e.g., Carlson et al., 2017). In addition, qualitative surveys have been recently adopted by researchers as they are unobtrusive, unrestrictive, straightforward, and allow users to complete them in anonymous ways (Creswell & Creswell, 2018; Braun et al., 2021). Furthermore, Braun et al. (2021) also noted that such methods can provide researchers insights into a participant’s knowledge, beliefs, experiences, understandings and perceptions, and behaviours, which is relevant to the current study.
Once participants completed the survey and submitted their answers, they were downloaded via google forms and thematically analysed (Maguire & Delahunt, 2017) using NVivo (version 1.6.1). Researchers use a thematic analytic approach to identify repeating patterns and themes in participants’ responses (Maguire & Delahunt, 2017). Specifically, in this study, I reviewed, coded (Rossman & Rallis, 2012), and grouped their responses into various categories that emerged from the data. The categories that emerged were about the four specific questions posed to the participants. In the following section, the participants’ responses to each question from the three scenarios are analysed and discussed.
4. Results
In the following subsections, the participants’ responses to the four questions for each scenario are explored. In Section 4.1, participants’ reports about what they noticed in each scenario are presented. Secondly, in Section 4.2, participants’ reports on the similarities and differences they noticed in the scenario when compared to their own culture are discussed. Thirdly, in Section 4.3, participants’ claims about whether the situation was managed appropriately are reported. Lastly, in Section 4.4, participants’ reports about how they would manage similar future situations are explored. These responses are explored in order to answer the first research question.
4.1. Noticing Concepts
Participants responded to the question, “What IC concepts do you notice in the scenario?” Table 2 shows the various concepts participants reported in scenario 1. The majority of participants identified time perception as the key intercultural concept in the scenario. This is the most salient concept as the scenario depicts issues surrounding time orientation, as well as worktime and meeting time expectations. While this is the most salient intercultural concept in the scenario, communication style is also evident. While 20.8% of participants identified cultural values, 18.75% of participants acknowledged issues in communication styles. Specifically, participants referred to appropriate ways in which employers should inform employees as to when meetings are scheduled. While this is discussed further in Section 4.2, two participants responded to this question with the following:
MP02: To let her know about the meeting beforehand or not.
FP17: Not reminding her of the meeting and the Asian woman didn’t provide any information on the meeting.
Lastly, 16.6% of participants gave varied responses to the question. For example, responses included “culture shock”, “misunderstanding each other”, and “the secretary seems very unkind to the woman”. While these concepts (or descriptions) are relevant to the scenario, they are not the focus necessarily (i.e., other concepts are more salient). This issue is discussed further in Section 5.1.
![]()
Table 2. Scenario 1 (Question 1) results.
Similarly, in scenario 2, participants were asked to respond to the same question as in scenario 1. Table 3 demonstrates the various ways in which participants responded. The majority of participants identified communication styles as the most salient concept in the scenario. Furthermore, 22.9% of participants identified context, and 18.75% of participants identified attitudes as the most salient concepts in the scenario. Finally, 12.5% of participants identified a variety of other concepts such as “culture shock” and “ethnocentrism”. In the scenario, the main concepts include appropriate workplace language, context, power distance, and communication styles. While the majority of participants articulate awareness of these concepts, some participants note other concepts that are not the primary focus.
Lastly, in scenario 3, participants responded to the same question as the previous scenarios. Table 4 demonstrates their responses to the scenario. The intercultural concepts in scenario 3 include cultural differences (behaviours and expectations), non-verbal communication, and the concept of self. As shown in Table 4, participants were able to articulate awareness about several concepts found in the scenario. The majority of participants identified various cultural differences that arose in the scenario. Furthermore, 14.6% of participants identified the various communication styles that were discussed in the scenario. However, only 10.4% of participants could identify non-verbal communication as the central concept in the scenario. Lastly, 12.5% of participants responded in various ways, identifying concepts that required further explanation
![]()
Table 3. Scenario 2 (Question 1) Results.
![]()
Table 4. Scenario 3 (Question 1) Results.
such as “stereotypes” and “prejudice”. In the next section, participants’ reports on comparing their cultural backgrounds to the scenarios are discussed.
4.2. Self-Situation Comparison
In this section, participants’ responses to the question, “What similarities and differences do you notice in the video compared to your own linguistic/cultural background?” are reported. In Table 5, the number of participants who reported in each category is shown. Participants’ responses were categorised as whether they could highlight similarities and differences. Some participants offered alternative interpretations while others did not respond.
Overall, the majority of participants across all scenarios identified similarities to their own cultural and linguistic backgrounds. That is, they found similar concepts that applied to their cultural milieu. A substantial number of participants also reported in other ways such as attempting to analyse the scenarios further or explicitly relating them to their experiences, which are shown in the following three examples:
Scenario 1:
FP12: race and ethnicity are different.
Scenario 2:
MP04: participation at meetings.
Scenario 3:
FP39: she never accept[sic]the advice made by the woman with grey suit.
The responses above show that participants do not accurately respond to the question all the time but do provide alternative insights and analysis about the scenario and how they perceive it. In the next section, the participants’ responses about the way in which the situation was managed are reported.
4.3. Appropriateness
Participants were asked to consider whether the critical incidents were appropriately managed. Table 6 shows participants’ responses across the three scenarios. These responses were categorised as either affirmative, negative, other response, or no response. The question also prompted students to consider what linguistic and non-verbal behaviours helped them reach this decision which is discussed later in this section.
Table 6 shows that in scenarios 1 and 3, the majority of participants reported
![]()
Table 5. All scenarios (Question 2) results.
![]()
Table 6. All Scenarios (Question 3) results.
that the situations were not managed appropriately. For example, the following examples outline participants’ claims from scenarios 1 and 3:
Scenario 1:
MP23: I think she didn’t handle with it since she complains about the difference of time management with her home country and seems she already had the same things before.
Scenario 3:
MP10: No, she did not enough try to accept or adapt to other student’s cultural things.
Alternatively, in scenario 2, most participants reported in other ways. For example, two participants reported the following:
FP06: I’m not sure but if he wanted to work there he has to do things employers need.
MP27: [He] should have accepted the different culture.
Closely following the other category in scenario 2, 33.3% of participants noted that the situation was not handled appropriately. For example, participants reported the following:
MP28: No he didn’t. He won’t change his attitudes.
MP08: Not really because the boss didn’t look satisfied with his answers. Maybe he could have used more gestures to explain what he feels if he couldn’t tell well verbally.
The reports show that participants were able to articulate awareness of whether such incidents were effectively managed.
In this question, participants were also asked to reflect on the verbal and non-verbal behaviours featured in the video. Although participants were able to articulate responses about how the incidents were managed, only 10 participants (20.8%) in scenario 1, and six participants (12.5%) in scenarios 2 and 3 were able to report on specific verbal and non-verbal behaviours in their responses. For example, a female participant in scenario 1 reported that “Both ladies seemed confused. Facial expression[s] helped me”. That is, by examining the facial expressions of both interlocutors in the video, she was able to determine that the situation had not been managed correctly and confused body language confirmed her claims. Other responses focused on:
· Non-verbal behaviours
· Communication styles
· Paralanguage (e.g., intonation)
· Verbal behaviours
In the next section, participants’ reports on the way in which they would manage the situation in the future are analyzed.
4.4. Future Management
In this section, participants’ responses to the question: “How would you handle this type of situation?” are explored. Tables 7-9 demonstrate the variety of strategies they would use if they were to encounter a similar problem in the future.
![]()
Table 7. Scenario 1 (Question 4) results.
![]()
Table 8. Scenario 2 (Question 4) results.
![]()
Table 9. Scenario 3 (Question 4) results.
In Table 7, the majority of participants reported that they would use various strategies to manage a similar situation in the future. Participants noted that they would apologise or use other redressive actions. For example, one male participant also reported:
MP43: First apologise but explain that it was different from my culture so next time I can adjust to it.
Other participants (31.25%) suggested that they would learn and develop specific knowledge in order to avoid such situations in the future. For example, three participants reported:
MP23: Change my mind and adapt to the culture of country I stay [sic].
FP09: Understand other people’s habits.
FP06: I’ll study like that before the trouble.
Some participants (16.6%) reported that they would speak directly to their superior to sort out the problem. For example, three participants reported:
MP40: Talk to my boss directly.
MP35: I’m going to talk to the boss. Learn from my mistakes. Adjust.
FP15: I would probably just send an email to my boss or ask my colleagues what the meetings were about, or set a reminder.
A small number of participants (12.5%) reported in various ways, such as explaining the situation from their perspective, what they expect as an employer, or offering advice to management. Three participants noted that:
MP20: The boss should tell the time.
MP29: I think she should have checked the schedule herself.
FP48: Talk on the day before the meeting.
The reports in this scenario, while varied, do highlight the ways in which these participants would manage such situations in the future. In Table 8, participants’ responses about how they would manage a situation where they are asked to speak up in the workplace are explored. The majority of participants reported that they would adapt to their new work environment by developing their knowledge and skills. Participants also claimed that they would behave according to new expectations. This included responses such as additional study and demonstrating such skills at appropriate times. For example, one participant noted:
FP06: I’ll practice a lot to show feeling or reaction to tell people what I’m thinking.
Moreover, 12.5% of participants responded that they would seek help by asking their colleagues about the appropriate ways to behave in the workplace. Furthermore, participants also noted that they would reflect on their behaviours and attempt to improve. These responses are exemplified below:
MP02: I would ask people what are the differences.
MF15: I would probably ask them the same thing, ask them to point me [sic] if they want me to speak up.
MP23: I would try myself and also ask help to understand about my habits to others.
Similarly, 10.5% of participants reported that they would address the issue directly with their superiors and offered some strategies to improve the situation. For example, one participant noted:
FP24: I will explain why I cannot say anything in the meeting. Then, I will suggest to my boss to wait for me being able to get used to the culture and language for several months, and ask him to invite some teacher/professor to give other colleagues a talk about intercultural communication just as many company [sic] would have kind of the one to avoid sexual harassment.
In Table 9, the participants’ reports about how they would deal with intercultural misunderstandings in the workplace are shown. The majority of participants in scenario 3 reported that they would directly address the issue with their colleagues. Participants noted that they would confront the person; however, they would be respectful and considerate, as shown in the examples below:
MP04: Talk to him and have a sincere communication [sic], only accepting is not the way to interact [with] others, but speaking up is necessary sometimes.
FP12: Talk to him face-to-face and using [sic] little humour and say them our culture.
MP46: I would kindly ask the person to not take certain actions that made me uncomfortable, but remind the person that other interactions are positive.
Furthermore, 12.5% of participants also noted that they would seek help and advice from other people in dealing with such an issue. This involved talking with people from the same culture as well as their colleagues. For example, participants reported:
FP06: I’ll talk with Chinese people…
MP02: I would talk about it and receive some advice…
While some participants reported on how they would handle the situation, 10.5% of participants reported in more varied ways, including those that expressed uncertainty. Further examples are shown below:
FP19: I might when I speak to a person who has higher status. This happen [sic] in Japan a lot.
FP45: Tell the manner of Japan.
Lastly, 8.5% of participants reported that they would attempt to understand their colleague’s behaviour. However, most of these responses also included statements that suggested that they would also let the person know that they did not appreciate their behaviour. These examples are shown below:
MP10: I want to try to understand other culture and accept it as much as possible. Then, if I could not do that, I would like to talk with them about it until we are both convinced.
FP21: I think I would say the way I feel to the person, and try to understand each other.
MP18: I would like to respect the other’s culture and try to fill in the gap.
This section has highlighted the various ways in which participants may manage these types of situations in the future. The responses show that the majority of participants have particular strategies that align with intercultural communicative competence (e.g., show understanding, adapt) and their own cultural behaviours (e.g., confront, seek advice). While the participants offer varied responses, they do show the developmental nature of intercultural competence, which is explored further in the next section.
5. Discussion
To answer the second research question, in this section, I discuss the quality of the participants’ reports and the efficacy of critical incident videos in determining intercultural competence.
5.1. Quality of Participants’ Reports
The results in the above section show that participants are able to articulate knowledge and awareness of intercultural concepts in various ways. Furthermore, participants show that they can observe and identify issues in the intercultural workplace critical incidents. However, participants’ responses differed in various ways across each question. Firstly, participants across the cohort identified different IC concepts that are not necessarily consistent with the targeted concepts in the scenarios. For example:
Scenario 1:
Targeted concepts: time orientation, directness and face.
Participants identified: perception of time (33), cultural values (10), communication styles (9), misunderstanding (3).
This does not necessarily mean that participants who noticed and then reported different concepts are incorrect. Participants demonstrate a critical analysis of these scenarios. However, this may have implications for how the assessment might be graded or how their responses are rated.
Secondly, the range of responses varied from detailed to specific to limited. For example, from scenario 1:
MP04: Severe about time.
MP10: In my country, we are supposed to remind the time when we do something before that. But, in US, it was not common, so I sometime confused about it as the women in the video.
This could be a result of issues such as the inadequate predictive validity of the task (Griffith et al., 2016), or lack of training to be able to analyse such critical incidents.
Thirdly, participants were able to demonstrate competence in all three areas of Deardorff’s (2006) model of intercultural competence: knowledge, skills, and attitudes. For example, Table 10 outlines participant MP07’s reports across the four questions:
![]()
Table 10. MP07’s responses across four questions.
Overall, MP07 was able to identify concepts, demonstrate observation skills, relate to the situation, and reflect on their own cultural behaviours, and offer a potential future strategy to deal with such issues. However, as shown in Table 11, FP48 was unable to accurately report on all areas of intercultural competence across each question. Specifically, while FP48 was able to articulate knowledge about the intercultural concepts, relate to the situation, and provide a future strategy, they were unable to provide accurate or concrete details for question 3. In MP07’s report, they clearly state that the situation was not managed appropriately and then provide a reason why they came to that conclusion (e.g., “Another one looks and sounds angry”). As directed, the participant was asked to support their reasons by observing and commenting on any verbal and non-verbal behaviours in the video. Contrarily, this is not present in FP48’s report.
The reports in this study show that participants’ intercultural competency is a developmental process with no end point (Deardorff, 2009). More specifically, the variety and depth of responses highlight participants’ various stages of learning and intercultural competency development. Furthermore, the responses also indicate the complex nature of accounting for intercultural competency. Participants may have noticed various elements of intercultural communication in the scenarios during this “test” but may perform differently on a different day (Griffith et al., 2016), which poses questions about the instrument, which is further discussed in the next section.
5.2. Efficacy of Situational Judgement Tests
Overall, the situational judgement tests used in this study were easy to implement and engaged students using technology. Participants were able to access this material from home and respond to the questions in their own time (Griffith et al., 2016). Furthermore, from a researcher’s perspective, the tests were able to elicit a range of responses addressing elements of intercultural competence (e.g., knowledge, skills, attitudes). However, besides the variability of the responses, two additional issues need to be considered when implementing this assessment tool in classrooms.
![]()
Table 11. FP48’s responses across four questions.
Firstly, from a teacher’s perspective, judging and grading the responses consistently is difficult. Due to the variability of the responses, additional rubrics and raters are required to judge the accuracy of the responses (Griffith et al., 2016). This form of assessment may be time-consuming for the teacher/facilitator to grade. It may not be practical in classroom settings to use as an effective tool for measuring intercultural competence (Sercu, 2004; Griffith et al., 2016). Overall, it may be difficult to judge a student’s competency or intercultural development accurately, consistently, and equitably.
Secondly, while there are some limitations in the evaluation of responses, the tests do provide students with a chance to engage with intercultural workplace critical incidents that they may experience in the future. Perhaps the use of critical workplace incident videos is better used as an activity in a classroom setting, rather than a “test-like” assessment, leading to more context-sensitive curricula (Allen, 2021). Students may also benefit from undertaking some discourse analytic training before completing the activity, which may result in more consistent and considered responses. Furthermore, by implementing these as experiential activities (Jackson, 2011; Yang, 2017), students can experience authentic examples of issues they may face working in a multicultural environment (Gilmore, 2007; Kozhevnikova, 2014). Previous research has shown that authentic materials increase motivation (Zhang, 2020). These kinds of activities showcase various forms of diversity which are difficult to deliver in homogenous classrooms in Japan (i.e., (re) experiencing situations that they typically do not face) (Tsuneyoshi, 2018; Allen, 2021), and thus is a positive element of experiential intercultural learning.
6. Conclusion
In this study, I have explored how students articulate awareness, knowledge, and reflection of the various IC concepts through their responses to critical incidents in the workplace. Furthermore, analysis and discussion of the results show how such an activity can be used to interrogate students’ intercultural competencies in the areas of knowledge, skills, and attitudes.
While I have shown how situational judgement tests are implemented in the classroom to assess students’ intercultural competency, there are some limitations in the study. Firstly, the number of participants in the current study only offers a limited view of their intercultural competency. Secondly, as participants undertook this survey outside of their course assessment, they may not have fully demonstrated their intercultural competency. In other words, it was a low-stakes activity. If a grade was attached to the task, participants may have provided more in-depth responses. Furthermore, the activity should be accompanied by specific grading requirements, which would encourage students to answer questions more fully, and possibly yield more insightful commentary. Future studies should also incorporate a greater number of participants (from various cultural and linguistic contexts) to provide researchers and teachers with greater insights into the effectiveness of such instruments to determine learners’ intercultural competence.
Acknowledgements
The study was supported by Grants-in-Aid for Early Career Researchers (19K13307). I wish to thank my participants for contributing to this study.
NOTES
1See Allen (2021) for a full review of the study context.