Development of Pictorial Situational Judgement Test of Affect

Abstract

Personality and emotion theories conceptualize the affect which refers to differences in how individual response to subjective aspect of emotional and motivational states as well as the processes. Prior researches have explored the dynamics of momentary emotional experiences by situational process; and the enduring dispositional differences in study of traits. The current study was aimed at developing Pictorial Situational Judgement Test (P-SJT), based on a semi-projective approach for assessment of affect. Initially a pool of 100 items covering positive affect and negative affect dimension was developed based on critical incidents obtained from male and female college-going students of India. The 60 item P-SJT was evaluated with content validity. The article emphasizes the developmental procedure of pictorial situational judgment test of affect and the P-SJT should be useful for the researchers interested in investigating individual differences in identifying emotional states and traits. Limitations and directions for future research are also addressed.

Share and Cite:

Sharma, N. & Nagle, Y. (2015). Development of Pictorial Situational Judgement Test of Affect. Psychology, 6, 400-408. doi: 10.4236/psych.2015.64037.

1. Introduction

The concept of affect includes emotion, motivation, moods and feelings. Motivation and emotion originated from the word “moving experience”. Emotion is a personal and subjective internal feeling whereas motivation is the internal drive to attain a specific goal. The whole emotional experience which is sustained for a specific period of time is referred to as affective state. According to the dictionary of Psychology, the term “Affect” in a broad sense refers to the feeling quality of experience, i.e. “affection” as distinct from cognition and conation. More specifically the term is used as equivalent to emotion and even more narrowly to refer to the subjective aspect of emotional states, in contrast to their observable signs.

Batson, Shaw and Oleson (1992) and Watson (2000) used the concept of affect to denote positive and negative feeling state that is less intense and more obscure than emotions and moods. The PEN Model of Eysenck (1990) gave Psychoticism, Extraversion and Neuroticism as the fundamental traits of personality. The Five Factor Model by Costa and McCrae (1992) focused on Neuroticism, Extraversion, Openness to Experience, Agreeableness and Conscientiousness. The Behavioural Approach System (BAS) and the Behavioural Inhibition System (BIS) connects emotion with neurobiological system. This is based on Gray’s (1972) theory of personality which suggests that Extraversion and Neuroticism results from two neurological systems (BAS and BIS) which respond to cues of reward and punishment respectively. Individual’s experience positive affect when BAS is activated (Gray, 1994) and BIS is responsible for inhibition, withdrawal behaviour and negative affects like anxiety (Gray, 1994) . The Temperamental Theory of Personality by Cloninger (1986, 1991) described about the positive and negative affect. The dimension of this Biosocial Model is Novelty Seeking and Harm Avoidance (Cloninger, 1986, 1991) . The dimension of Novelty Seeking closely relates to Extraversion and the dimension of Harm Avoidance relates to Neuroticism. Watson & Tellegen (1985) have proposed two independent behavioural systems, known as positive affectivity and negative affectivity (Tellegen, 1985; Watson, 2000; Watson & Tellegen, 1985) . According to this theory, general positive affect and general negative affect are not subsumed as components of broader factors or superfactors but rather themselves are fundamental dimensions of personality (Tellegen, 1985; Watson & Clark, 1984) . Watson (2000) makes a strong case for positive and negative affect as the central cores of Extraversion and Neuroticism. Thus, many theories assume that positive and negative affect are fundamental components of personality structure. It is evident from the review of various theories of personality that Positive Affectivity correlates positively with Extraversion and negatively with Neuroticism and Negative Affectivity correlates positively with Neuroticism and negatively with Extraversion.

Situational Judgment Tests (SJTs) are a “measurement method that can be used to assess a variety of constructs” (McDaniel et al., 2001; McDaniel & Nguyen, 2001) and have been extensively researched for their utility in personnel selection (McDaniel, Morgeson, Finnegan, Campion, & Braverman, 2001) . Researches on SJTs indicate that they are effective and are frequently used selection tools both in the U.S. and Europe (McDaniel et al., 2001) . Motowidlo and his colleagues (Motowidlo et al., 1990; Motowidlo, Hooper, & Jackson, 2006) stated that SJTs emanate from the tenet of behavioral consistency (i.e., that past behavior is the best predictor of future behavior). The respondents are provided with situations to elicit current behaviour in the form of choosing alternatives one can predict the way the individual is likely to behave in the future (Wernimont & Campbell, 1968) . The Implicit Trait Policy given by Motowidlo et al. (2006) describes that there are inherent beliefs about causal relationships between personality traits and behavioural effectiveness. Motowidlo et al. (2006) argued that individual differences in personality traits affect judgment of the effectiveness of behavioral episodes that are manifested in personality traits. Thus there are individual differences in the selection of alternatives which are related to the personality trait of the individual.

SJT formats are varied in nature; some of which are paper-and-pencil tests with written descriptions of situations (Chan & Schmitt, 2002) and others use computerized multimedia scenarios (McHenry & Schmitt, 1994; Olson-Buchanan et al., 1998; Weekley & Jones, 1997) , and computer-based videos (Chan & Schmitt, 1997; Weekley & Jones, 1997) . Mostly, the SJT items are presented in paper and pencil format, but video-based SJTs presuppose less reading skills in comparison to paper-pencil based procedures; therefore show a greater fairness with regard to minorities with lower linguistic abilities (Chan & Schmidt, 1997) . Using videos, a larger amount of information can be presented in the same time span; this gives a fairer knowledge about the individual’s behaviour. In the video presentation the respondent receives visual as well as auditory information (Paivio, 1986) . It is often argued that the simultaneous processing of visual and auditory information might be disadvantageous, unless the respondent is able to watch the film over and over again at his/her own discretion. The individual has to be very attentive in order to be able to grasp in all of the relevant information. Paper pencil texts enable the individual a slow processing of the information as one can read the text very slowly or also several times in succession. It has also been argued that watching videos is easier and less stressful for a respondent than reading a text and this is particularly true for persons who have difficulties in reading (Chan & Schmitt, 1997) . Furthermore, the use of video technology as a stimulus leads to a higher fidelity of the simulation (Motowidlo et al., 1990) . The video scenarios give a more realistic view about the situations, thus make it easier for the respondent to imagine that he or she is actually part of the situation. A study by Richman-Hirsch, Olson-Buchanan, and Drasgow (2000) showed that multimedia-based tests are rated more positively than paper and pencil tests only if the technical possibilities offered by a computer are actually fully used (Chan & Schmitt, 1997; Schmitt & Gilliland, 1993) .

SJT situations can be described in terms of three variables: Stimulus component, Response component and Interactivity. Each situation consists of a stimulus and a response component (Motowidlo, Dunette, & Carter, 1990) . When the situation is presented the respondent has to deal with situational judgment test situation, by taking into consideration the focal stimulus of the situation. Response component is the way in which the response options of a situation are obtained. Interactivity of a diagnostic procedure relates to the possibility of giving direct reactions to a participant’s responses. Interactive SJTs take into account a respondent’s previous answers, i.e., a respondent’s decision in favour of previous item changes the way in which further situations are presented to the respondent. In general, SJTs are not interactive.

SJT Response Options are of varied formats. Some SJTs propose solutions to problems, to which respondents rate their agreement (Chan & Schmitt, 2002) . Others offer multiple solutions from which respondents choose the best and/or worst option (Motowidlo et al., 1990; Olson-Buchanan et al., 1998) . Respondents either have to choose from a number of alternatives the kind of behaviour he or she would most prefer, or one has to rate all the given alternatives (McDaniel & Nguyen, 2001) .

Accordingly, there are two types of Response Instructions for SJTs―Knowledge Instructions and behavioural Tendency Instructions. Knowledge Instructions ask respondents to display their knowledge of the effectiveness of behavioural responses. The respondent is asked to display their knowledge about the effectiveness of the response alternatives, by marking either the best or the worst response option or by rating the response options. Behavioural Tendency Instructions ask respondents to report how they typically respond by asking them “What would you do?/What would you most likely do? or rate each response on likelihood that you would do the behavior”. Traditional SJTs are composed of situations that describe a problematic work scenario wherein the respondent has to consider several behavioral responses to the situations provided. Performance on SJTs is related to both personality traits and job performance. In a meta-analysis, McDaniel, Hartman, Whetzel, and Grubb (2007) found that job knowledge measured by SJTs was a valid predictor of job performance and personality traits were positively related to knowledge measured by SJTs.

2. Development of Pictorial SJT

This section reports the three phases of development of Pictorial Situational Judgement Test (P-SJT). The first phase involved identification of constructs and initial item development, second phase involved selection of items for final form through panel of experts and development of scoring key, third phase involved administering the test on a sample of undergraduates.

2.1. Phase I

Affect was operationally defined in terms of two dimensions of Neuroticism and Extraversion, each of these include six constructs, viz. Anxiety, Angry Hostility, Depression, Self Consciousness, Impulsiveness and Vulnerability; and Warmth, Gregariousness, Assertiveness, Activity, Excitement Seeking and Positive Emotions.

For item construction, themes were developed under each of the construct, through inductive (to arrive at a general principle from a specific one) and deductive approach (to arrive at a specific principle from a general one). Inductive approach was followed in discussions and brainstorming sessions by nine subject matter experts (psychologists and academicians). Deductive approach was followed in collecting critical incidents from the target population of 120 college going students with age range of 17 - 20 years, for each facet of Extraversion and Neuroticism. The target population of 120 students were asked to recall incidents in their day to day life which correspond to the twelve identified constructs i.e. anxiety, angry hostility, depression, self consciousness, impulsiveness, vulnerability, warmth, gregariousness, assertiveness, activity, excitement seeking and positive emotions. The students were also asked to write their responses as to “how they responded to that incident” and also to give three alternate responses as to “how they might have responded to the incident”. Themes were generated from these critical incidents which were clustered under respective constructs. These themes were later taken into consideration for development of the item-stem i.e. situations. The main and alternate responses given by the students were used to develop the response alternatives for each situation. Achromatic pictures which portray the situation were developed for each test-item by a commercial artist under the guidance of the researcher. A dialogue box was also included in the pictures to help the subject identify the focal figure and to trigger the required affect in the student (Figure 1). It also gives clarity to the thought process of the subject to

Figure 1. Example of pictorial SJT item (Sharma, Nagle, & Gupta, 2015) .

specific construct (Sharma, Nagle, & Gupta, 2015) .

Response alternatives are the behavioural manifestations of the way the respondent would react if placed in a similar situation. Four response alternatives were developed for each item. The response alternatives are made keeping in mind the social desirability factor. All the response alternatives have equal probability of getting selected by the subject. Initially 100 items were developed which include a situation, an achromatic picture and four response alternatives. The items are presented in a linear fashion, one after the other through a slide show. There is a constant time interval between the presentations of the test-items. The nature of test-items is such that it caters to frequency and intensity of the occurrence of a particular emotion. It is hypothesized that affective pre-disposition of a respondent is assessed by presenting a pictorial stimulus and capturing his response in the form of varying intensities of response alternatives (Sharma, Nagle, & Gupta, 2015) .

2.2. Phase II

Second phase of the study comprises of selection of items for final form and development of the scoring key. Items were given to a panel of fifteen subject matter experts (SMEs) including scientists, practicing and amateur psychologists, counsellors and academicians. The concordance of the situation with operational definition of constructs, picture with situation; and intensity of response alternatives was ensured. The criterion for final selection of items was 75% agreement among the panel of experts.

Observations made by them were incorporated in the test-items and accordingly eighteen pictures were modified and five pictures were re-made. The mean score of the ratings given by panel of experts was determined. The items with a mean ≥ 3.5 were retained (Table 1). Thus out of 100 items 60 items were retained in the final form of test (Table 2).

The scoring method of the test was developed based on the intensities given by the panel of experts on each of the four response alternatives for all the 60 items (30 items each from Neuroticism (N) and Extroversion (E) scale). The average intensities were calculated for all the SMEs and reverse scoring was followed for Neuroticism and Extroversion scale. Two best and two worst responses were decided for each item. Two best responses were given a score of “1” and two worst responses were given a score of “0”. The raw scores of both the N scale and E scale of the test were converted to Quotient (Q ? converted score). For each N scale and E scale, two scores are obtained in form of best response and worst response. The formula to calculate Quotient: total best responses ? total worst responses/total best responses + total worst responses.

Table 1. Mean and SD of items.

Table 2. Final retained items.

2.3. Phase III

Third phase of the study comprises of preliminary try-out for assessment of affect on a sample of 250 undergraduates (age range 17 - 20 years) from various colleges of Delhi, India. Test was conducted in two sessions. Session I included an open-ended response format and Session II included forced-choice response format in order to minimize socially desirable responses. The responses given by the students in open-ended format were content analyzed by 9 Subject Matter Experts. As the pattern emerged was found to be highly socially desirable in nature so this open-ended response format was rejected and finally forced-choice response format was retained. In order to check the fidelity of the response alternatives, frequency of ratings of all the four response alternatives across the sample of 250 students was plotted on a normal probability curve. Each response alternative has a probability of getting picked by the students, though it was not equal for all the response alternatives (Sharma, Nagle, & Gupta, 2015) .

3. Conclusion

An attempt was made to develop semi-projective method for assessment of affect based on an innovative approach of pictorial SJT. The present article emphasizes only the developmental procedures of the test. Finally 60 items were developed and tried out on a small sample to find out the feasibility for assessment of affect. This technique seems to be sound enough to assess certain personality attributes. However, there are some limitations to the present research. This study is based on a sample of 250 college-going students. It needs further validation on a larger population. This test is meant only for male population. The test can be validated on a larger sample and can be used for employees’ selection. As the test is pictorial in nature, it can be effectively used for identification of affect and intervention at school and college set-up.

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1] Batson, C. D., Shaw, L. L., & Oleson, K. C. (1992). Differentiating Affect, Mood, and Emotion: Toward Functionally-Based Conceptual Distinctions. In M. S. Clark (Ed.), Review of Personality and Social Psychology (Vol. 13, pp. 294-326). Newbury Park. CA: Sage.
[2] Chan, D., & Schmitt, N. (1997). Video-Based versus Paper-and-Pencil Method of Assessment in Situational Judgment Tests: Subgroup Differences in Test Performance and Face Validity Perceptions. Journal of Applied Psychology, 82, 143-159.
http://dx.doi.org/10.1037/0021-9010.82.1.143
[3] Chan, D., & Schmitt, N. (2002). Situational Judgment and Job Performance. Human Performance, 15, 233-254.
http://dx.doi.org/10.1207/S15327043HUP1503_01
[4] Cloninger, C. R. (1986). A Unified Biosocial Theory of Personality and Its Role in the Development of Anxiety States. Psychiatric Development, 3, 167-226.
[5] Cloninger, C. R. (1991). Brain Networks Underlying Personality Development. In B. J. Carroll, & J. E. Barrett (Eds.), Psychopathology and the Brain (pp. 183-208). New York: Raven Press.
[6] Costa, P. T., & McCrae, R. R. (1992). Normal Personality Assessment in Clinical Practice: The NEO Personality Inventory. Psychological Assessment, 4, 5-13.
http://dx.doi.org/10.1037/1040-3590.4.1.5
[7] Eysenck, H. J. (1990). Biological Dimensions of Personality. In L. A. Pervin (Ed.), Handbook of Personality (pp. 224-276). New York: Guilford Press.
[8] Gray, J. A. (1972). The Psychophysiological Basis of Introversion-Extraversion: A Modification of Eysenck’s Theory. In V. D. Mcbylitsyn, & J. A. Gray (Eds.), The Biological Basis of Individual Behavior (pp. 182-205). New York: Academic Press.
[9] Gray, J. A. (1994). Personality Dimensions and Emotion Systems. In P. Ekman, & R. J. Davidson (Eds.), The Nature of Emotion (pp. 329-331). New York: Oxford University Press.
[10] McDaniel, M. A., Morgeson, F. P., Finnegan, E. B., Campion, M. A., & Braverman, E. P. (2001). Predicting Job Performance Using Situational Judgment Tests: A Clarification of the Literature. Journal of Applied Psychology, 86, 730-740.
http://dx.doi.org/10.1037/0021-9010.86.4.730
[11] McDaniel, M. A., & Nguyen, N. T. (2001). Situational Judgment Tests: A Review of Practice and Constructs Assessed. International Journal of Selection and Assessment, 9, 103-113.
http://dx.doi.org/10.1111/1468-2389.00167
[12] McDaniel, M. A., Hartman, N. S., Whetzel, D. L., & Grubb III, W. L. (2007). Situational Judgment Tests, Response Instructions and Validity: A Meta-Analysis. Personnel Psychology, 60, 63-91.
http://dx.doi.org/10.1111/j.1744-6570.2007.00065.x
[13] McHenry, J. J., & Schmitt, N. (1994). Multimedia Testing. In M. J. Rumsey, C. D. Walker, & J. Harris (Eds.), Personnel Selection and Classification Research (pp. 193-232). Mahwah, NJ: Lawrence Erlbaum Publishers.
[14] Motowidlo, S. J., Dunnette, M. D., & Carter, G. W. (1990). An Alternative Selection Procedure: The Low-Fidelity Simulation. Journal of Applied Psychology, 75, 640-647.
http://dx.doi.org/10.1037/0021-9010.75.6.640
[15] Motowidlo, S. J., Hooper, A. C., & Jackson, H. L. (2006). Implicit Policies about Relations between Personality Traits and Behavioral Effectiveness in Situational Judgment Items. Journal of Applied Psychology, 91, 749-761.
http://dx.doi.org/10.1037/0021-9010.91.4.749
[16] Olson-Buchanan, J. B., Drasgow, F., Moberg, P. J., Mead, A. D., Keenan, P. A., & Donovan, M. A. (1998). Interactive Video Assessment of Conflict Resolution Skills. Personnel Psychology, 51, 1-24.
http://dx.doi.org/10.1111/j.1744-6570.1998.tb00714.x
[17] Paivio, A. (1986). Mental Representations: A Dual Coding Approach. New York: Oxford University Press.
[18] Richman-Hirsch, W. L., Olson-Buchanan, J. B., & Drasgow, F. (2000). Examining the Impact of Administration Medium on Examinee Perceptions and Attitudes. Journal of Applied Psychology, 85, 880-887.
http://dx.doi.org/10.1037/0021-9010.85.6.880
[19] Schmitt, N., & Gilliland, S. W. (1993). Computer-Based Testing Applied to Selection of Secretarial Applicants. Personnel Psychology, 46, 149-165.
http://dx.doi.org/10.1111/j.1744-6570.1993.tb00871.x
[20] Sharma, N., Nagle, Y. K., & Gupta, A. (2015). Pictorial SJT: A Technique of Personality Assessment for Special Task. Proceedings of the 41st National Annual Conference of the Indian Association of Clinical Psychologists, Gujarat.
[21] Tellegen, A. (1985). Structures of Mood and Personality and Their Relevance to Assessing Anxiety, with an Emphasis on Self-Report. In A. H. Tuma, & J. D. Master (Eds.), Anxiety and the Anxiety Disorders (pp. 681-706). Hillsdale, NJ: Erlbaum.
[22] Watson, D., & Clark, L. A. (1984). Negative Affectivity: The Disposition to Experience Aversive Emotional States. Psychological Bulletin, 96, 465-490.
http://dx.doi.org/10.1037/0033-2909.96.3.465
[23] Watson, D., & Tellengen, A. (1985). Towards a Consensual Structure of Mood. Psychological Bulletin, 98, 219-235.
http://dx.doi.org/10.1037/0033-2909.98.2.219
[24] Watson, D. (2000). Mood and Temperament. New York: Guilford Press.
[25] Weekley, J. A., & Jones, C. (1997). Video-Based Situational Testing. Personnel Psychology, 50, 25-49.
http://dx.doi.org/10.1111/j.1744-6570.1997.tb00899.x
[26] Wernimont, P. F., & Campbell, J. P. (1968). Signs, Samples, and Criteria. Journal of Applied Psychology, 52, 372-376.
http://dx.doi.org/10.1037/h0026244

Copyright © 2024 by authors and Scientific Research Publishing Inc.

Creative Commons License

This work and the related PDF file are licensed under a Creative Commons Attribution 4.0 International License.