1. Introduction
In a world in which the kinds of things that are easy to teach and test have also become easy to digitise and automate, the capacity of individuals to imagine, to create and to build things of intrinsic positive worth is rising in importance. But this has not automatically led to corresponding changes in intended, implemented and achieved curricula. Too much of what happens in today’s classroom is geared towards having students reproduce what they have learned, rather than extrapolating from it and applying their knowledge creatively to novel situations.
Many observers view creativity or creative thinking as something that we can observe in its impact and consequences, but that in itself is hard to define and assess. Some would go further and argue that assessments, which generally look for the capacity of students to find and refine pre-defined answers, stand in direct opposition to efforts to strengthen creativity. And yet, what we cannot see is hard to improve and what we cannot measure will not get attention.
For this reason, the OECD Programme for International Student Assessment (PISA), which has been conceived as a global yardstick to measure educational success, is making efforts to build a novel assessment that can capture elements of creative thinking. While the jury on the validity and reliability of these metrics is still out—the assessment will go into the field in 2022—this paper summarises the thinking underpinning this assessment. The paper is based on the assessment framework for PISA 2022.
2. The Case for Assessing Creative Thinking
Why assess creative thinking?
Creative insights and advances have driven forward human culture across the world in diverse areas (Hennessey and Amabile, 2010): in the sciences, technology, philosophy, the arts and humanities. Creative thinking is thus more than simply coming up with random ideas. It should be defined as a tangible competence, grounded in knowledge and practice, that supports individuals in achieving better outcomes, often in constrained and challenging environments. Organisations and societies around the world increasingly depend on innovation and creative skills to address emerging challenges (OECD, 2010), giving urgency to innovation and creative thinking as collective enterprises.
While it is true that creative thinking drives the types of innovation that have a society-wide impact, it is also a more universal and democratic phenomenon than one might first believe. That is to say that every individual, to a greater or smaller degree, has the potential to think creatively (OECD, 2017). Furthermore, there is a general consensus among psychologists and educators alike that creative thinking, understood as engagement in the thinking processes associated with creative work, can improve a host of other individual abilities, including metacognitive capacities, inter- and intra-personal and problem-solving skills, as well as promoting identity development, academic achievement, future career success and social engagement (Beghetto, 2010; Plucker, Beghetto and Dow, 2004; Smith and Smith, 2010; Torrance, 1959; National Advisory Committee on Creative and Cultural Education, 1999; Spencer and Lucas, 2018; Long and Plucker, 2015; Barbot, Lubart and Besançon, 2016; Barbot and Heuser, 2017; Gajda, Karwowski and Beghetto, 2017; Higgins et al., 2005).
Developing an international assessment of creative thinking through PISA seeks to encourage positive changes in education policies and pedagogies.
What is the role of education in creative thinking?
A fundamental role of education is to equip students with the competences they need—and will need—in order to succeed in society. Creative thinking is a necessary competence for today’s young people to develop (Lucas and Spencer, 2017). It can help them adapt to a constantly and rapidly changing world, and one that demands flexible workers equipped with “21st century” skills that go beyond core literacy and numeracy. After all, children today will likely be employed in sectors or roles that do not yet exist, using new technologies to solve novel problems. Educating for creative thinking can help young people to adapt to develop the capacities to undertake work that cannot easily be replicated by machines and address increasingly complex local and global challenges with out-of-the-box solutions.
The importance of nurturing creative thinking in school also extends beyond the labour market. Schools play a crucial role in helping young people to discover, develop and define their talents—including their creative talents. Schools play a vital role in making children feel that they are part of the society they live in, and that they have the creative resources to contribute to its development (Tanggaard, 2018).
Creative thinking can also benefit the way in which students learn by supporting the interpretation of experiences, actions and events in novel and personally meaningful ways (Beghetto and Kaufman, 2007). Student imagination and curiosity can drive the learning process: creative thinking can thus be a vehicle for understanding, even in the context of predetermined learning goals (Beghetto and Plucker, 2006). In order to increase students’ motivation and interest at school, new forms of learning that engage with the creative energies and recognise the creative potential of all students need to be developed. Such development may particularly help those students who show little interest in school, and guide them to express their ideas and achieve their potential (Hwang, 2015).
Just like any other ability, creative thinking can be nurtured through practical and targeted application (Lucas and Spencer, 2017). For some educators, developing students’ creative thinking skills may seem to imply taking time away from other subjects in the curriculum. In reality, students can think creatively in a range of subjects. Creative thinking can be developed while promoting the acquisition of content knowledge and skills through approaches that encourage exploration and discovery rather than rote learning and automation (Beghetto, Baer and Kaufman, 2015). Teachers need to understand how creative thinking can be recognised, the circumstances that encourage it, and how they can effectively guide students to become more creative in their thinking. A greater understanding of how creative thinking unfolds may in turn motivate teachers to allow their students to take time “incubating” creative ideas in their learning processes (Csikszentmihalyi, 1996).
3. Evidence-Centred Design as a General Framework for the PISA 2021 Assessment
Evidence-centred design (ECD) (Mislevy, Steinberg and Almond, 2003) provides a conceptual framework for developing innovative and coherent assessments that are built on evidence-based arguments, connecting what students do, write or create on a computer platform, with multidimensional competences (Shute, Hansen and Almond, 2008; Kim, Almond and Shute, 2016). ECD starts with the basic premise that assessment is a process of reasoning from evidence to evaluate specific claims about students’ capabilities. In essence, students’ responses to the assessment items and tasks provide the evidence for this reasoning process, and psychometric analyses establish the sufficiency of the evidence for evaluating each claim. Using ECD as an organising framework for the PISA 2021 creative thinking assessment can help to address a series of important test design questions, namely: which creative thinking constructs or processes does each task within the assessment reveal? Do the proposed scoring methods effectively recognise and interpret the evidence generated by students’ responses and interactions with the assessment platform? How is all of the evidence that is generated by students’ choices synthesised across multiple tasks, i.e. do the tasks share a common metric? Is all of the evidence for a particular construct comparable when different students attempt the tasks, i.e. are the tasks invariant to students attributes other than their knowledge and skills relative to the task?
ECD provides a strong foundation for the development of a valid assessment of the complex and multidimensional construct of creative thinking. It requires documented, explicit linkages among the test purposes, the assumptions made about the test takers and that the test seeks to measure among them, and the evidence supporting the claims. Adopting the ECD process for the PISA 2021 creative thinking assessment led to the following steps:
1) Domain definition: reviewing the relevant literature and engaging with experts to define the domain of creative thinking in an educational context. This foundational work clarifies the creative thinking competences that policy makers and educators wish to promote, and the types of creative expressions that 15-year-old students can achieve and that can be most meaningfully and feasibly assessed in PISA.
2) Construct definition: describing the precise construct the PISA test will assess and specifying the claims that can be made about what test takers know and can do relative to the construct. In ECD terminology, this step is generally referred to as defining the Competency or Student Model (Shute et al., 2016).
3) Evidence identification: describing the evidence that needs to be generated in the test to support the subsequent assumptions made about test-takers (i.e. the behaviours or performances that demonstrate the skills being assessed, for example what students might select, write or produce, and which constitute evidence for the claims). In ECD, this is referred to as defining the Evidence Model. This step includes providing rules for scoring the tasks and for aggregating scores across tasks that extract the evidence required to support the claims (including process data stored in log files).
4) Task design: identifying, conceptualising and prototyping a set of tasks that provide the desired evidence within the constraints of the PISA assessment. This stage corresponds to the Task Model step in ECD terminology.
5) Test development: assembling the tasks into test formats that support all of the stated assessment claims with sufficient evidence. This corresponds to the Assembly Model step in ECD terminology.
6) Cross-cultural validation: ensuring that all assessment instruments provide reliable and comparable evidence across countries and cultural groups. This step is generally not discussed in ECD approaches, but is clearly important in the context of PISA.
7) Analysis and reporting: developing appropriate, meaningful and easy-to-communicate representations of the assessment results.
Validation and pilot studies can increase the iterative nature of this design cycle: for example, the analysis of validation data can inform choices regarding evidence identification and task design.
The structure of this framework document follows this sequence of evidence-centred design steps. First, creative thinking is outlined, both in general and specifically in an educational context. Then, the elements of the construct and the methods of evidence identification and collection are explicitly set forth. Finally, the framework discusses issues related to validation and reporting.
4. Defining the Assessment Domain
What is creative thinking?
PISA employs a definition of creative thinking that seeks to be relevant to 15-year-old students around the world. Creative thinking in PISA 2021 is defined as the competence to engage productively in the generation, evaluation and improvement of ideas, that can result in original and effective solutions, advances in knowledge and impactful expressions of imagination.
While creative thinking is still an emerging construct, the broader yet intrinsically related construct of creativity has a strong research tradition. Plucker, Beghetto and Dow (2004) define creativity as “the interaction among aptitude, process, and environment by which an individual or group produces a perceptible product that is both novel and useful as defined within a social context”, reflecting its multidimensional and social nature.
Achieving creative outcomes requires the capacity to engage in creative thinking, but it can also demand a wider and more specialised set of attributes and skills, such as intelligence, domain knowledge or artistic talent. For example, the “Big C” creativity that is associated with technology breakthroughs or art masterpieces demands that creative thinking be paired with significant talent, deep expertise and high levels of engagement in a particular area, as well as the recognition from society that the product has value. Conversely however, “little c” or everyday creativity (e.g. creatively arranging family photos in a scrapbook; combining leftovers to make a tasty meal; or finding a creative solution to a complex scheduling problem at work (Kaufman and Beghetto, 2009)) can be achieved by nearly all people capable of engaging in creative thinking.
Overall, the literature agrees that “little c” creativity can be developed through practice and honed through education. The PISA 2021 test of creative thinking thus focuses on tasks related to this “little c” creativity in order to minimise the importance of innate talent for performance and to put a stronger focus on the malleable capacity of individuals to engage in creative thinking. This type of creative thinking can be applied not only to learning contexts that mainly require the expression of one’s inner world, such as creative writing or the arts, but also to other areas where the generation of ideas is functional to the investigation of issues, problems or society-wide concerns.
Domain generality versus domain specificity
A “domain” can be understood as “any specific area of knowledge, such as art literature, history, or astronomy” or “the set of representations that underlie and support thinking in a specific area of knowledge” (Baer, 2011). Researchers have long debated whether creative abilities are domain specific: are creative people creative in everything they do, or only when engaging in specific activities? This debate on the nature of creativity logically extends to creative thinking: is creative thinking in science different to creative thinking in the arts? Are those who can easily generate ideas to explain a scientific phenomenon also good at finding creative solutions to social and political issues?
The first generation of creative thinking tests mainly reflected the notion of domain generality, based on the idea that a set of general attributes influence creative endeavours of all kinds. Confluence approaches of creativity.
“Confluence approaches”, or “componential theories”, describe creative thinking and creativity as multi-dimensional phenomena (Lucas, 2016; Amabile, 1983; Amabile and Pratt, 2016). Componential theory of creativity outlines four necessary components for any individual to produce creative work: domain-relevant skills, creativity-relevant processes, task motivation, and a conducive environment. The model specifies that creative production fundamentally requires some base resources or raw materials (i.e. domain-specific skills, including knowledge and technical skills), a set of processes or skills for combining these base resources in new ways (i.e. creativity-relevant processes, including appropriate cognitive styles such as breaking out of performance scripts and keeping response options open), and a driver in order to do so (i.e. task motivation). It also suggests that a number of environmental factors can serve as either inhibitors or facilitators of creative engagement. These four components include both relatively stable elements and elements that are more amenable to development and social influences.
Sternberg and Lubart’s (1991, 1995) “investment theory of creativity” suggests that six distinct yet interrelated resources are necessary for creativity: intellectual skills (such as synthetic and analytical skills); domain-related knowledge; particular “thinking styles” (such as a preference for thinking in new ways); motivation; specific personality attributes; and an environment that is supportive and rewarding of creative ideas. Sternberg (2006) later elaborated on the importance of the confluence of these resources, explaining that creative endeavours are far more complex than the simple sum of each respective component. Interactions between different components may lead to a variety of outcomes: for example, high levels in many components could multiplicatively enhance creative engagement; in contrast, there may be a minimum threshold for each component below which creative achievements are not possible, irrespective of the presence or the degree of other components.
5. Understanding and Assessing Creative Thinking in the Classroom
Confluence approaches of creativity emphasise the importance of various internal resources for successfully engaging in creative work, as well as the importance of the environment in which creative work takes place. They thus provide a useful schema for the PISA assessment of creative thinking. However, in order to better understand children’s creative thinking, it is necessary to contextualise these approaches in a way that is relevant to students in their everyday school life (Glaveanu et al., 2013; Tanggaard, 2014).
Figure 1 sets out some key points of observation of creative thinking in the classroom, as well as the relationships between the respective elements. This model builds upon the five-dimensional model of creative thinking proposed by PISA’s Creative Thinking Strategic Advisory Expert Group (OECD, 2017).
Schools can influence several dimensions of students’ internal resources (described henceforth as “individual enablers”) for engaging in creative thinking, including: cognitive skills; domain readiness (domain-specific knowledge and
![]()
Figure 1. Enablers and manifestations of creative thinking in the classroom.
experience); openness to new ideas and experiences; willingness to work with others and build upon others’ ideas (collaboration); willingness to persist towards one’s goals in the face of difficulty and beliefs about one’s own ability to be creative (goal orientation and beliefs); and task motivation.
As for the features of students’ social environments that might stimulate or hinder creative thinking (described henceforth as “social enablers”), the classroom culture, the educational approach of schools and wider education systems, and the broader cultural environment all represent distinct social environments for students. Where educators with to see students engage in creativity, they need to provide opportunities for students to experiment and a holding space for open discourse. Experimenting entails risks and risk inevitably lead to mistakes. Where educators do not support students to learn from and with mistakes, they may foster compliance rather than creativity. So the classroom culture can influence the extent to which students value and invest in their own creative abilities, and can provide incentives or obstacles for engaging in creative thinking.
Finally, schools are arenas in which students’ manifestations of creative thinking, either as individuals or as part of a group, can be observed and measured. Creative achievement and progress in the classroom can refer to forms of creative expression (i.e. communicating one’s internal world and imagination through writing, drawing, music or other arts), knowledge creation (i.e. generating knowledge that is new to the group and understanding in a collaborative enquiry process), or creative problem-solving (i.e. finding creative solutions to a variety of problems across domains).
These distinct enablers of creative thinking in the classroom are strongly interconnected. Social enablers are inherently shaped by cultural norms, which in turn affect how students’ individual enablers are developed and honed.
Individual enablers of creative thinking
Cognitive skills
Creative thinking is often described in divergent thinking terms, and most assessments of creative thinking to-date have focused on measuring divergent thinking cognitive processes. However, the literature clearly highlights that convergent thinking cognitive processes, such as analytical and evaluative skills, are also important for creative production (Cropley, 2006; Reiter-Palmon and Robinson, 2009; Tanggard and Glaveanu, 2014). For example, the ability to generate novel and valuable ideas may depend on the prior execution of other activities, such as successfully defining the problem space, or on “late cycle” processing skills, such as evaluating the creative value of several possibilities or successfully assessing the extent to which a potential solution corresponds to the given task constraints (Runco, 1997). Indeed Getzels and Csikszentmihalyi (1976) found that art students’ success in “problem construction” was strongly correlated with measures of the originality and aesthetic value of their resulting paintings, and that these measures were furthermore linked to long-term artistic success.
Schools can promote the use of pedagogies that encourage the development of the cognitive skills and approaches inherent to the creative process (Beghetto and Kaufman, 2010) . For example, Mayer (1989) demonstrated how learning strategies for forming mental representations can lead to improvements in students’ creativity in science, mathematics and computing problems.
Domain readiness
Domain readiness conveys the idea that an individual requires some degree of pre-existing knowledge and experience within a particular domain in order to successfully produce creative work (Baer, 2016) . The assumption is that the more knowledge one possesses and the better one understands the relationships between pieces of information within a domain, the greater the likelihood one has of generating a creative idea (Hatano and Inagaki, 1986; Schwartz, Bransford and Sears, 2005).
Openness to experience and intellect
There is a vast literature dedicated to identifying the personality traits that characterise “creative people”. Empirical studies examining the personality and behaviour of creative individuals typically employ questionnaire instruments and operationalise creativity as a relatively enduring and stable personality trait (Hennessey and Amabile, 2010). These studies have shown that many creative people share a core set of tendencies, but particularly “openness”: both “openness to experience” and “openness to intellect” (although both variants are seen as comprising the larger “openness” factor) (Amabile, 2012; Batey and Furnham, 2006; Feist, 1998; Prabhu, Sutton and Sauser, 2008; Sternberg and Lubart, 1991; Sternberg and Lubart, 1995).
Kaufman et al. (2009) found that openness to experience was the only one of the “Big Five”1 personality dimensions that were significantly and positively correlated with creative achievements across all domains. The study was then repeated with Chinese participants, who recorded similar results (with the exception of creativity in the maths/science domain) (Werner et al., 2014) . McCrae (1987) also found that divergent thinking was consistently associated with openness to experience, but not with the other remaining dimensions of personality. Meta-analyses of studies on creativity and personality have confirmed that openness to experience appears to be a common trait in creative achievers across domains, whereas other personality traits appear to interact with creativity only insofar as they benefit individuals within specific domains of endeavour (for example, “conscientiousness” seems to enhance scientific creativity but detract from performance in the arts) (Batey and Furnham, 2006; Feist, 1998).
More specifically, “openness to experience” refers to an individual’s receptivity to novel ideas, imagination and fantasy (Berzonsky and Sullivan, 1992). It has been suggested that its predictive value for creative achievements across domains is due to its “broad constellation of traits with cognitive (e.g. fantasy, imagination), affective (e.g. curiosity, intrinsic motivation) and behavioural manifestations (e.g. being adventurous, stepping outside of one’s comfort zone, actively trying new things), all of which are related to creativity” (Werner et al., 2014). Several scholars have further emphasised the importance of a sense of curiosity for successfully producing creative work (Feist, 1998; Guastello, 2009; Kashdan and Fincham, 2002).
“Openness to intellect” is a related yet distinct trait that has also been shown to predict creative achievement. This construct refers to cognitive engagement with abstract and semantic information, primarily through reasoning (DeYoung, 2014). In contrast to openness to experience, openness to intellect seems particularly correlated with scientific creativity (Kaufman et al., 2016).
Goal orientation and creative self-beliefs
Persistence, perseverance and creative self-efficacy are all attitudes that have been shown to influence creativity by providing individuals with both a strong sense of goal orientation, and the belief that they can go on to achieve those goals.
Persistence—the act of single-mindedly continuing to invest effort towards one’s goal in spite of difficulty—and perseverance—enduring and overcoming difficulty to achieve one’s goal—are essential for creativity. Cropley (1990) characterised creative individuals by “their willingness to expend effort”, and Torrance (1988) emphasised perseverance as one of the main traits of creative individuals. Amabile (1983) argues that the ability to concentrate effort for long periods and to persevere in the face of frustration is an important component of creative capacity.
Creative self-efficacy refers to the beliefs that individuals have about their own ability to perform a task creatively (Beghetto and Karwowski, 2017). Goal orientation and creative self-beliefs are closely linked: several researchers consider creative self-efficacy essential in determining whether an individual will sustain effort in the face of resistance (i.e. persist) and ultimately succeed (i.e. persevere) in performing tasks creatively (Bandura, 1997) . These beliefs can in turn be influenced by prior performance history, mood and the social environment in which a task is performed (Bandura, 1997; Beghetto, 2006).
Efforts to stimulate creative thinking in the classroom might therefore aim to strengthen students’ beliefs in their creative abilities and their proficiency in self-regulatory attitudes and behaviours (including persistence and perseverance) (Davis and Rimm, 1985).
Collaborative engagement
Contemporary research is increasingly looking beyond creative thinking as a purely individual construct and towards creative thinking as a collective endeavour, for example by examining the actions of teams in generating new knowledge (Thompson and Choi, 2005; Prather, 2010; Grivas and Puccio, 2012; Scardamalia, 2002). This particular understanding of creative thinking posits that creative work is the result of the interaction between an individual and their environment, including other individuals within that environment. Creative thinking and engagement is thus structured as a continuous cycle of “doing” (actions directed at the environment) and “undergoing” (taking in reactions of the environment) (Glaveanu et al., 2013). Through collaborative engagement, teams can provide new answers to complex problems that are beyond the capabilities of any one person (Warhuus et al., 2017).
Task motivation
The role of task motivation as a driver of creative work has been well documented in research, namely in the works of Amabile (1997) , Amabile and Pratt (2016), and Amabile (1983). The basic assumption is that individuals may possess the ideal constellation of components for high creative potential, and yet still not produce creative work if they are not sufficiently motivated to do so.
Motivation to be creative can be both intrinsic and extrinsic in nature. Individuals who experience intrinsic task motivation: find their work meaningful, engage in the task purely for reasons of enjoyment, self-interest or desire to be challenged; and are relatively insensitive to incentives, contingencies or other external pressures. Csikszentmihalyi (1996) proposed that creative work is powerfully facilitated by the related experience of “flow” because, in the state of flow, people “persist… single-mindedly, disregarding hunger, fatigue, and discomfort” (Nakamura and Csikszentmihalyi, 2002) precisely because they are fully engaged in a task for reasons inherent to the work itself. Conversely, extrinsic task motivation refers to the external incentives, goals or pressures that can motivate people to engage in a particular task.
In general, research has emphasised the conducive role of intrinsic task motivation and the detrimental effect of extrinsic task motivation on creative performance (Amabile, 2012; Sternberg, 2006). More recent theories, however, have acknowledged that extrinsic motivators such as pressures (e.g. deadlines) or rewards (e.g. incentives and recognition) can successfully motivate people to be or persist in their creative endeavours (Eisenberger and Shanock, 2003; Amabile and Pratt, 2016).
Social enablers of creative thinking
Cultural norms and expectations
Creative outputs are embedded within social contexts (Baer, 2016; Csikszentmihalyi, 1996), and these social contexts are inherently shaped by cultural norms and expectations. Cultural norms and expectations affect creative thinking as they can influence the skills and cognitive processes that individuals prioritise for development, the emergence of values that shape personality development, and the differences in performance expectations within a given society (Niu and Sternberg, 2003; Wong and Niu, 2013). Cultural norms can also encourage creative thinking in some situations and for some topics, but discourage it for others (Lubart, 1998). Some studies have investigated the effect of cultural differences on measures of national creativity and innovation. In general, they conclude that only variations along the individualism/collectivism axis of cultural difference have reliably demonstrated a significant impact on creative outputs (Rinne, Steel and Fairweather, 2013; Ng, 2003).
Educational approaches
Cultural norms affect educational approaches, in particular the outcomes an education system values for its students and the content it prioritises in the curriculum. These approaches may, in some cases, result in a lack of encouragement or even the active discouragement of certain creative behaviours at school (Wong and Niu, 2013). The investment theory of creativity argues that being creative is in large part a decision that anyone can make yet few actually do because they find the social costs to be too high. Schools therefore play an important role in encouraging students’ creative thinking by increasing the rewards and decreasing the social costs associated with it in the classroom (Sternberg, 2006). For example, it has been argued that the pressures of standardisation and accountability in educational testing systems have reduced the room afforded to students for creative thinking in their school work (DeCoker, 2000). Some researchers have even claimed that narrow educational approaches and assessment methods are at the root of a “creaticide” affecting today’s young people.
Classroom climate
Organisational research has demonstrated the effects of certain features of the working environment on the creativity of workers. Informal feedback, goal setting, positive challenges, teamwork, relative freedom in carrying out tasks, and appropriate recognition and encouragement to develop new ideas are all environmental enablers of creativity (Amabile, 2012; Zhou and Su, 2010). Conversely, harsh criticism of new ideas, emphasis on the status quo, low-risk attitudes among top management, and excessive time pressures are among the environmental factors that can inhibit creativity (Amabile, 2012) . It could be argued that the effects of similar environmental factors could also apply to creative thinking in the classroom.
With regards to schools specifically, Nickerson (2010) provides a list of school practices that can stifle creative thinking: 1) perpetuating the idea that there is only one correct way to do a task and only one correct answer to a question; 2) cultivating attitudes of submission and fear of authority; 3) adhering to lesson plans at all costs; 4) promoting the belief that originality is a rare quality; 5) promoting beliefs in the compartmentalisation of knowledge; 6) discouraging curiosity and inquisitiveness; 7) and above all, never permitting learning and problem solving to be fun.
Teachers are more likely to focus on teaching creatively and developing learner creativity within school and policy environments that encourage innovation (and accept its associated risks) and that allow them to develop and express their own creativity. Teachers thus need to understand the importance of students’ idea diversity, risk taking, and working with peers in order to accomplish difficult tasks. These approaches are all supported by teachers’ beliefs that creative thinking competences are something that can be developed in the classroom, even if this development takes time.
Creative engagement
The creativity of students’ products provides indicators of their capacity to think creatively, particularly in tasks where much of the creative thinking process is “invisible”. Students’ creative products can therefore be useful to determine whether their creative thinking process has been successful (Amabile, 1996; Kaufman and Baer, 2012).
Over the years, an impressive body of literature on the importance and analysis of creative products across a range of domains has emerged. According to accepted definitions within the literature, creative products are both novel and useful as defined within a particular social context. In the context of schools, creative engagement can take distinct “everyday” forms: for example, through expressive activities of writing, drawing, music or other “arts” subjects; the creation of new knowledge and understanding; or the generation of creative solutions to different types of open problems. These forms of creative engagement in the classroom are multi-disciplinary and extend beyond traditional subjects, such as art and science (Beghetto and Kaufman, 2010; Sawyer, 2011).
Creative expression
Creative expression consists of both verbal and non-verbal forms of creative engagement, in instances where individuals communicate their internal world and imagination to others. Verbal expression refers to the use of language, including both written and oral communication. Non-verbal expression includes not only drawing, painting, modelling and musical expression, but also expressive movement and performance, for example dance and drama.
Knowledge creation
Knowledge creation refers to the advancement of knowledge where the emphasis is placed on progress rather than achievement per se, for example by establishing improved conceptual ideas such as better explanations or theories. Knowledge creation is not only reserved for discoveries of historical importance, but can also occur at all levels of society and in all domains. It’s elaborated parallels among the work of scientists, designers and young students in creating knowledge: for example, it can be helpful for all, regardless of domain, to reconstruct knowledge in order to interpret the findings of others and to make sense of existing theories.
Creative problem solving
Closely linked to knowledge creation is creative problem solving. Not all cases of problem solving require creative thinking: creative problem solving is a distinct class of problem solving characterised by novelty, unconventionality, persistence, and difficulty in problem formulation (Newell, Shaw and Simon, 1962). Creative thinking becomes particularly necessary when students are challenged with problems outside of their realm of expertise, and where the techniques with which they are familiar do not work (Nickerson, 1999).
6. Implications for the Design of the PISA 2021 Creative Thinking Assessment
Focus and objectives of the PISA 2021 assessment of creative thinking
PISA 2021 focuses on the creative thinking processes that one can reasonably expect from 15-year-old students. It does not aim to single out exceptionally creative individuals, but rather to describe the extent to which students are capable of thinking creatively when searching for and expressing ideas, and how this capacity is related to teaching approaches, school activities and other features of education systems.
The main objective of PISA is to provide internationally comparable data on students’ creative thinking competence that have clear implications for education policies and pedagogies. The creative thinking processes in question therefore need to be malleable through education; the different enablers of these thinking processes in the classroom context need to be clearly identified and related to performance in the assessment; the content domains covered in the assessment need to be closely related to subjects taught in common compulsory schooling; and the test tasks should resemble real activities in which students engage, both inside and outside of their classroom, so that the test has some predictive validity of creative achievement and progress in school and beyond.
Collecting information on the complex set of enablers of creative thinking in PISA is challenging yet achievable, at least in part. The PISA 2021 creative thinking assessment is composed of two parts: a test and a background questionnaire. The test provides information on the extent to which students are able to mobilise their creative thinking cognitive processes when working on tasks requiring the generation, evaluation and improvement of ideas. The background questionnaires complement this information with data on other enablers of students’ creative thinking, including creative attitudes (openness, goal orientation and beliefs), perceptions of their school environment, and activities they participate in both inside and outside the classroom.
In the assessment, some enablers of creative thinking are better covered than others. For example, while collaborative skills are a key enabler of knowledge creation in the classroom, students’ capacities to engage in collaborative, creative thinking is not directly measured (although several test tasks ask the students to evaluate and improve the work of others) due to the organisational and technical difficulties of making students work together in PISA. Nonetheless, collaboration skills are recognised as an important individual enabler of creative thinking in the classroom in this framework, in the hopes of inspiring future assessments of creative thinking.
Domains of creative thinking included in PISA 2021
The literature suggests that the larger the number of domains included in an assessment of creative thinking, the better the coverage of the construct. However, certain practical and logistical constraints of PISA have had important implications for the possible domains included in the PISA 2021 assessment of creative thinking.
The first relates to the age of test-takers. Given that the PISA target population (15-year-old students) only has a limited amount of knowledge and experience in many domains, those selected as assessment domains need to be based on the knowledge and experiences that are common to most students around the world (such as drawing, writing or problem-solving). The assessment domains (and related tasks) must also be reflective of the realistic manifestations of creative thinking that 15-year-old can realise in this context.
A second constraint is the amount of available testing time. Under the current design of PISA assessments, students will take a one-hour creative thinking test. This means that the range of possible assessment domains must necessarily be limited, in order to ensure that a sufficient amount of data is collected in each domain. As PISA aims to provide comparable measures of performance at the country level, rather than at the individual level, it is possible to apply a rotated test design in which students take different combinations of tasks within domains (with some overlap). Nonetheless, ensuring the ability to produce reliable measures of country-level student performance by each domain requires that a sufficient amount of testing time be dedicated to the tasks within each domain, therefore limiting the number that can reasonably be covered in the assessment.
A third constraint is the necessity to implement the creative thinking test within the standard PISA testing platform. The PISA test is administered on standard desktop computers with no touch-screen capability and no internet connection. The platform currently supports a range of item types and response modes, including multiple choice, text entry, drag and drop, hot spots (clicking on areas within a text or image), a chat interface, and interactive charts and graphs. While it has been possible to include new functionalities to the platform during the development of this assessment, such as a drawing tool, both the choice of assessment domains and the design of the tasks had to take into due consideration the technical limitations of the platform.
Taking these main constraints into account, and building upon the literature that discusses the different domains of creativity, the PISA 2021 creative thinking assessment focuses on two broad thematic content areas: “creative expression” and “knowledge creation and creative problem solving”. “Creative expression” refers to instances where creative thinking is involved in communicating one’s inner world to others. This thematic content area is further divided into the domains of “written expression” and “visual expression”. Originality, aesthetics, imagination, and affective intention and response largely characterise creative engagement in these domains. By contrast, creative engagement in “knowledge creation and creative problem-solving” involves a more functional employment of creative thinking that is related to the investigation of open questions or problems (where there is no single solution). It is divided into the domains of “scientific problem solving” and “social problem solving”. In these domains, creative engagement is a means to a “better end”, and it can thus be characterised by generating solutions that are original, innovative, effective and efficient.
The four assessment domains represent a reasonable coverage of the creative thinking activities in which 15-year-old typically engage, and reflect the nature of real world and everyday creative thinking. While they clearly do not exhaust all possible manifestations of creative thinking in school, they do provide a sufficiently diverse coverage of the construct of creative thinking as well as adequately respect the various logistical and technological constraints of the PISA 2021 assessment.
Finally, given that differences in cultural preferences for certain forms of creative engagement exist, as do differences in what is valued in education and in how subjects are taught across the world, we can expect some degree of variation in student performance across domains. By having students work on more than one domain, it will be possible to gain insights on country-level strengths and weakness by domain of creative thinking. The data may also uncover the differences in the extent to which students are encouraged to search for their own solutions and ways to express their ideas, with important implications for how creative thinking in different domains should be taught in school (Figure 2).
Written expression
Written work represents a natural means for creative expression both inside and outside of the school context, and creative writing is important for developing children’s cognitive and communication skills (Tompkins, 1982). Good creative writing requires logical consistency; creative writers ask the readers to understand and believe in their imagination, and this requires that they focus on details and continuity. For example, even stories that are based on fantasy, with monsters and space aliens, need to obey a certain set of rules of logic and to make sense within the universe the author has created.
Individuals engaged in creative writing reflect upon the craft and process of writing, define expectations for their work, and respond imaginatively to the text of others (Carter, 2001). These processes can stimulate many new areas of intellectual and emotional development for students, deepening their understanding of themselves and of the world (Essex, 1996). Moreover, creative writing does
![]()
Figure 2. Proposed focus domains for the assessment.
not only apply to works of fiction: engaging in non-fictional writing can also be creative, such as writing slogans and tag-lines, and these forms of creative written expression can help students to understand and master basic rules of effective communication they need for their life.
In the cognitive test, students will need to demonstrate a capacity to express their imagination in a written format, respecting the rules and conventions that make written communication understandable and appreciated for its originality by different audiences. Several test unit templates have been designed for the domain of written expression. Students are asked to: engage in open and imaginative writing (with constraints limiting the length of written text that human raters will need to evaluate); generate ideas for various written formats by considering different stimuli, such as cartoons without captions or fantasy illustrations; and make an original improvement to someone else’s written work (as provided in the task stimuli).
Visual expression
In the domain of visual expression, students explore, experiment and communicate ideas and their own experiences using a range of media, materials and processes (Irish National Teacher Association (INTO), 2009). Producing visual representations can help students to interpret both overt and subtle images and to develop a better understanding of how information, communication and design work in general. Creative visual expression has arguably become more important in recent years: with the ubiquity of desktop publishing, digital imaging and design software, nearly everyone will, at some point, be making visual communications that will affect either themselves or the wider public (think, for example, about the importance of the visual quality of a curriculum vitae).
The test unit templates designed in the domain of visual expression ask students to: engage in open visual design tasks, using a digital drawing tool; generate visual design ideas based on the scenario and stimuli provided in the unit (e.g. specific details to include, provision of certain drawing tools); and suggest or make original improvements to different forms of visual expression (as provided in the task stimuli), following given instructions or additional information.
Social problem solving
In their everyday life, students use creative thinking to tackle (inter-)personal, and social problems. Creative thinking in this context involves looking at the problem not just from a technical perspective but also from the social perspective, in other words trying to understand and address the needs of others to find solutions to central problems—be they at a personal, school, wider community or global level. Creative thinking in this domain depends on the students’ ability to empathise with and evaluate the needs of a specific group, recognise patterns, and construct ideas that have emotional meaning, as well as propose innovative yet functional solutions (Brown and Wyatt, 2010).
The test unit templates designed in the domain of social problem solving ask students to: engage in open problem-solving tasks with a social focus, either individually or in simulated collaborative scenarios; generate ideas for solutions to social problems, based on a given scenario; and suggest original improvements to problem solutions (as provided in the task stimuli).
Scientific problem solving
Creative thinking in science can manifest itself in various ways: in the conception of new ideas that contribute to advancing scientific knowledge; in the conception of experiments to probe hypotheses; in the development of scientific ideas or inventions applied to particular domains of practical interest; or in the novel implementation of plans and blueprints for scientific/engineering activities (Moravcsik, 1981). Students can demonstrate creative thinking as they engage in inquiry sessions during which they explore, manipulate and experiment with materials in any way they choose (Hoover, 1994).
Creative thinking in science is closely related to scientific inquiry skills, yet several characteristics of this test fundamentally differentiate it from other assessments of mathematics and science. First, this assessment focuses on the generation of new ideas, rather than on the application of taught knowledge. Secondly, the originality of students’ approaches and solutions are credited (provided that responses are valid). The third difference is the use of open problems that have multiple possible solutions and where there is no clear optimal solution. Lastly, this assessment focuses on students’ processes of creative thinking in scientific contexts—i.e. the ways in which students go about solving open problems and searching for original ideas—rather than their ability to produce a “right” or “most optimal” solution.
The test unit templates in the scientific problem solving domains cover these different aspects of creative thinking in various scientific contexts. In general, students are asked to: engage in open problem solving tasks in a scientific context; generate ideas for hypotheses or solutions to problems of a scientific nature, based on the given scenario; and suggest original improvements to experiments or problem solutions (as provided in the task stimuli). Possible units might present students with observations on a scientific phenomenon and ask the student to formulate different research questions or hypotheses to explain the phenomenon; others might ask students to invent something in a laboratory environment, utilising different tools. Units with a more mathematics focus could require students to develop different methods to demonstrate a given property of data or geometrical figures, or might ask students to make as many valid inferences as possible from a given set of data. Alternatively, units might present students with an open engineering problem that requires an innovative solution, or presents a system that can be made more efficient or effective.
Interactive simulations and games are particularly appropriate modes for assessing creative thinking in scientific problem solving because such environments provide immediate feedback to students on their choices and actions; observing how students react to this feedback can provide relevant measures of their capacity to engage in the process of failure and discovery that often characterises scientific innovation.
The importance of domain readiness is clearly an issue that inevitably arises with most tasks that can be imagined in this domain. Originality has little value without validity (i.e. appropriateness), and validity in turn requires at least some level of background knowledge or understanding of basic scientific principles. Moreover, finding scientific tasks that are equally demanding with regard to the level of the necessary background knowledge, across all countries and groups of students, is challenging. This issue could be mitigated by incorporating learning supports, such as short tutorials, that adequately cover the knowledge necessary to complete the task. Another alternative is to design tasks that obey scientific rules, but for which all students would have very limited experience.
Competency model of creative thinking
Figure 3 outlines the competency model for the PISA 2021 creative thinking test. The competency model deconstructs creative thinking into three facets for measurement purposes: “generate diverse ideas”, “generate creative ideas”, and “evaluate and improve ideas”.
The test measures creative thinking by asking students to engage productively in the cognitive processes of idea generation (the generation of diverse or creative ideas respectively) and idea evaluation and improvement. It therefore does not only look at the divergent cognitive processes of creative thinking (the ability to generate diverse or creative ideas); students are also asked to evaluate other people’s ideas and develop and suggest original improvements to those ideas.
“Ideas” in the context of the PISA assessment can take many forms: for example a story, a drawing, a solution to a social problem, or a research question concerning a scientific phenomenon. The test units provide a meaningful context and sufficiently open tasks in which students can prove their capacity to produce multiple ideas and think outside of the box. The test units will be assembled in such a way that the test provides, as a whole and at the population level, an adequate coverage of all the facets of creative thinking. However, not every unit within the test provides points of observation for all of the facets of the competency model.
![]()
Figure 3. Competency model for the PISA test of creative thinking.
The skills demanded by the cognitive processes of idea generation and idea evaluation and improvement are partly defined by context. For example, although composing a poem and considering viable scientific hypotheses to explore in a laboratory can both be conceived as acts of creative idea generation, the actual cognitive and domain-relevant skills an individual needs to successfully think creatively in these two activities are somewhat different and can rely on a different set of domain knowledge and experience. In written expression, idea generation generally involves the writer identifying a memory probe based on the topic of the writing and using this probe to explore long-term memory (Bereiter and Scardamalia, 1987) . In a scientific setting, idea generation mainly originates from an inquiry process that involves formulating new questions and carrying out experiments in order to collect evidence concerning those questions (Getzels and Csikszentmihalyi, 1967).
Similarly, idea evaluation and selection can involve distinctive cognitive skills, domain knowledge and experience across different creative domains. For example, creative written expression requires revision based on an effort to achieve clarity and coherence, and address audience needs (Bereiter and Scardamalia, 1987); in a scientific context, evaluation entails verifying that a solution is effective and is feasible.
The balanced coverage of four domains will make it possible to investigate the extent to which students who are proficient in one area of creative thinking can also demonstrate proficiency in others.
Generate diverse ideas
A common indicator of someone’s capacity to think creatively is the number of ideas he or she is able to generate, often termed ideational fluency. In fact, ideational fluency has long been the most-used measure for assessing an individual’s potential for creative work. However, more than the simple generation of many ideas, which all may be very similar to one another, it is the diversity of those ideas, or ideational flexibility, that truly demonstrates creative thinking and the ability to avoid functional fixedness in the idea generation process (Amabile, 1983).
In the measurement of idea generation, those ideas offered in distinctly different categories should be weighted more than those that fall within the same category (Guilford, 1956). For example in a hypothetical task asking students to list possible uses for a piece of paper, a student who suggests “writing, making a funnel, cutting paper dolls, using as insulation” (four distinct categories of use) shows a higher level of skill in idea generation than a student who suggests “writing, scribbling, printing and drawing” (all in the same category, i.e. paper as a canvas).
The facet “generate diverse ideas” of the creative thinking test focuses on students’ capacities to think flexibly across domains: for example, by providing different solutions for a problem, writing different story ideas, or creating different ways to visually represent an idea. In tasks relating to this facet, students are presented with an open scenario and instructed to provide two or three answers that are different from one another. It should be noted that the measure of the diversity of students’ ideas is contingent upon the responses being appropriate with respect to the specific task.
Generate creative ideas
Creative thinking begins with an intention and ends with a tangible product or idea. Despite the differences that exist in the conceptual and empirical research on creativity, the literature generally agrees that creative outputs are both novel and useful.
However, this new-and-useful criteria for measuring creative ideas nonetheless requires further qualification. Firstly, there is uncertainty in the literature about whether “new” means completely unique or only pre-eminent, or whether creative outputs need only be new for the creator or for society at large (Batey and Furnham, 2006). Clearly, measuring 15-year-olds’ creative ideas against the criteria of total uniqueness and society’s positive judgement in PISA is inappropriate. In this context, the related and often cited criterion of “originality” for measuring novelty is a useful concept to measure creative ideas. Defined by (Guilford, 1950) as “statistical infrequency”, this criterion encompasses the qualities of newness, remoteness, novelty or unusualness, and refers to deviance from patterns observed within the population at hand. Essentially it poses the question, how frequent is this kind of response? In the PISA assessment, originality is thus relative to a reference point: the responses of other students who complete the same task.
Secondly, there is also the issue of whether the new-and-useful definition of creative ideas applies uniformly across domains. The requirement of novelty may be less appropriate for some scientific endeavours, where the efficiency, feasibility and effectiveness of advancements in knowledge or solutions to problems provide greater value than novelty, just as a requirement of usefulness may be less essential for creative engagement in the arts (Batey and Furnham, 2006). These differences in the meaning and relative value of “usefulness/relevance” and “originality” across domains need to be taken into account in the test design: for example, it is important to provide to students a clear justification for searching for an original scientific explanation when not-original explanations might be more plausible.
In the PISA test, the facet “generate creative ideas’ focuses on students” capacities to search for appropriate and original ideas across different domains (e.g. an original story idea, an original way to communicate an idea in visual form, or an original solution to a social or scientific problem). In other words, students are asked to provide an appropriate, task-relevant response that other people might not have thought of. The appropriateness criteria means that the response must comply with the basic requirements of the task, respect the task constraints (if present), and reflect a minimum level of usefulness in the response. This is to ensure that students are truly thinking creatively (i.e. generating ideas that are both original and of use) rather than making random associations (i.e. producing original ideas of no use with respect to the task context). In tasks relating to this facet, students are presented with an open scenario and asked to elaborate, in some detail, one original idea.
Evaluate and improve ideas
Successfully engaging in creative thinking is not simply characterized by producing something new by deviating from the usual, but also something that works for its intended purpose; a creative output therefore generates “effective surprise” (Bruner, 1979). Evaluative cognitive processes support the production of novel ideas that are at the same time adequate, efficient and effective (Cropley, 2006). They may serve to remediate deficiencies in ideas, and often lead to
![]()
Table 1. Possible ways to measure creative thinking facets across domains.
further iterations of idea generation or the reshaping of initial ideas to improve the creative outcome. Evaluation and iteration are thus at the heart of the creative thinking process. The capacity to identify and provide feedback on the strengths and weaknesses of others’ ideas is also an essential part of any collective effort of knowledge creation.
The facet “evaluate and improve ideas” of the test focuses on students’ capacities to evaluate limitations in given ideas and find original ways to improve them. In order to reduce problems of dependency across items, students are not asked to iterate upon their own ideas but rather to change or continue someone else’s work. In tasks relating to this facet, students are presented with an open scenario and asked to suggest an original improvement for the given idea. Similarly to tasks in the other facets, any measure of “evaluate and improve ideas” is contingent upon the appropriateness of a student’s response. In these tasks, an appropriate response must be an original improvement. An “original improvement” is defined as a change that preserves the essence of the idea presented in the task but that incorporates original elements, thus incorporating both elements of new-and-useful that characterise creative ideas (Table 1).
NOTES
1Also referred to as the Five Factor Model of personality traits: Openness to experience; Conscientiousness; Extraversion; Agreeableness; and Neuroticism (see McCrae and Costa (1987)).