Interventions Set in Context to Engage More Women in Computer Science Undergraduate Programs: A Realist Synthesis Approach ()
1. Introduction
The underrepresentation of women in undergraduate computer science programs has been a prominent issue for over 25 years, and has recently gained the attention of businesses and industries are striving to diversify their technical workforce. Reasons for this underrepresentation have been studied and published in various journals (see the references section of this paper). Most of the research has focused on understanding the situation at the authors’ home institutions and identifying successful interventions. Even meta-analyses have typically studied only a few colleges and universities, and rarely provided context about where the research was conducted. Studies frequently mix small highly selective colleges with large public universities without discussing the vastly different conditions at these institutions. For example, research done at Harvey Mudd College and the Colorado School of Mines—both highly selective small colleges where all students take an introductory computer science course—shows how they have increased the number of women computer science majors. These studies often imply that the same techniques would be very effective at very large public universities, where not all students take an introductory computing course, without considering the significant differences in institutional contexts.
This paper uses a realist synthesis research approach, examining interventions within the specific context of each institution. It seeks to uncover the underlying mechanisms and conditions associated with successful interventions. The first part of the paper describes the realist synthesis approach and evaluates high-quality research papers to generate six Program Theories, which serve as an organizational framework for the main body of the paper. The last part of the paper identifies the mechanisms that lead to specific outcomes and makes recommendations for others in similar contexts to implement.
2. What Is a Realist Synthesis Research Approach?
Pawson and Tilley (1997) were the first to advance program and policy evaluation through a realism lens. They argued that, to be useful to decision makers, program and policy evaluations must answer, “what works for whom and in what circumstances.” Realist synthesis is a form of theory-based evaluation and/or research grounded in the philosophy of science and social science. The approach is based on ‘critical realism’, initially developed by philosopher Roy Bhaskar, which holds that science advances through uncovering “generative mechanisms” (Bhaskar, 2013). Through an investigative approach, realist analysis attempts to unpack the underlying mechanisms and contexts or conditions that influence interventions in complex systems. Rather than identifying “replications of the same intervention”, as is done in traditional systematic review, realist researchers and evaluators iteratively seek out wide-ranging contexts in which the same mechanisms have been attempted. In this regard, realist synthesis is not defined by topic boundaries (Pawson, Greenhalgh, Harvey, & Walshe, 2005).
Realist synthesis is characterized by two distinct, but iterative, phases. In the first phase, the reviewer identifies candidate theories of change—collectively called program theories—that explain how the intervention’s mechanisms are understood to impact outcomes. In the second phase, the reviewer tests and refines these theories against the research evidence (Pawson, 2006b).
Realist approaches are well suited to address more complex and interdisciplinary research questions. They are particularly appropriate for evaluating: 1) interventions that seem to work, but the ‘for whom and how’ is not well understood; 2) interventions that have yielded mixed results; and 3) interventions that will be scaled up, to understand how to adapt them to new contexts. Realist synthesis approaches are increasingly used to synthesize research in education, mental and physical health, and social services (Chung & Ram 2009). The research quality of primary studies is appraised and only those deemed to be of high quality are analyzed; the remainder serve as background material. Realist synthesis rejects the hierarchy of evidence utilized in systematic review (Wong Greenhaugh, Westhorp, Buckingham, & Pawson, 2011) (Figure 1).
Figure 1. Steps for realist synthesis (Pawson, 2006a), graphic by Barbara Cozzens.
3. Methodology
3.1. Preliminary Theory Formulation
At the start, we define important terms that we used in determining the theories of change, and which are used for the remainder of this paper:
Change strategy: approach to effect change in one area.
Conditions: relevant qualities of the context in which an intervention is implemented.
Factor: an independent variable studied via research.
Goal: overarching goal, such as retaining women in CS.
Intervention: implementation of a change strategy or collection of change strategies aligned with overarching goal and vision.
Mechanism: an actionable step or process that is part of an intervention.
Program theory: how an intervention’s mechanisms are understood to impact outcomes.
Relevance: pertaining to undergraduate women in computer science including the factors and settings that impact their success.
Rigor: of sufficient research quality including clear methodology (e.g., sample size and description, measures), findings, warranted conclusions (i.e., what worked for whom in what setting), and discussion of generalized implications.
We began with a scoping review to get a broad view of the available literature related to women in computer science, why they enroll, why they stay enrolled, and why they leave. In addition, a number of papers reviewed provided recommendations for recruitment, engagement, and sustainability of women in computer science. We progressively refined our goal to focus on the challenges facing women in computer science and what can be done to change the culture and systems to be supportive of women in their early semesters in computer science. Our initial set of program theories (theories of change) included such things as ‘interventions must supply the appropriate type of sequence and change mechanisms’, and ‘interventions that are easy to implement are probably not sustainable’. It became clear that the program theories had to directly address the problems faced by women and describe how to address them. Thus, we began the iterative phases of the research synthesis approach to theory elicitation.
3.2. Searching and Appraisal of Studies
We followed the guidelines provided by the RAMESES Publication Standards for realist analysis (Wong, Greenhaugh, Westhorp, Buckingham, & Pawson, 2013). The search for relevant papers was exhaustive, encompassing publications from a broad range of journals, including those focused on women in science to ACM journals, and extending beyond computer science to other STEM fields. In total, 56,795 papers were identified using the CS Database, and 61 of these were selected for detailed review. Two main criteria guided our decisions about quality: 1) Relevance. Each study had to provide relevant evidence. Determining this required extensive preliminary reading, and individual evaluations of relevance for each study. 2) Rigor. The study needed to meet sufficient quality standards within type, whether qualitative or quantitative research, process evaluation, or a randomized control trial. Papers were excluded if they merely listed problems and possible solutions without indicating where these solutions could be applied or proving their effectiveness in at least one context.
Each paper was read and rated on relevance and rigor by two of the four authors of this present paper, using a scale from 1 (best) to 5. Papers that received ratings of 1 or 2, and occasionally 3 coupled with a 1, were chosen for one-page write-ups, that informed the next iteration of the set of program theories. Unlike traditional systematic reviews, where researchers extract the same information from all included studies using a standard data-extraction, realist synthesis acknowledges that not all studies address every aspect of every program theory. Typically, evidence of causal structure, outcome patterns, interventions, implementation, and context are uncovered through various research strategies and are found in different types of literature.
In place of a standardized form, we developed and continuously refined two tables to record extracted data, facilitating the sorting and annotating of data source materials. We utilized coding techniques to track interventions, mechanisms, contexts, outcomes and other relevant information related to the intervention, mapping studies to evidence and program theories. The code words included algorithm, apps, self-efficacy, control structures, data structures, decision-making, functions, Java, object orientation, pair programming, problem solving, programming concepts, stereotyping, peer pressure, belonging, and more.
3.3. Second Iteration of Program Theories
We synthesized the data and determined nine candidate program theories. These were expressed in the form of hypotheses to be tested and refined against the empirical data. Some of the nine program theories were combined to create six program theories, for example peer culture and self-doubt were combined:
Student specific:
1) A sense of belonging to CS is an essential requirement early on.
2) Peer culture can create a hostile/unwelcome environment and contribute to self-doubt unless addressed directly and quickly.
3) Stereotyping in the media and elsewhere tends to discourage women from pursuing a major in CS, and this includes the stereotype that only the brilliant can major in computer science and this needs to be counteracted explicitly.
Department specific:
4) Curriculum in introductory CS courses must meet students where they are.
5) Faculty create an inclusive and welcoming environment and implement curricular changes while department Chairs are the front-line administrators and as such control what the faculty do.
Overarching:
6) Transparency and communication are essential to creating impactful changes.
The evidence synthesized here is presented in an incremental and cumulative manner. The reporting and analysis of the primary studies is organized into
Program Theories |
No. of Articles |
Sources *studies which appear in two or more theories |
A sense of belonging to CS is an essential requirement early on |
2 |
Sax et al., 2018*; Strayhorn, 2018 |
Peer culture, self-doubt and support |
16 |
Alvarado and Dodds, 2010; Beyer, 2014; Cohoon, 2001*, 2005; Frieze and Treat, 2006; Barker, O’Neill and Kazim, 2014; Camp, Liebe, and Slattery, 2020*; Charleston et al., 2014; Haynie and Counts, 2021; Lin, 2016; Margolis, Fisher, and Miller, 2000; Pantic and Clarke-Midura, 2019*; Sax, Blaney, Toven,
Lehman, Rodriquez, George, Zavala, 2018*;
Varma, 2007; |
Stereotyping and brilliance |
6 |
Beyer, 2014*; Cheryan et al., 2013; Pietri et al., 2019; Leslie, Cimpian, Meyer and Freeland, 2015; Pantic and Clarke-Midura, 2019*; Tinto, 1993 |
Curriculum must meet the student where they are |
3 |
Alvarado, Dodds, Libeskind-Hadas, 2012; Camp, Liebe, Slattery, 2020; Sax et al., 2018*; |
Faculty and department chairs |
5 |
Cohoon, 2001*; Lewis, 2021; Kezar and Sam, 2013; Sax et al., 2017; Scragg & Smith, 1998 |
Transparency and
communication |
1 |
Camp, Liebe, and Slattery, 2020* |
sections relative to the program theories: student specific, department specific and an overarching program theory. We created a table that cross-referenced the six program theories with the high-quality journal papers.
4. Data Synthesis and Analysis
A sense of belonging to CS is an essential requirement early on (Theory 1)
Many scholars agree that women's sense of belonging in CS develops in gendered social environments prior to college and while in college. The degree of belonging often fluctuates given varied social spaces and norms in these spaces. The CS culture is generally known as a male dominated space reinforcing male gender norms. These norms can range from “cut throat” and “make or break” to isolating, and are often hostile or alienating for women students. Sense of belonging is fostered among and within peer groups, faculty relationships, classroom settings, and other spaces on and off campus.
The most extensive and highest quality study on the sense of belonging among students in computer science introductory courses is the paper by Sax et al. (2018), “Sense of Belonging: The role of introductory courses for women and underrepresented minority students.” The quantitative study is done by surveying students twice, once at the beginning and once at the end of the first computer science course. A total of 1355 students completed both surveys in the study. 42% were women and 26% of that number were underrepresented women. The context of the Sax et al. (2018) study was explicit—13 large public research universities and 2 smaller private universities, all BRAID Schools. (i.e., ‘Building, Recruiting, and Inclusion for Diversity”, part of Anita Borg.org.).
Their major conclusion is that students come into the university with a sense of belonging in CS, positive or negative and that the first course does not change that sense of belonging for women, and indeed makes it worse, whereas underrepresented students come in with a relatively strong sense of belonging and it increases during the course (Sax, 2018: p. 9-12). Additionally, departments and college environments matter in building a positive sense of belonging for women and URM in CS. The main intervention strategy is to foster environments that allow for collaboration, communal work, and peer-to-peer engagement (Sax, 2018: p. 15). Sense of belonging is crucial to retention.
Strayhorn (2018) affirmed that sense of belonging also develops among peers through academic support and is critical to building retention in CS. They believe that an intervention of academic supports, impact persistence and retention efforts. The way in which they do that is by making women feel they belong by providing validating experiences and or by providing encouragement when students hit roadblocks.
Intervention leading to positive outcomes: Developing a sense of belonging early in the CS pathway is a strong predictor of women’s persistence and success. On the contrary, women who experience bias, hostility, or other unwelcoming factors are more inclined to leave the major. Sense of belonging can be influenced deeply through experiences with peers, instructors, curriculum, and other department procedures. This was seen to be true regardless of the size of the school, selectivity, public or private.
Peer culture, self-doubt, and support (Theory 2)
CS environments are largely impacted by peer culture and peer interactions. Peer experiences are nuanced, fleeting, and happen daily during a typical college day. Producing data around peer culture is a difficult yet a necessary step to fully understand the experiences of women in CS, particularly those that leave the field in college. Several empirical studies attempt to understand these experiences.
“Decoding the Female Exodus from Computing Education, Information, Community and Society” by Varma (2007) explores these cultures and subcultures in CS/CE. This is a high-quality interview study of 150 undergraduates majoring in CS/CE about their experiences in 2004-2005. The student samples were drawn from seven institutions that granted degrees in CS/CE and were minority-serving institutions, thus the context was made explicit. In each of five racial/ethnic groups, 15 female and 15 male students were interviewed: White, Black, Hispanic, Asian, and American Indian.
The study found more women (59%) than men (44%) consider changing majors. Several factors cited include male peer and faculty antagonism, animosity, and refusal to take them seriously, overt questioning, female socialization to believe gender limits their career options including lack of encouragement and a masculinized and often inhospitable atmosphere in the department. Some women felt devalued independent of objective evidence of their performance, and male social dominance coupled with stronger female need for social connection and community belongingness also contributed to their feelings of wanting to leave. This study highlights the need for women to be affirmed and recognized in positive ways much more than men to counterbalance historical socialization experiences. Therefore women, “are more sensitive to and dependent on the contextual-interactive dynamic undergirding and reproducing a social context that does not recognize them as legitimate contributors” (Varma, 2007: p. 191).
Another study depicted the positive possibilities in a gender affirmative culture. Cohoon’s (2005) meta-analysis showed that same-gender peer support was very powerful in reducing attrition. This study involved 18 CS departments at 17 schools (including 10 urban schools), 143 faculty/chairpersons, and 182 undergraduates (43% women). The basic finding was, "gendered attrition rates are small in departments where the gender balance of enrolled majors approaches parity". Although earlier, Cohoon (2001) found no effect of institutional selectivity on attrition rate (she did not consider institutional size), her 2005 meta-analysis found a negative correlation between class size and women's rate of CS enrollment.
Self-doubt in computer science is a lack of confidence in one’s computer science abilities. There is ample evidence that women have stronger and more pervasive experiences of self-doubt in CS compared to men. For example, Pantic and Clarke-Midura meta-analysis (2019) of 72 studies1 (89% empirical) found extensively that women quickly felt overwhelmed and intimidated upon enrollment in CS majors, often having a hard time relating to the institutional culture. A number of theorists found that women generally self-reported lower programming abilities than their actual performance, suggesting that women tend to underestimate their CS abilities (Pantic & Clarke-Midura, 2019: p. 128), Pietri (2019).
The consequences of self-doubt in CS are lower enrollment and retention rates for women. This pattern has individual and societal impacts. In her groundbreaking study (that did not report institutional sizes), Cohoon (2001) found that females were twice as likely as males to leave the CS major. Varma (2007) studied 150 non-freshman CS students’ experiences at seven minority-serving institutions (MSIs) and found that more women (59%) than men (44%) were considering changing their major.
Probable causes of women’s self-doubt in CS are related to their prior individual experiences both within and outside of school (e.g., personal experiences, classroom learning environments, peer culture), as well as collective beliefs. Pantic and Clarke-Midura (2019) found that women tend to have less computer, gaming, and programming experiences. Women entering the CS undergraduate program at Carnegie Mellon without extensive previous experience often compared themselves to the “programming gods” and “boy wonders” who “dream in code”. Pantic and Clarke-Midura (2019) also found that women’s intrinsic interest can be extinguished via external confidence-shaking factors such as comments from male peers that can trigger stereotype vulnerability and lead to women's disidentification in the field. Varma’s (2007) study (undergraduates from MSIs) also found that masculinized learning environments, based on social dominance, were a strong factor in the loss of women in CS. These challenges do not stop with university graduation—African American women deemed “successful” in CS majors still frequently experienced isolation and subordination in their subsequent work environments (Charleston et al., 2014). Beyond their personal experiences, women—even at top institutions—often feel discouraged by the cultural expectations of the computer science field (e.g., Margolis, Fisher, and Miller, 2000).
As indicated in the research above, many interventions have been proven to be successful in combatting self-doubt in all size colleges and universities with varying levels of selectivity. There are myriad ways to overcome women’s self-doubt and support their confidence in computer science that might be used to design interventions. Controllable, “proximal” factors include encouragement (enrollment, engagement, problem-solving, persistence), social support (peers, faculty, etc.), learning environment quality, and curriculum interventions. Women need to be encouraged and engaged in ways that develop their self-efficacy and identity as computer scientists (e.g., Camp, Liebe, and Slattery, 2020 study of systemic change at CS Mines). Women tend to prefer learning environments where more encouragement and support are provided (Pantic & Clarke-Midura, 2019). Also, solving hard problems that require persistence builds confidence. Haynie, in her CS evaluation work (e.g., Haynie & Counts, 2021), has found a strong link between problem-solving persistence and CS self-efficacy for female students, including for 38 black high school women in Alabama who prepared for and took the AP CS Principles course (e.g., Haynie and Counts, 2021). Lin (2016) provides large-scale evidence that CS majors’ self-efficacy is susceptible to social influence. Barker, O’Neill, and Kazim (2014) found that undergraduates’ sense of belonging in a Data Structures course was increased by requiring students to learn from each other in a socially safe environment using highly structured pair programming. On a larger scale, Sax et al. (2018) studied 418 students at 15 universities and found that women’s sense of belonging decreased throughout intro CS courses, and that gains in belongingness were associated with pair programming experiences and departmental support.
Pantic and Clarke-Midura (2019) also found that CS courses that increased academic outcomes, students’ senses of belonging, and/or retention of women (i.e., functioning as interventions) had one or more of the following mechanisms/ qualities: inclusive environment, flipped classroom format, teaching preferred types of programming, choosing relevant course material, offering pre-introductory programming boot camp for women students, providing a design-based focus where students made computational products to address community needs for real-world applications, and structures around collaborative work (e.g., pair programming, same sex project groups, learning communities where students take groups of classes simultaneously). Interventions with these mechanisms/qualities are aimed to increase student preparation, social collaboration, and the relevancy of content.
Interventions leading to positive outcomes: Undergraduate women in CS spend a significant amount of time interacting with peers in various social and academic settings. While same-gender peer support tends to boost confidence and provide positive environments, these groups cannot always be guaranteed to all students at all times. Examining peer culture and support is critical to understanding the trajectory of women in CS, regardless of size of the institution.
Self-doubt is a pervasive challenge for women in CS. The causes are myriad and historically entrenched. Controllable factors that can alleviate self-doubt include encouragement, social support, learning environment quality, and social learning structures such as collaborative work. Creating a supportive, encouraging environment is essential early on so that female students do not give up on CS study in their first college course.
5. Stereotyping and Brilliance (Theory 3)
There are a number of stereotypes in the field of CS that impact gender diversity within the field. The first includes stereotypes that are portrayed in media, education, and other publicized spaces about who computer scientists are, what they do, and/or how they behave. The second type speaks more to stereotype threat and the impact this has on women entering the field. Because these stereotypes are ingrained in everyday culture, many members of an educational community including women themselves fall into the trap of perpetuating them.
Scholars and researchers try to better understand stereotypes and their impacts by conducting research with undergraduate students. Usually, this research involves an experimental act such as providing women and control groups with readings or videos that display stereotypical behaviors or “non-stereotypical” content. Next, they try to formulate understandings of how and what each group absorbed from the task or how students will react. Other studies look at what happens to women when they fully understand and can acknowledge the pervasive stereotypes that exist. Does this in turn derail their confidence levels and negatively impact their sense of belonging?
Cheryan, Plaut, Handron, and Hudson (2013) conducted a two-part study to determine undergraduate students’ stereotypes of computer scientists, then to see if exposure to a media article confirming or opposing those stereotypes impacted students’ interest in computer science. The sample for part one was 293 undergraduate students (57% female), non-CS majors, at Stanford (n = 193) and UW (n = 100), large selective universities. The sample for part two was 54 undergraduate students (56% female) at Stanford (n = 38) and UW (n = 16). Data for a baseline group (n = 62) was collected; this group was poorly specified in the article, thus the validity of the baseline comparison is in question.
In part one, students were asked to describe computer scientists, and their written descriptions were coded for presence or absence of the following qualities: intelligence, technological-orientation, singular focus on computers, lack of interpersonal skills, masculinity, physical traits (e.g., thin, pale, unattractive, glasses). Women and men did not differ in likelihood of mentioning the categories; however, women with one or more CS courses were less likely to offer stereotypical categories. In part two, students read congruent or incongruent articles and were then asked their level of interest in computer science. Women who read the non-stereotypical article had significantly higher interest than women who read the stereotypical article or were in the baseline group. The authors state that pervasive use of the “computer nerd” stereotype (in media) discouraged women from pursuing a CS major. Increasing women’s participation may require drawing attention away from stereotypical representations and towards more inclusive ones.
Beyer (2014) and Pietri et al. (2019) also discuss stereotypes in the CS/STEM field. Beyer’s (2014) study is similar to that mentioned above—students were given stereotypical content and its impact on their thoughts and sense of belonging were measured. Initially, negative stereotypes about CS were held about equally for women and for men; but when women saw “geeky” images related to CS, it tended to reduce their sense of belonging in the field. The study clearly explained that stereotypes and classroom experiences impact expectancies of success or self-efficacy.
Pietri et al. (2019) took the issue of stereotyping to another level. Participants were broken into various groups to watch different videos (the control group watched no videos). After the videos, participants completed surveys on bias literacy. Researchers “found that videos acted as an external cue that heightened women’s anticipated social identity threat at a hypothetical STEM company.” In other words, as women knew more about bias literacy this became “the mechanism underscoring their enhanced social identity threat.” It is made very clear that bias literacy training is necessary, however, we have to be sure that it does not cause more harm to women. This study did not parse out the CS field, but instead talked about STEM broadly.
A particular type of stereotyping is the notion of brilliance. Some have conjectured over the years that certain disciplines expect only the most brilliant as majors. This creates the possibility that those who don’t consider themselves brilliant will not pursue certain majors; often these are women and minorities. The paper “Expectations of brilliance underlie gender distributions across academic disciplines” by Leslie, Cimpian, Meyer and Freeland (2015) in the journal Women in Science hypothesizes that the fields where women are underrepresented are those where raw innate talent appears to be the main requirement for success, such as the fields of computer science, mathematics, physics, economics, music composition, and philosophy. The study poses three competing hypotheses, notably: 1) that there is a gender difference in willingness or ability to work long hours, 2) the more selective a discipline the fewer women, 3) fields that favor systematizing over empathizing have fewer women. The authors used a large-scale nationwide study of academics in 30 disciplines for their field specific hypothesis, along with the three competing hypotheses. They surveyed 1820 faculty, postdocs and graduate students from 30 disciplines, 12 of which are STEM disciplines, at geographically diverse high-profile public and private research universities across the United States. Participants were asked questions concerning their own discipline in a survey where they were to each answer in agree-disagree mode with 1 = strongly disagree up to 5 = strongly agree. Responses were averaged within disciplines so that a high score close to 5 implied a great expectation of brilliance. In each case regression calculations were made for each of the theories and percentage of women receiving degrees in that field. Field specific ability beliefs had a significant R2, when compared to the competing hypotheses that did not.
Besides determining what interventions work where and in what combination, a goal of this realist synthesis paper is to indicate what can be changed and what is nearly impossible to change. Field specific perceptions of a requirement of brilliance may be the most difficult to change and may indeed be impossible to change. Knowing these beliefs exist is however important for “work arounds” that can be developed to construct the interventions needed.
In the Pantic and Clarke-Midura (2019) paper the authors focused on factors contributing to low percentages of women in CS. The authors included studies that completely or partially addressed retention of undergraduate women in CS, included female participants, and were available via academic journals. They used empirical studies and included 42 quantitative studies, 11 mixed method studies, and 10 qualitative studies. They organized their studies according to Tinto’s (1993) model. The factors they reviewed were broken into individual (pre-entry), institutional, and societal factors. It is interesting to note that even though this was a synthesis paper, aimed at better understandings of retention of women in CS, there was no attempt to consider the mindset of faculty relative to whether men or women were better intellectually able to complete the course sequence in CS.
Interventions leading to positive outcomes: Discussing and recognizing the deep seeded everyday stereotypical thoughts, behaviors, and processes is key to unlocking the sense of belonging for women in CS. As long as gendered stereotypes exist in the minds and actions of actors within educational spaces and even media, women will not feel they fully belong. The sooner we can move from recognition of these behaviors to their disruption, the better institutions can support gender diversity in CS with belonging and inclusion at the center.
Preconceived notions of who is intellectually capable of pursuing various majors influences much of what happens once students are in college, but also influences decision makers on who gets into a particular college (Admission counselors) when students indicate their intended major
Curriculum (Theory 4)
Curriculum, especially in introductory courses which may be the students’ first exposure to computing, can impact the students’ sense of belonging and their ability to feel included in the field, directly impacting their success and retention in college (Sax, Blaney, Lehman, Rodriguez, George, & Zavala, 2018: p. 6).
An intervention that appears to work regardless of size of the institution or level of the student is an inclusive curriculum which employs student-centered pedagogical and curricular practices, which in turn become critical to women’s success in computing. An inclusive curriculum, especially in introductory courses, is a key mechanism in retaining women in STEM fields. The Sax, Blaney, Lehman, Rodriguez, George, and Zavala (2018) analysis revealed that students’ sense of belonging in their field increased as they were exposed to more inclusive pedagogical practices in the introductory course. An inclusive curriculum can help mitigate women’s self-doubt. As mentioned earlier, women often doubt their abilities in CS as they compare their lack of computer, gaming, and programming experience to the “programming gods” and “boy wonders” who “dream in code” (see Self Doubt section).
The mechanism of creating an inclusive curriculum is designed to meet students where they are, motivate their interest in the field and equally prepare students for the next level in their studies. One important mechanism to consider as part of an intervention design to meet students where they are and to increase women’s retention in CS is a multi-track introductory course (Alvarado, Dodds, & Lebeskind-Hada, 2012). Such courses with at least two tracks for students with and without programming experience provide students without programming experience, including women, with a classroom in which their peers’ level of experience matches their own. Furthermore, students with prior programming experience can explore in-depth problem-solving techniques to enhance their CS engagement. Additional tracks created in collaboration with other departments can be specially designed to attract a more diverse group of students to continue with CS (Camp et al., 2020).
In their study, Alvarado, Dodds, Lebeskind-Hada (2012) detail changes implemented at Harvey Mudd College to broaden participation in CS. This was particularly important at Harvey Mudd because all students (N = 200) are required to take and pass the introductory computer science class. One of the changes was to offer multi-track introductory courses where different sections for introductory courses are offered to accommodate the different experience levels of freshman students.
In a qualitative case study at the Colorado School of Mines, development of female CS majors (CS@Mines) from 2012-2020, Camp et al. (2020) utilized the NCWIT’s Systemic Change Model with 81 recommendations in key focus areas.
In addition to the mechanism of multi-track introductory courses, an additional mechanism for novice programmers with less computer experience is a supplemental 1-credit course offered alongside the introductory course (Florida International University offers such a course—by personal communication on CIC session, 2021). The goal of the supplemental course is to support these students by boosting their computer skills and self-efficacy in CS.
Interventions that lead to positive outcomes: Multi-track introductory courses have a two-fold positive impact: reduction of peer pressure felt by novice programmers in the classroom, and opportunities for more experienced programmers to explore in-depth problem-solving techniques that otherwise would not be available to them.
Faculty and Department Chairs (Theory 5)
Stereotypes in CS impact gender diversity and belonging within the field. Students are exposed to these stereotypes in media, education, and other publicized spaces about who computer scientists are, what they do, and/or how they behave. Computer scientists are portrayed as brilliant which deters those that don’t consider themselves brilliant from pursuing a CS major, often these are women and minorities.
Women that can see themselves in a computing career may arrive at college to face a competitive, isolating, and often hostile environment. Faculty-led classroom discussion to disrupt stereotypical thoughts and behaviors (that are ingrained in everyday CS culture) is a potential mechanism as part of an intervention to increase female retention.
Microaggressions are subtle or unintentional comments or actions towards members of marginalized groups that can have a negative impact on these individuals as they experience being belittled, minimized, left out or stigmatized. Such comments arise from bias and socialization, implicit and explicit, unconscious, and conscious. A guided conversation about the microaggressions faced by women and minorities brings awareness that these stereotypes are not real. These discussions allow students to challenge their own stereotypical behaviors, and empower women and minorities with means to counteract microaggressions. Faculty can also incorporate awareness of bias and microaggressions into the teaching assistants’ training (Lewis, 2021) to support women’s engagement, confidence, and persistence. Lewis, 2021 offers a training course for teaching assistants to help students learn about stereotypes, implicit bias and handling microaggressions in the classroom.
Faculty and CS department chairs can influence gendered attrition rates by increasing the time faculty devote to mentoring students, and by increasing the presence of female faculty (Cohoon, 2001). Mentoring females with personalized advice and encouragement can reduce the differential between male and female attrition rates. These are additional mechanisms for increasing women’s retention in CS.
In this study, Cohoon (2001) aimed to identify conditions affecting female retention in undergraduate CS with respect to departments rather than individual students. Moreover, the paper looks to identify departmental actions and characteristics that are linked with women’s attrition. The State Council of Higher Education for Virginia provided attrition rate data from 23 CS departments (from 1992 to 1997). The evidence supports the finding that institutions that provide CS departments with enough support retained more women, and that departments located in regions where there were career opportunities also retained more women.
CS department chairs face numerous challenges in trying to encourage more women and students of color to pursue degrees in computing. CS department chairs are the administrators at a college or university who negotiate budgets based on enrollments, hire faculty, and generally make teaching assignments. In the paper “Diversifying Undergraduate Computer Science: The Role of Department Chairs in Promoting Gender and Racial Diversity”, Sax, Blaney, Toven-Lindsey, Lehman (2017) describes a qualitative interview study that focused on the background of department chairs, their goals for broadening participation in CS and the departmental culture for women and minorities. The authors used a sample drawn from 15 BRAID schools—all research universities from across the country, 13 of which are public universities. The institutions have an average of 828 undergraduate CS majors; and 4 of the 15 have women Department Chairs.
In this study Sax, Blaney, Toven-Lindsey, Lehman (2017) found that most CS Department Chairs were comfortable talking about increasing gender diversity but less comfortable talking about increasing racial diversity. Department Chairs focused mostly on external factors—recruitment, lack of female role models, etc. Those Chairs who were actively involved in making strategic changes focused on mechanisms in the areas of pedagogy, mentoring for women, promoting research/internship opportunities, and supporting student organizations. Those Chairs actively working on change also believed in the need to infuse a theory of institutionalization, which refers to policies and practices that become part of the underlying assumptions and norms embedded in the culture of an organization (Kezar & Sam, 2013) in order for the changes to become permanent.
The study found that Department Chairs could not stay focused solely on the external barriers (e.g., recruitment), but needed to focus on the persistence of existing students. This article recommended that Department Chairs continue to participate in professional networks such as BRAID to keep them educated and informed on best practices and to “enhance the sense of agency around diversity work.” This study was important because it moved the conversation towards institutional level factors that impede or support change in the CS department. In terms of learning something brand new, this article did not necessarily provide that but it did encourage departments to determine what unique barriers exist in their college so as to potentially leverage the BRAID program.
A study by Scragg & Smith, (1998) indicated that departments need to look beyond the obvious—they may be doing everything right without it yielding an increase in the number of women undergraduate CS majors. In this study, persistence was not a problem—the problem was recruitment at the college entry level. Improving recruitment efforts required working closely with admissions to solve this problem. A mechanism where students are allowed to choose the CS major as freshmen or sophomores may also produce the desired outcome regardless of context.
Interventions that lead to positive outcomes: Faculty can disrupt stereotyped thoughts and behavior through classroom led discussions and/or course staff training on bias and microaggressions faced by minorities. Faculty can influence gendered attrition rates by increasing the time devoted to student mentoring. The literature of the last twenty years suggests that a Chair’s most important work should be to support their faculty in making pedagogical changes and to communicate with their department as a whole about the importance of gender diversity in computer science.
Transparency and Communication (Theory 6)
Transparency across academic and non-academic units is an essential component of creating impactful changes for undergraduate women in CS. Academic units both at the college and departmental level need to have clear and frequent communications about new and existing initiatives. One mechanism for transparency and success can be recurring meetings to share information about what is happening in each unit. Including non-academic units in these meetings is just as critical. Learning centers or other parts of the college/university with vested interest in advancing women in STEM/CS also need to know and understand what is happening within academic units and vice versa. This level of transparency can encourage cross-unit collaboration leading to deeper and more sustainable changes for undergraduate women in CS.
Camp, Liebe, and Slattery (2020) reported the pursuit of a systemic change model over the past decade for broadening participation in CS at Mines. Using the NCWIT (2023) model as a guide for strategic recommendations, faculty identified problems, implemented solutions, and continually monitored changes. Although each university environment is different and efforts require a different balance of interventions and mechanisms, core to achieving the goal is to align stakeholders—the faculty and staff—around the objectives, resources, activities/interventions, intended outcomes, and evaluation for systemic change, building a shared understanding of the overarching goal and how all the component parts work together. This includes the will to participate in the process, at least at the level of staying informed about efforts and results. At Mines, a small selective public college in Colorado, strategic interventions focused in the areas of increasing enrollment (e.g., outreach to high schools, targeting recruitment of girls into CS classes), teaching interventions (e.g., using collaborative learning activities, peer-mentoring), curricular interventions, student-student and student-faculty support interventions (e.g., community building, connection and service), institutional policies and support, and evaluation and tracking. Without ongoing faculty and staff communication and engagement around a systemic change plan, the best an institution can hope for is uncoordinated, piecemeal efforts. Communication around a shared vision, goal and set of interventions/mechanisms is a necessary but not sufficient factor in successfully broadening student participation in computing.
Interventions that lead to positive outcomes: Communication becomes even more essential as pieces of the plan are enacted, so that efforts are aligned with the systemic change plan and stakeholders' understandings.
6. Summary
This realist review focuses on program theories that impact the number of women CS majors in a variety of settings and contexts. Rather than merging all contexts together, we considered how the interventions functioned in the context of the locations described by the authors and what might be changed to fit other contexts. Contexts included size of the institutions, degree of selectivity, public or private, prior attempts to increase the number of female majors, etc. We cross tabulated program theories with interventions to display our findings.
a. The student-specific interventions and corresponding mechanisms that are intended to create a sense of belonging, spur positive peer interactions, lessen self-doubt and lessen the impact of old stereotypes are easier to execute in small colleges and universities, with smaller class sizes, and extensive faculty interactions. This has been proven by Harvey Mudd, Colorado School of Mines, Williams, Mt. Holyoke, and others. From the data cited in this paper, some more selective large schools, such as Stanford, have also been successful with interventions that increase the number of women CS majors. Large universities need to try harder.
b. Curriculum changes and faculty support for interventions have been shown to work at all sizes of schools and selectivity, public and private. A dominant characteristic of these curricular changes is that they meet students where they are, students with no programming experience, students who don’t even understand what CS is, and students who are at the sampling stage before selecting their major. Examples of promising curricular changes in any context are: 1) having two tracks of the initial CS course, one for students with programming experience and one for those without programming experience; 2) having a one credit course for students with no CS experience to fill in the gaps; and 3) having CS exercises and projects that appeal to socially conscious students.
c. The potential role of the faculty in teaching introductory CS courses is extensive, especially at larger universities where faculty are responsible for creating an inclusive atmosphere in the classes, addressing student biases, and thwarting students who want to disrupt the class and/or dominate communication in the class. The faculty teaching the introductory classes need the support of the department chairperson and the research faculty, especially in large public universities.
7. Recommendations for Change
7.1. Inclusive Classrooms
Create an inclusive classroom atmosphere in introductory CS courses at both public and private universities and colleges, regardless of class size. The data clearly shows that women thrive in non-confrontational environments and where they are regarded as equal to men in their abilities and contributions. Although this recommendation is easier to implement in smaller colleges with class sizes under 40, this paper provides several suggestions for implementing changes in larger schools. For example, creating multiple sections of the introductory course—such as one for students with prior programming experiences and one for those without—can help. Additionally, offering a one-credit course to provide support for students needing additional background help is recommended
7.2. Counteract Explicit and Implicit Biases
It is essential that faculty, with the guidance of their Chairperson, to address and counteract explicit and implicit biases as soon as they surface. This should be done directly and consistently, regardless of class size or university. Upperclassmen can play a supportive role in this regard, and various co-curricular activities can reinforce the message that computer science should be inclusive, both in educational settings and in the workplace.
7.3. Belonging Is a Strong Predictor of Success
Aligned with the first recommendation, it is crucial to foster a sense of community belonging among students, both male and female, within their major. Students at large and small, public and private universities want to see themselves represented in videos about the field, and seek encouragement from faculty that computer science is the right field for them. They also desire peer support both inside and outside of their major. Co-curricular activities play a vital role in helping students feel part of a community of peers who study computer science and aspire to careers in technology.
8. Conclusion about the Realist Synthesis Approach
Realist synthesis is a new but emerging approach to evidence review in education research. In this paper, we have described our use and development of the approach. It is particularly appropriate for unpacking the impact of complex interventions because it works on the premise that one needs to understand how interventions work in different contexts, and why. It is not an easy option. Realist review demands much of the reviewer, including an ability to think flexibly and deal with complexity. In a realist synthesis approach, there is a set of principles that the reviewer must relate to the question at hand, while still being true to the realist approach. Given that the fundamental interest in realist synthesis is about finding out what works in what contexts, the recommendations one can make will not be generalizable. Rather a realist review approach results in findings that are theoretically transferable; ideas (‘theories’) that can be tested in different contexts, with different stakeholders.
We have not tried to distinguish between interventions that support retention of women and those that support engagement of women. SUNY Geneseo (Scaggs) discovered that they were doing “everything right” in terms of retention so they asked why didn’t their numbers of women in CS go up? Perhaps an engagement issue. It turned out that the admission process restricted the number of women admitted to CS.
The ACM project (Stephenson, Miller, Alvarado, Barker, Barr, Camp, Frieze, Lewis, Mindell, Limbird, Richardson, Sahami, Villa, & Zweben, 2018) identified the difference between retention and engagement relative to increasing the percentage of women CS majors, where engagement was coupled with recruitment. Our belief is that engagement is critical to retaining more women in CS, engagement not only adds more women but also retains the ones you have. The ACM project’s case studies include Stanford (large private university with high admission standards, UC San Diego (large public university with moderate admission standards and large numbers of transfer students), and Colorado School of Mines (small highly competitive engineering school), with references to other small schools like Harvey Mudd College, Williams College, and Mt. Holyoke University. This work comes closest to collecting data and drawing conclusions in the context of the size of the schools and the admission standards.
The authors wish to thank Aadarsh Padiyath at the University of Michigan and others for their helpful suggestion on this paper.
Positionality
As authors, we understand that we approach this research with varied perspectives and lived experiences. Dr. H is a Latina woman who has a vested interest in broadening participation for women in STEM broadly, and especially in CS. Having just completed her dissertation studying the experiences of Black and Latina women in CS, she approaches this research with preconceived notions that these particular women are often excluded in social and academic settings. She often leans on a community of other scholars and researchers to discuss this work. Dr. H is currently serving as Director of DEI for a highly selective Ivy League University.
Dr. A.C. is an Associate Teaching Professor at a large public university. Her interests lie on the impact of the current CS education practices on minority populations, including women, in the field. She has led the revision of the CS department introductory course sequence to facilitate students' transition through the sequence, reduce classroom intimidation, and increase women’s interest in computing. Dr. A.C. chairs the Advancing Women in CS's effort to reduce the gender gap in the major at a large public university.
Dr. KH is the director of a Research and Evaluation organization, a nationally recognized educational research and evaluation organization with a mission of improving K-16 STEM education. The organization has wide array of broadening participation in computing projects include: 1) Peer-learning Communities to Develop Rural, African American Girls’ Computer Science Knowledge and Career Readiness, 2) Collaborative Research: Infusing Cooperative Learning into Computer Science Principles Courses to Promote Engagement and Diversity, 3) Broadening Participation: The Development, Implementation, and Evaluation of an AP Computer Science Preparatory Sequence for Underrepresented High School Students, 4) Bringing a Rigorous Computer Science Principles Course to Largest School System in the U.S, 5) Pathways for Alabama Computer Science and 6) Exemplary School-level Approaches to Broadening Participation in AP CS Principles. Dr. KH received her Ph.D. from Stanford University in Educational Psychology. and her M.Ed. from Rutgers University in Educational Statistics and Measurement, and has taught at Rutgers University, San Jose State University, and Dominican College. Dr. C has served as evaluator for a large public university on STEM projects that include: Advancing Women in Computer Science (AWICS), Data Science Principles of the Human-Machine Convergence, Socially Cognizant Robotics for a Technology Enhanced Society, and Teacher Professional Development for Embedding Computational Thinking in Mathematics and Science High School Classes.
Dr. MC is a mathematician, who has spent the last 50 years promoting quality STEM education for all. She retired from administrative positions from Chairperson of Mathematics at Northeastern University, to President of the Colorado Institute of Technology. Included in her administrative positions she spent ten years as Division Director in the Education and Human Resources Directorate at NSF during the 90’s. She has been a Distinguished Research Professor at a large public university for the past fifteen years. She has led the research and evaluation portion of the Advancing Women in Computer Science project at a large public university.
Protection of Vulnerable Populations
This research paper did not involve working with human subjects. The research was in the form of database searches and synthesis analysis.
NOTES
1Vetted studies in Pantic and Clarke-Midura’s 2019 meta-analysis always involved female undergraduate students in computer science, and included contexts of large U.S. institutions (e.g., Carnegie Mellon University, Rutgers University, University of Illinois, University of Wisconsin-Madison), small U.S. colleges (e.g., Harvey Mudd), and international university settings (e.g., Australia, India, Israel, Nigeria, a former Soviet republic); findings were not parsed by institution type.