Program Evaluation 2012 of Afghanistan National Teacher Inservice Project ()
1. Introduction
The most significant finding of this mixed-methods study is that researchers from vastly different countries can mutually and productively collaborate on universal (worldwide) educational opportunities and problems. It is not necessary for educators of one country to assume superiority when working with a cooperating nation. Certainly, international education will continue to need these findings and our lived experience (Karlsson & Mansory, 2007).
This is a final report of an educational program evaluation document. It is a brief summary of the original report of 2012 (Peterson & Ponzio, 2013) Final Report: External Assessment/Evaluation Basic Education and Gender Equality in Kandahar, sandwiched between the section Decision to Site Visit and an Afterword section using information available after the Final Report. The evaluation was of a national (Afghanistan) schoolteacher (k-12) and postsecondary inservice educational program which was at the end of its first three implementation years. The inservice program no longer exists, but it demonstrated that it should and needed to exist for a decade or longer. Because of the great change in context in Afghanistan, this report also includes two new sections. The first is a brief discussion of our decision to undertake this program evaluation, and the concluding third section reflects back on our experience and data with the hindsight of new information about the context in which we, and the program implementors, worked.
Field data were gathered over a decade ago; their value is permanent because they document a major international three-year effort to upgrade the performance of a national inservice teacher (k-12, teacher educator) development and upgrade of teaching in three areas of demonstrated high need and sociological-professional significance. The three curriculum and instruction (C&I) themes were: Reading in all content areas, Modern methods of C&I, and Methods of teaching students having Post Traumatic Stress Disorder (PTSD) symptoms. Original data were archived in-country, but the fates of personnel and archives (Appendices or Annexes) are currently unknown. Personnel range from resistance to Diaspora. From creative continuance in other governance functions such as international trade or foreign higher education.
Data were analyzed during a 31-day site visit and months after over a decade ago. The quantitative and qualitative data were test scores, surveys, interviews, videos, observations, demographic counts, cost-benefit analysis, use of a competitive comparison program, and formative and summative reports. Such a comprehensive and extensive analysis required Michael Scriven’s Pathway Analysis (Scriven) for the design, front-line data gathering, and conclusions of a client (Teacher Education Directorate of Afghanistan and UNICEF) Report (Peterson & Ponzio, 2013).
Creative innovations in this paper include: Field Application of Pathway Analysis Evaluation, multinational cooperation of program evaluators, and statistical methodology (e.g., population kurtosis and skewness in addition to parametric and nonparametric [e.g., rank order data analysis and sociological analysis].
2. Decision to Undertake This Evaluation
Ponzio and Peterson were, and are NOT, experts in international relations, the country of Afghanistan, or its government. However, it was our opinion that the need for teacher inservice was great (United Nations Educational, Scientific, and Cultural Organization (UNESCO, 2010)), and the three-year United Nations International Children’s Fund [UNICEF] program actually had an encouraging track record (UNESCO, 2009; UNESCO, 2012). The program relied upon an existing Afghan national teacher educational infrastructure (Teacher Education Directorate 2011a), received funding from UNICEF, the United States, Canada (CIDA, 2009), Germany, Switzerland, and Sweden, as well as instructional materials from these countries, and was managed by the Afghan Teacher Education Directorate (TED). The TED was a sub-organization of the National Ministry of Education, but separate in facilities, personnel, and funding.
We external evaluators had extensive program evaluation skills and projects, as well as international experience in teacher education programs, including those in Lebanon and Israel (e.g., Peterson & Yaakobi, 1980), when the two were at war. We professionally and personally knew the chief U.S. contractor serving in the administration of the Teacher Education Directorate (2011b). Authors thought, and continue to think, that we had a good opportunity to assist Afghan professional colleagues in doing their own and international education community upgrading and creation of new scholarly knowledge.
Recently, and seemingly ultimately, all of the advances of the successful original project have been reversed and negated. But we see the effort, including ours, was a good one, doing all we could do to assist their efforts. Ponzio and Peterson’s pre-site visit study of the political background was a delicate and difficult one (Aslan, 2011; Clammer, 2007; Morgenthau, 1967). Until 1973, Afghanistan had a King and Queen who had some success in keeping the factions of the country together (Braudel, 1993; Runion, 2017). Afghans have long used a loya jirga system of justice in which warlords dispensed order at the province or national level regardless of events in the many other provinces. Several national government institutions—the President, courts, legislature, and constitution—had standing, resources, and influence to support our efforts at the project (Harris & Nawaz, 2015). Finally, if the data and analysis warranted the continuation of the project, there was reason to expect documentation of the good quality of the three-year inservice project to lead to continued funding from international sources in addition to extended UNICEF funding.
Peterson and Ponzio were also aware of Afghan and outside countries’ threats to teacher development. We were aware of our own country’s (USA) previous disruptions in Afghanistan, some of which were revealed in “secret” documents made available years later. (Clammer, 2007; Coll, 2018; Coll, 2021; Coll & Entous, 2021; Rashid, 2008; Rashid, 2010). An example of cautions follows next:
Afghanistan [is] known for its dramatic mountain scenery and the unparalleled hospitality of its people. At the turn of the 21st century, the country was more synonymous with war and terrorism, the picture of a failed state. The fall of the Taliban regime in 2001 and the subsequent reconstruction attempts have done much to address this view, but in early 2007, Afghanistan’s future remained on a knife-edge. (Clammer, 2007: p. 15).
By any stretch of the imagination, Afghanistan is not the simplest country of travel in. It’s a country recovering from three decades of war, with a host of continuing problems. You’ll need to invest time in getting the latest safety information (p. 16).
Is the Afghan glass half-empty or half-full? Continued and improved international commitment is crucial for Afghanistan’s success. Afghans still welcome foreigners who come to the country to help, knowing full well the cost of neglect. [As of 2007]…international sabre-rattling over Iran fuels Afghan fears that the country will again slip off the radar. Progress is slow and painful, but possible. A peaceful, stable Afghanistan is still there to be won—the costs of losing it again are simply too high for everyone. [emphasis added] (p. 15).
Our decision to undertake this external evaluation assumed an international scholarly effort to address and solve universal academic problems of educational evaluation found in every country world-wide, in collaboration with collegial resident Educators. Our 32 days of site visits showed this assumption of opportunity to be correct. We found the quantitative data and qualitative evidence that good training had gone on, revealing the expected expert Afghan educators and resources (e.g., author Baheer’s TED computer center and staff) encouraging. Outside forces overwhelmed this smallish project relative to international relations. We got out just before danger came to our region. Later, we used one salary to buy solar panels in the U.S., which slowed imperceptibly the melting of the Hindu-Kush glaciers, which supply much of Afghanistan with water (as well as worldwide water supplies). We worked with the expected fine, bright, and creative fellow humans the same as any with whom we have collaborated anywhere in the world (though noticeably more hospitable (see also Clammer, 2007)). We saw and experienced first-hand the dedicated schoolteachers and other educators in that country. We regret we cannot say their names in this article.
We’ve been waiting to this journal technical article until it will appear after 10 years to contribute to safety as well as render honor for our many Afghan colleagues on the project.
Type of Evaluation
The evaluation was focused on the inservice program, but necessarily included the many Afghan implementors Teacher Education Directorate (TED)), the national teacher training network (Colleges of Teacher Education), and national and provincial trainers. Originals and photocopies of all Annexes (Appendices) were left in the offices of TED.
The evaluation was primarily summative for potential funders, but also formative for a program continuing to build (Scriven, 1973). Ponzio and Peterson were two Independent contractors to UNICEF from two U.S. universities. Data with nationwide implications were gathered locally and made available on our arrival. We met Baheer as Director of the new TED Computer Center and worked with him and his staff almost every day during the site visit. Teachers and administrators were personally available to us in Kabul for our site visitation. Teacher’s Colleges (Normal Schools) professors and instructors available in Kabul TED staff available for summative data; they provided unusually rich documents and measures. Unfortunately, we did not visit Kandahar Province schools for safety reasons; we were so advised by the United Nations transportation system (Whitlock, 2021). These visits would’ve been made for fidelity and credibility. Afghan hospitality, reported universally in our pre-visit study, turned out to be outstanding, prevalent, and real, as shown to us in the residence compound, interviewees, TED staff, and teachers. The author immediately began to team up with the other authors to manage the quantitative and qualitative data central to the Final Report of 2012. Afghans distinguished themselves as speakers and listeners in every personal conversation and observation we had in making the report. Next, we turn to this document to the heart of our evaluation document to UNICEF and the Project personnel.
3. Summary of Final Report 2012
3.1. Program under Evaluation Three Themes, Components, or Packages
Each training package, or component, focuses on a targeted approach to veteran teacher professional performance. The three program themes were selected for development and teacher training because of their importance in the lives of Afghan students. Each of these components has a standing in learning theory, research, professional teacher experience, and curriculum models and materials. The choice of specific individual Afghanistan Kandahar Province as the initial target evaluation area was also both representative and politically strategic. (Figure 1)
Figure 1. Components and connections of teacher inservice project.
Psychosocial Training enables teachers to increase academic learning by inclusion in teaching the whole person who does the learning. The need arises from the intense and sustained disruption of the lives of families and children in areas of continuing conflict and insecurity. In this case, teachers in the province were the initial recipients of the training. The training is to identify emotionally and psychologically vulnerable children with symptoms of Post Traumatic Stress Disorder (PTSD). The training is grounded in developmental psychology and pedagogy for every province in Afghanistan and benefits parents as well. Both students and their families benefit when teachers identify children whose academic progress is inhibited by emotional factors, but all children in a class, along with their teachers, benefit from the processes that improve the classroom environment. Cooperative learning strategies strengthen student relationships and individualize instruction without stigmatizing students who need special help.
Advanced Pedagogy extends understanding of classroom methodology and deepens awareness of the aims of education for the future of Afghanistan. Advanced concepts of goals of education. Select curricula to meet national needs and be consistent with Islamic values. Who should be the teachers of the nation’s children, and with what qualifications? Who should be educated, and how do children learn? What should teachers know, and how should they teach? What is the teacher’s knowledge base? Practical application of research-based methodology, student learning, and classroom diversity, including gender, language, socio-economic differences, and individual differences in learning styles and abilities. Students whose first language is not the mainstream language of the classroom. Methodology appropriate for different age levels, different language speakers, and different subject specialties, as well as multi-disciplinary and interdisciplinary teaching. Understanding the teaching-learning process and useful techniques and methods. Ways to plan and maximize class time and effort. Methods of class management and discipline for students to be themselves and good citizens of the class, to include all students in learning, assess learning and growth, and the human complexities of learning. Help teachers connect with students, develop healthy peer relationships, connect subject matter with lives, make classrooms cooperative places to learn, and enable students to purposefully and happily learn. These payoffs are especially important for young people whose school and community lives have been interrupted by violence.
Enhanced Literacy in reading, writing, speaking, and thinking in their studies and lives. Incorporates up-to-date materials, activities, processes, and theories. The training material is not a teacher handbook on how to teach young children to read, although there is much information that will help a teacher assist a beginning reader regardless of the learner’s age. Assign responsibility to every teacher, regardless of subject specialty, to be a teacher of reading, writing, speaking, and thinking. Enhancing literacy across the curriculum in every class. Basic teaching techniques are to be used in any classroom to approach different types of texts (narrative and/or expository) to ensure purposeful reading, resulting in comprehension and retention of information. The goal of independent readers includes teaching skills of metacognition—becoming consciously aware of how they learn, think, and deconstruct a text by questioning the author and approaching every text with questions that promote active and independent learning. Techniques include talk-aloud/read-think-aloud process, inference and visualization, and graphic organizers.
Evaluation Design was based on the Scriven Evaluation Pathway Analysis—(Peterson, 2000; Scriven, 1973). One of many innovations in evaluation by Michael Scriven was that evaluators do NOT begin with the goals of the program designers, but work first to document what the actual program is doing and structured according to evaluator’s expertise (#2 Characterization). This means that the external evaluators do not begin with the goals of the designers. Rather, they begin with their own data about what they see in front of them: the type of activity, the actual emphasis on the ground, the needs of the subjects and clients, and the obvious dynamics of the program as it exists. It is only a short time later that they add the stated goals of the program designers and implementers, and finally (#15 Evaluate the Program Goals) in valuation terms.
We did our Step #5, Clarification With Clients, at the end of the first week of the site visit. We sparred over the length of the final report: the TED Director wanted a minimum of 100 pages, while we argued for a maximum of 15 pages. The Directorate (TED) left the role and needs of UNICEF vague while we pushed for a joint meeting ASAP. We insisted on a further site visit to Kandahar Province in the South. It turned out that the United Nations daily airplane staff never intended to take us because of extreme danger. We were not encouraged to video record our visits, we insisted and did. TED asked us to log our daily agenda and completed activities, we complied. During the last week, they used our log entries to write our contract of service. They were able to show a completed and fulfilled contract to the Directorate of Education. Two-thirds of our final meeting with UNICEF officials consisted of quarreling until we interrupted for a 20-minute report with a draft edition.
After we began designing goal-free, we added “first do no harm” to Scriven’s SPA methodology. It was clear that we external evaluators needed to quickly get up to speed on the major realities and stakes of a major, important, necessary innovation here. Of course, this is a judgment (subjective) call to the overall impact, value, and merit of the evaluation itself. For example, we saw the crucial role of Leave-behinds (infrastructure) and printed materials, which were clearly key features. Another strength of Pathway Analysis is that the site-visit begins with evaluators not knowing the designer’s or implementor’s GOALS, but to begin the evaluation looking for stakeholders, their priorities, and the needs that are being addressed in actuality. List 1 is our SPA guide:
List 1: Scriven Pathway Analysis
1) Strategizing—Why is the Program evaluation to be done? (For example: Decisions for program continuation? Generalization to multiple settings?) How can the evaluation tasks best be sequenced and designed to achieve purpose? What should be done first?
2) Characterization—What is the nature of the program? What does it emphasize and include? [At this point, evaluators avoid the program goal statements (i.e., “goal free”)., but use the general expectations of programs like this one under review; e.g., cognitive, affective, sociological, communication, behavioristic].
3) Client Analysis—What are the characteristics of the clients of this program? [described by the evaluator, not the program planners]. Who are the stakeholders for the program or the evaluation. Gather or perform needs analysis of clients—students, teachers, administrators, funders.
4) Identify Interesting Comparison Programs—Other similar programs in similar settings. If necessary, create ideal comparisons & competitors (synthetic or inexpensive versions).
5) Clarification with Clients—Review the purposes, needs, and assumptions of the evaluation of, and for, the clients of the evaluation—students, teachers, and administrators (their ideas and those of the evaluator).
6) Identify Consequences—Document results, effects, outcomes, payoffs, and performances for teachers and students. Includes quantitative and qualitative data: performances, perceptions, and quantities. These should be gains or changes and not merely final performances. Describe the educational (“real life”) significance of gains in addition to statistical significance. What is the absolute size and nature of the gains? Consequences should be in actual settings—not just laboratory, hot house, or pilot special applications.
7) Estimate Durability of Consequences—Initiate long-term data gathering and analysis of the effects of this Program on student behavior/learning and teacher behavior/attitudes.
8. Personnel Management—Hiring, supervision, development, supporting, evaluation, dismissal. [In the Kandahar Project, this applies specifically to the Trainers and Coordinators at all levels (Mitchell, 2012)].
9) Comprehensive Cost Data—Document the costs of the program: installation, maintenance, materials, time, morale, personnel, and displacement. Document costs of evaluation.
10) Identify Side Effects—What unintended outcomes happen? Positive or negative.
11) Comparisons with Interesting Programs—Include steps Six through Nine for the other Programs.
12) Estimate Value of Consequences—Independent expert assessment of the importance, significance, and magnitude of the results of the program.
13) Identification of Justice Issues—Equity, opportunity, treatment, and protection concerns for students, teachers, administrators, and the community.
14) Conclusions and Communications—Write ups, presentations, dissemination.
15) Evaluation of Program Goals—Clarify and revise with program planners based on evaluation findings (Ministry of Education, 2011a; Ministry of Education, 2011b).
This Scriven Pathway Analysis led to a number of questions (List 2) we asked of the data. List 2 gives us the guiding questions to answer with data and documents as we proceed through the 15 “Steps”. Evaluation is subjective; good evaluation uses the best objective evidence available, controls bias, has explicit logic, and involves the interests of all parties.
List 2: Evaluative Questions for the Teacher Inservice Project
Should the project have been done?
Should it have been done this way?
Was it well supported?
Was it well organized?
Was it well staffed?
Was it well done?
Did it make a difference?
Are the Participants Better Off? Are the students learning? Are the teachers better? Are the Trainers better? Are the materials better? Are the schools better?
Will it make a difference in the future? In the long run?
Is there a better way to make a difference?
Are there better materials with which to make a difference?
Are the “student gains” defensible?
Is it “cost-effective”? Is there “value for the money”?
Has it “increased capacity” in the educational system? What does it “leave behind?”
Are there better programs of this kind?
How should those responsible continue to monitor and evaluate?
How should the personnel be evaluated?
3.2. Evaluation Research Design
The research design for this assessment and evaluation report is Mixed methods (Cresswell & Clark, 2007) in which verbal and visual descriptions are combined with quantitative and qualitative data and evidence to document the intent, progress, and outcomes of the Teacher Inservice Project.
Much of this report is a Meta-Analysis in which the main descriptions and data are taken from reports and findings of the project itself. This is possible and appropriate because this project was well-documented and monitored. The function of external assessors and evaluators is to examine and validate existing reports, methods, and data for credibility, accuracy, and utility. Not using these existing resources and products would be inefficiently ignoring the good products already invented, developed, and translated into several languages, and it would cause unnecessary expenses. At this point, the task is to sample, validate, and cross-check the reports and instruments used to make them. This is done by interview, observation of instructional sessions, focus groups, inspection of completed documents (forms, surveys, tests, observation reports, demographic descriptions), and deeper statistical data analysis (e.g., reliability estimates and factor and kurtosis analysis to examine reasoning by those who complete the forms and reports).
3.3. Field Work, Site Visit, Sample Workshops, Guests, Focus Groups, Interviews, Document Reviews, Data Analysis
Figure 2 shows many of the monitoring lines operating from early on in the Project. These devices range from planning meetings, surveys, and training session observations to direct observations and reports of teachers and students while classes are in session. These all were valuable built-in internal features of a well-functioning project and organization. We were given a large sack of original forms, tests, questionnaires, surveys, and observation reports. The unbound thousands of individual papers were in clumps of like documents. On the first day, with the help of a fluent translator and our office director, we were able to sort them into piles and identify each set. Also, on the first day, we met our office assistant, who heated the large room, brought our wonderful local lunches, and got us boiling water for the first week of weak green tea and then three weeks of filtered coffee. Our internet connection worked for several hours per day; we met the gracious and available staff and professionals of the Teacher Education Directorate (TED). We also were treated to the assembly of the TED Computer Center, which began a daily productive collaboration.
![]()
Figure 2. How the teacher inservice project was monitored prior to site visit.
We had four foci for making judgments: Need is the case that there was reason for conducting this training effort at all (especially for the primary stakeholders—the Afghan youth (Agency Coordinating Body for Afghan Relief, 2008). What evidence compelled the designers and funders (and all stakeholders) to make the effort? Value is the importance of the project and its outcomes to the stakeholders and larger Afghan society (Barber, 1995; Fukuyama, 2004). Merit addresses the quality of the materials, concepts, delivery system, outcomes, and personnel conducting the teacher inservice Project (Scriven, 1971; Scriven, 1973; Scriven, 1974).
3.4. Examples of Data Gathered by Program, Examined and
Analyzed by External Evaluators
This paper (2022) presents examples of all of the data collected, analyzed, and summarized for the project. The complete set was presented in the original report (2012) and archived in appendices and original documents left at the Teacher Education Directorate (TED) office. For example, in addition to summary statistics, we did factor analysis for each questionnaire, survey, and examination that was available. But in this present version (2022) paper, we include only one example to show our overall strategies for program evaluation, which we reported on in the 2012 Final Report.
3.5. Demographic Data (Table 1 & Figure 3)
These reports include demographic data about:
Students: numbers, ages, class size, genders, groupings
Teachers: numbers, subjects, schools, grade levels served
Trainers: numbers
Trainers of Trainers: numbers
Sites: Training, schools, city vs. village: numbers
Table 1. Training of teachers in psychosocial component: example of one region of implementation.
Level of Training |
Gender |
% Female |
Days of
Training |
Dates of Training |
Batch 5 Teachers |
18 F,
656 M (674) |
3% |
15 |
Oct 20-Nov 3, 2011 |
Batch 4 Teachers |
70 F,
676 M (746) |
9% |
15 |
June 25-July 11, 2011 |
Batch 3 Teachers |
178 F,
422 M (600) |
30% |
15 |
Feb 26-Mar 14, 2011 |
Batch 2 Teachers |
125 F,
475 M (600) |
21% |
15 |
Oct 28-Nov 14, 2011 |
Batch 1 Teachers |
66 F,
114M (180) |
37% |
15 |
Various |
TOTALS |
2800 teachers 457 Women |
16% |
|
Sept 26-Nov 3, 2011 |
Figure 3. The actual paper trail used in the Kandahar project: tracking need, training, impact.
Reports All Three Components Preliminary Report on Classroom Observation Data—Kandahar, Wardak, & Other Provinces, 2010, total classroom number = 400, 90 from Kandahar,
40-item checklist Psychosocial Training package, schedule, and content Overall evaluation by monitors Psychosocial Project for Afghan Teachers, Preliminary Report for UNICEF, February 2011 Preliminary report (June 2011) Monitoring Report: Psychosocial Project, July 5-14, 2011 Number of Core Trainers Trained Report of Classroom Observation Data Advanced Pedagogy Advanced Pedagogy Package, December 9, 2011 Training of TED Core Trainers in Advanced Pedagogy 6/9/12 to 6/27/12 Advanced Pedagogy: Final Evaluation of the Workshop by the Participants) Literacy Literacy Chapters in English, July 2012 Literacy Training Manual, September 2012 Afghanistan, Needs Assessment, November 2012 Enhanced Literacy Kits |
3.6. Examples of Data Analysis
Reports All Three Components Preliminary Report on Classroom Observation Data—Mixed Provinces, 2010, total classroom number = 400, 40-item checklist, (Annex O) Psychosocial Training package, schedule, and content Overall evaluation by monitors Psychosocial Project for Afghan Teachers, Preliminary Report for UNICEF, February 2011 (Annex U) Preliminary report (June 2011) Monitoring Report: Psychosocial Project in Kandahar Province, July 5-14, 2011 (Annex W) Number of Core Trainers Trained Report of Classroom Observation Data, Kandahar Province Advanced Pedagogy Advanced Pedagogy Package, December 9, 2011 Training of TED Core Trainers in Advanced Pedagogy 6/9/12 to 6/27/12 (Annex F) Advanced Pedagogy: Final Evaluation of the Workshop by the Participants (Annex J) Literacy Literacy Chapters in English, July 2012 Literacy Training Manual, September 2012 Afghanistan, Kandahar Needs Assessment, November 2012 Enhanced Literacy Kits |
Figure 4. Total number of correct scores by item—post-instruction knowledge test.
Figure 4 shows the number of correct responses for each item in the 25-item multiple-choice test of knowledge used for the post-test of Advanced Pedagogy. Items with the highest scores were: #2 “list used to measure quality is ‘standards to establish expectations for primary teachers’”, #1 “task force created eight standards”, #3 “highest teacher priorities are for subject matter competence and teaching methods”, # 12 “physically handicapped students should be taught in regular classrooms”. Items with the fewest correct answers were: #16 “diverse minority, language, Islamic groups understand and adapt for learning”, #9 “students are encouraged to work together, support and assist each other”, #6 “base student grades on several different assessment methods, and #11 “intelligence not fixed at birth, but influenced by experience and opportunities to learn. Future instruction and materials should strengthen low-scoring concepts.
3.7. Item Number
Internal Reliability (Cronbach’s Alpha)
We used this measure of internal consistency of items to see how closely a related set of items is as a group. This gave us one clue that the form was getting a pattern of consistency to match what the designers claimed (overall knowledge, attitude). Also, we got clues about the population who completed the document. Did they show a consistency that indicated they understood what they were reporting, and was there a plausible range of responses that were predictable or explainable? We were quite familiar with these documents in other evaluations of populations we knew well.
Population Structure (Skewness, Kurtosis) Analysis
Knowing the population of teacher and supervisor subjects’ tests, self-reports, surveys, and demographic data is important for several reasons. We needed to be sure they understood what they were asked, how they compared with other nations and cultures for norms, and even if the returned documents were faked by supervisors or administrators (fidelity). We used the distribution of ratings and scores in the U.S. in novel applications (not reported in the literature by other authors) so we knew what patterns of values meant. We found no instances to doubt the results of Afghan subjects. Author Baheer had Fortran programs written in the TED mainframe for these purposes of large scale analysis.
Factor Analysis (Principal Components Analysis, Varimax Rotation)
We have used factor analysis as a quantitative tool in the educational program and personnel evaluation since 1986. Factor analysis reduces a large number of variables into a smaller number by first finding common variance from all underlying observed variables, and then putting them into a common score (Kaiser & Cerny, 1979; Kim, 1978;). First, it developed to reduce the number of items in tests by discovering the items that best represent the actual items stating the underlying concept, knowledge, and understanding while dropping those with the least representation. We found that subjecting even questionnaires and population demographic data to factor analysis was useful in understanding dynamics of the population of subjects under analysis. If we knew the patterns of similar analyses in known and trusted populations, we could infer judgments about new populations’ data on the same topics, representation, and even veracity. For example, it is possible to find fake results from fake data factor analysis.
A factor analysis was also performed on the participant evaluation of the Enhanced Literacy training, in this case, Enhanced Literacy. This information is given here to provide an example of the depth of the assessment of the quality, validity, and reliability of the feedback forms used in our analysis of the various trainings, surveys, and tests. It is not necessary for the Project to continue this deep statistical psychometric analysis in its routine monitoring and formative evaluation activities in the future. This discussion is to provide a foundation for understanding the evaluation process used, and for having confidence in the outcomes of this program review and its recommendations. Figure 5 shows the factor analysis for one piece of the Kandahar Program, a teachers’ survey of their experience with the Trainers.
Figure 5. Items with the highest loadings on each of the eight factors with eigenvalues > 1.0.
This strong and distinct factor structure, with mathematical and logical (item topic) agreements, means that 1) the instrument was a good one with distinctive and pertinent items, and 2) the teachers took it seriously and responded with purpose and directness. These patterns matched population patterns we have seen in similar examinations with populations we had seen before. This analysis presented in Figure 5 was only of one survey, but the other tests, surveys, and questionnaires analyzed showed similar strong factor analyses and led to the conclusion that the program was well monitored and that there was a conscientious effort to plan and use the feedback from the training evaluation instruments (surveys) to provide formative improvement in the training.
Needs Analysis
All three Project components included a participant teacher needs assessment survey. This gave workshop designers and instructors a last-minute glimpse of what content was most effective and needed by participants. These surveys can also be used to refine training materials in the future.
#12: It would be important and useful to have a school-wide plan for improving reading and writing skills across the curriculum.
#20: It is, or would be, useful to have access to computers for learning in classes.
#17: It would be useful for me when someone models a strategy to show me how to use it in my class.
#10: It is important for me to have adequate time for the teaching of reading in my class.
#7: I know enough strategies, so I use two or more strategies to teach reading skills.
#19: The principal already shows enough interest in supporting reading across the curriculum.
#9: I have time to teach subject reading skills, such as how to read for meaning, how to interpret vocabulary, and how to look for key ideas.
#3: Textbooks are available to somewhat available for most students in 2 languages.
“Internal reliability” (Cronbach’s alpha) is expected to be low for a needs assessment instrument because it intentionally assesses an entity (need) which should differ in each individual. The reason for doing this analysis is to see that the prediction of lower reliability for this instrument matches the prediction of higher reliability for a survey evaluating a performance in which we are looking for persons to agree on their judgments of the performance. In this case, we found the expected low value of 0.35.
“Factor analysis” also shows multi-factor solutions because of the great variety of needs.
Findings, Recommendations
1) An internal review of the materials production and supply line should be made to identify the reasonably expected problems and solutions to the rapid and large implementation of the program. Workshops, at times, were hampered by delays in the delivery of training materials, implementation manuals, and take-home information for participants.
2) Keep at the materials revisions and translations. They are good for first efforts. However, the task of producing useful and accurate reference materials for people at many locations and with varying levels of literacy and usage calls for a series of upgrades based on feedback from the audiences, including those in the field.
3) An Introductory Level of all training materials be produced. This may be as elementary as a “cartoon” version of key instructional ideas for psychosocial expectations for teachers, fundamental teaching skills, basic psychology of learning, and ways to teach reading, writing, thinking, and speaking. Some of the teacher audiences are meeting these ideas (or are given permission for their own initiative) and putting them into practice. Some teachers are polite in not criticizing training materials that are too advanced for beginners.
4) It is strongly recommended that Focus Groups be regularly and widely used as a means of formative and summative evaluation projects of this type. This will have the benefit of quick feedback, generation of unexpected and vital problem-solving, brainstorming, and outlet for teacher and Trainer creative experiences and disappointments. Focus groups can comment on proposed materials and activities.
5) Infrastructural support lags behind field implementation. There were reports of payments delayed, computer data base development slowed, delays in printing, holdups in materials delivered to training workshops, and (inevitable) electrical and internet outages encountered. While some of these should be expected in a new program at this scale, infrastructure problems should be taken into account by stakeholders and reviewers.
6) Consideration and planning should begin to continue this training and its attendant materials and personnel. The substantial materials should be edited, expanded, and added to.
7) Investigations into how to expand local use of Project ideas and materials should be made. In particular, the many “teacher learning circles” (over 5000) and “principal learning circles” (or some equivalent) should be investigated as arenas for teachers and administrators themselves to expand and locally develop Project initiatives.
4. Decision to Evaluate, Afterwards, and Conclusions
We were, and are, not experts in policy and politics. This Part of the Report reflects our thinking and analysis since parting company and collaboration with our Afghan sisters and brothers. Since the original work, there have been a number of informative publications. For example, Ruttig (2021): Some things got better—How much got good? A review of 12 years of international intervention in Afghanistan, Coll (2018): Directorate S: The CIA and America’s Secret Wars in Afghanistan and Pakistan, and Whitlock (2021): Afghanistan papers: A secret history of the war. Since little scholarly work was done in Afghanistan after the time of our site visit, we have had to rely upon highly reputable Journalists, whom we have found to be uniformly in agreement (Anderson, 2022; Coll, 2021; Coll & Entous, 2021; Gopal, 2021; Jolie, 2022; Junger, 2022; Kermani, 2021; Power, 2019; Ward, 2021). For example, Anderson (2022: p. 51) reported that former Afghan President Hamid Karzai (2002-2014) said that U.S. positives were increased education, especially among women, and improved roads. Also, Junger (2022: p. 135) reported that Afghanistan had “…educated a generation of Afghan children, and helped bring the country into the modern era”, but also that “It is abundantly clear that four successive American administrations utterly betrayed the public trust”; We choose to appreciate the former positives, and continue to work and speak against the latter negative.
We were particularly struck by the great contribution and cost of the women involved in this project. We did not mention them in the 2012 report except to document the remarkable increases in female participation it required, prepared for, and, in fact, achieved. At our first inservice observation, we saw two male Afghan teachers hold a sign: “We love you {name of U.S. woman educator working in TED}”. But we also heard a male Afghan teacher berate one of the women presenters, whom we observed deliver a wonderful modern lesson. At a focus group meeting of three male/three female teachers, we heard from the women that they had no toilet facilities for a day-long workshop; it took us three weeks to carefully find out how they coped (they expected such treatment and were fully prepared for it). The Project was functioning so well and was so much needed that we decided not to mention that we had seen these things because the continuing project was so much needed. As in the history of the United States, few professions so benefited from women at the same time as exploiting women. Worldwide, humankind has more work to do. The outstanding Journalist Ward’s (2021: p. 222) words helped us understand and corroborate our experiences and program evaluation observations:
It is one of the great privileges of being a Western woman working in a conservative Muslim societies that I am allowed to pass freely between the two spaces, between the two genders. I remember someone joking in Afghanistan that female war correspondents were to be treated as honorary men. But we were also able to enter the intimate and private space forbidden to our male colleagues. Wherever I have gone, I have found women to be a fountain of information. They know what’s going on in the village and what the power dynamic is. Broadly speaking, they are less likely to lie than men and much more likely to feed you.
The atmosphere in the women’s room was completely different. Here, the fear was palpable. No one attempted to repress or avoid it, so there was far less tension. One woman was rocking back and forth in the fetal position, clutching a pillow. Another woman looked at me with her tired, bloodshot eyes. ‘She does this when the shelling starts,’ she muttered.
4.1. Documents Ten Years Later as We Write This
We selected the following accounts in Part III to include information on the context with which we concur. We selected statements to substantiate a significant distance between national intent, status, and what was popularly known about developments in Afghanistan. In general, we have not found contradictions to these substantial doubts.
“The CIA had no institutional view of what should be done in Afghanistan after the Taliban” (Coll, 2018: p. 116) in 2001 after a clumsy “war”. We did not much more report on politics because that was not our expertise. However, politics played a role in the acceptance (!) we saw of the Inservice Program and materials, as well as potential difficulties (slow improvement, payoffs) in the future. The United States military appeared to us as heavy handed as we driven daily to our work and spontaneously in interviews of educators who were Afghan citizens. In particular, our unreported reaction in the interviews we conducted was that “the U.S. soldiers were too psychological”. They deliberately humiliated Afghans when they interacted with them. For example, they dominated senior male members of family groups in front of women and children. We kept these conclusions only in our notes, not reports. However, journalist Ward (2021) wrote the following about her related experiences as a reporter in Iraq. We heard these views corroborated in more than half of our interviews with key players in the construction and implementation of the inservice program:
Dogs were howling and the stench of raw sewage seeped into the Humvee through the gunner’s turret. “We call this place shit’s creek,” one of the soldiers grunted. The first raid was quick. The soldiers stood outside the house and counted silently to three before bursting through the door. Dazed Iraqi men with sleep in their eyes stumbled out of bed, squinting into the bright flashlights shone into their faces and trying to shield the women who were not wearing headscarves. “Where’s the terp?” a soldier asked for the interpreter. “I’m here,” came a timid voice. Tell the men to get down on their knees,” the soldier barked. Some of the women began wailing. The interpreter hesitated. “I said tell them to get down on their goddamn knees,” the soldier shouted again. And kneel they did, on the ground facing a wall as the soldiers methodically handcuffed and blindfolded them and then began searching for weapons. There were no beds, just a mass of carpets and mattresses and blankets spread across the floor. The wailing was getting louder now as more women joined the chorus and a child had started crying too. “Tell them we are not going to hurt them or touch their women, we are just looking for weapons. Do they have any weapons in this house?” The interpreter haltingly translated for the men. But the women just kept wailing. It was unnerving, for me and for the soldiers. “Goddam it, get those women into the other room.” I looked at the children, some sobbing, some blinking in confusion into the bright light. Their cheeks were chapped from the dry cold, and thick snot poured out of their noses. My head was throbbing under my helmet. No one was communicating. The soldiers were hopped up on caffeine drinks and were nervous themselves. The Iraqi men were humiliated and angry, kneeling on the floor; the women were frightened and vulnerable, hastily trying to cover their hair; and the children seemed to be in shock, confused by all the screaming and chaos. (Ward, 2021: pp. 56-57).
This view of U.S. soldier brutality was confirmed in The Afghanistan Papers (Whitlock, 2021) repeatedly, including this excerpt:
In August 2002, an unusual report from the war zone caught the attention of Rumsfeld and other senior officials at the Pentagon. Written by a member of a team of allied commandos hunting for high-value targets, the fourteen-page email provided an unfiltered, firsthand account of conditions in southern Afghanistan. “Greetings from scenic Kandahar…Formerly known as ‘Home of the Taliban.’ Now known as ‘Miserable Rat-Fuck Shithole.’ (Whitlock, 2021: p. 17)
Kandahar was the province from which most of our highly credible data, focus group, interview, and inservice training observations were taken. We stand by our report and the hundreds of pages of photocopied documents we left at the TED in Kabul.
The educators we worked with were as good as anywhere else in the world with whom we had collaborated, including our own university institutions. We were only prepared for the most brilliant examples because our evaluation theoretical mentor was Michael Scriven. We saw one Afghan person, in particular, energize an audience about this project in a way only a verbal culture can do; the call and response was something with which we were familiar because of African American Methodist Episcopal churches in our own country. We talked privately with the individual Educator about the views in our country that confused political movements in Afghanistan with “tribalism”, Province differences rather than national aspirations, and our own failures with multiculturalism, misogyny, and international relations. All of these issues went unreported, except now in “first, do no harm”. We haven’t cleared that with Michael Scriven, but his overwhelming emphasis on evaluator responsibility to subjectively consider A.) Justice and B) Client Need gave us the direction.
4.2. More Excerpts from Craig Whitlock
“Greetings from scenic Kandahar,” it began. “Formerly known as ‘Home of the Taliban.’ Now known as ‘Miserable Rat-Fuck Shithole.’” (p. 17) A 38-year-old Green Beret with atypical credentials.
“We originally said that we won’t do nation-building but there is no way to ensure that al-Qaeda won’t come back without it, we just don’t have a post-conflict stabilization model that works. (p. 36). (Stephen, 2018)
“Of all the failures in Afghanistan, the war on opium ranked among the most feckless. During two decades, the United States spent more than $9 billion on a dizzying array of programs to deter Afghanistan from supplying the world with heroin. None of the measures worked. In many cases, they made things worse.” P254 voting system, judicial system, legislature, national bank, medical system, education, teacher training colleges, food distribution, water systems NGOs, international.
By 2007, “Afghanistan was nothing…it was a backwater second effort for everybody.” (p. 109) Richard Phillips, combat support hospital in eastern Afghanistan.
4.3. Neoliberal Critics: The Tyranny of Experts
We specifically reject the do-nothing, hypercritical, neo-liberal view of anti-expertence.
Economist Easterly (2014) “Forgotten Rights” radical critique of efforts such as ours of Western educational experts going to Afghanistan, focusing on the material suffering of the world’s poor. Ignoring the political oppression that create the problems in the first place. Even well-meaning unintentionally side with dictators against their subjects. Without a focus on liberty, freedom, equality, rights, or democracy.
Instead technocratic or autocratic, bureaucratic centralized government overcontrol, while avoiding or even prohibiting the language of human rights to liberty, freedom, equality, or democracy. Strengthened approach to governance or Consideration of budget and staffing implications, systematic planning and evaluation rather than the clearer, more robust language of men born and remain free and equal in rights and free development, spontaneous problem solvers, individualized autonomy, large scale centralized funding (such as world Bank and United Nations).
The “do nothing” point of view seems to be: Wait until the Afghans themselves, and the market take advantage of freedom, individuality, and spontaneity to create inservice or new public Afghan schoolteachers to come up with new practices for students with PTSD, new methods of indirect instruction, improved ways to teach reading, and most of all, so-called “experts” should not be given visas to impose their planned, large scale, data-based interferences from the United States. Or in other words: “do nothing”.
Steve Coll (2021) put into words precisely our sentiments reflecting on our decade-ago evaluation, and the current developments in Afghanistan.
It would be unfortunate if the lesson America draws from its Afghan debacle is that it should forswear large investments in human dignity and health in very poor countries. The climate crisis and the pandemic make plain that we face new border-hopping threats to our collective security. For both moral and practical reasons, the United States has cause to provide substantial humanitarian aid to troubled nations and even, in a supporting role, to strengthen their security—perhaps having fashioned a foreign policy, if it is not too much to hope, informed by a measure of humility and a capacity for self-reflection. (p. 14)
And Samantha Power (2019) spoke for us, calling for “…a clearer eye, a kinder heart, and a more open and civil hand in our politics and daily lives.” She also reminded us of Wm. Shakespeare Julius Caesar:
There is a tide in the affairs of men,
Which, taken at the flood, leads on to fortune;
Omitted, all the voyage of their life.
Is bound in shallows and in miseries.
On such a full sea are we now afloat;
And we must take the current when it serves,
Or lose our ventures. Act IV, Scene III.
Our venture, which we report here, was an international collaborative investigation into the problems of evaluating any worthwhile project, program, materials, or personnel. As journalist Ward (2021: p. 300) quoted her local aide Salma as saying: “Let’s try to see what can be done”.
5. Projections
The Afghan people have a long history of peaceful life and government. They have lived long and well among other nations. Afghan citizens have been interrupted by outsiders because of their geographical location as a trade route for minerals, food, garments, animals, and technology. The Afghanistan Nation-State, like many other countries, has been shaped by outside political forces. We visited a National Museum of impressive and admirable examples; however, it was a tiny remnant after the external looting of generations of aggressive foreign invaders.
In spite of these challenges, over many centuries, Afghans have excelled in local government, law, music, art, and education. Much of their success can be attributed to Islamic institutions and traditions, as well as unique tribal and family strengths. The overwhelming hospitality and safety we enjoyed from a variety of political and economic movements was palpable. See Rahim & Shah Kakakhel Marwa, 1993; Smith, 1958; Soroush & Roehrs, 2015.
We have experienced threats to education, democracy, and humanism at home in the West. But we expect and project a worldwide future strengthening in West, South, East, and North. The forces for optimism derive from family and neighbor affection. Humankind is not all power politics and evil. It longs for a place called Home. A place where everyone is known and counted, where everyone belongs, is accepted and heard, where the first educations take place, where memories are made, and hope resides.