Hierarchical Aggregate Assessment (HAA): An Assessment Process of Teams with Several Levels of Hierarchy in Education

Abstract

Usually, the assessment of teams with several levels of hierarchy is done in the Management field with Management Information Systems (MIS). The problematics studied in the present paper is to consider the assessment of teams with several levels of hierarchy in the field of Education. Regarding this issue, very few authors and scientists have done work on these teams that have an inverted treelike structure similar as large organizations. Both teams studied in the Management and Education fields are evaluated by the processing of the information contained in each node of the three or by the work, production and performance of each individual with three traversal algorithms. In that particular case, links and similarities are established between Management and Education fields because both assess treelike structure organizations with several levels of hierarchy. The concept of Hierarchical Aggregate Assessment (HAA) is based on the assessment of teams with several levels of hierarchy in education. This assessment process is done on a treelike organization similar as the ones in management information systems. The process is done in three phases: the first phase consists in the team formation and the attribution of hierarchical levels to team members that is the aggregation process; the second phase is the presentation of a test or an assessment task done in team to the student; and the third phase is the team dislocation and the return to the initial phase until the course is done. This iterative process consists of the course curriculum management. While the process iterates, assessment data are collected through the process as summative and formative assessment data that can be used to determine the course success or to guide student for improvement. The aim of this paper is to define the HAA process in education that is similar to Management Information Systems (MIS). There is a lot of research and literature produced on Management Information Systems and also on teamwork assessment. In education, most of the research concerning teamwork assessment has been done on teams with a unique level of hierarchy. The main measurement tools to assess team in education according to previous research are team leaders and team member’s assessment grids. To explore this field of research, an E-Learning Internet application named “Cluster” has been developed with a research and development (R & D) methodology and tested with high school students and Army Cadets. Resistance to change has been a major obstacle to the implementation of the “Cluster” application in organizations. Knowledge acquisition rate was similar as traditional classroom teaching but failure rates were 20% in traditional teaching and 80% in the case of distance learning with “Cluster” application. However, despite resistance to change, the “Cluster” application proved the HAA theory that teams with several levels of hierarchy could be assessed in an educational context.

Share and Cite:

Lesage, M. (2016) Hierarchical Aggregate Assessment (HAA): An Assessment Process of Teams with Several Levels of Hierarchy in Education. Creative Education, 7, 1974-1994. doi: 10.4236/ce.2016.714200.

Received 7 June 2016; accepted 28 August 2016; published 31 August 2016

1. Introduction

The actual research finds its origin in the field of teamwork evaluation. This field has two subfields that are the teamwork evaluation in management and the assessment of teams in education. The automation of these processes gives Management Information Systems (MIS) and Hierarchical Aggregate Assessment (HAA) systems that is the assessment of teams with several levels of hierarchy in education. HAA systems are able to present complex assessment tasks in teams with several levels of hierarchy as the assessment of the classroom’s newspaper, the assessment of navigation patrols, the assessment of education students’ teaching internships and group counseling assessment. HAA systems also can assess different knowledge, skill and abilities because the assessment criteria are related to the hierarchical position of the team member. Therefore team members are assessed on different criteria than team leaders and team managers. Both management and hierarchical assessment fields are evaluating a treelike structured organization. Hierarchical assessment process is applied everywhere and teams have several levels of hierarchy. This process could execute itself either manually or automatically with computerized algorithms executed on computer or Internet servers driving WIFI applications. This process finds its origins in the management field where it is applied since the human race worked in teams as part of large organizations. This process surely has been executed by Julius Caesars’s generals to assess the combat effectiveness of their soldiers and their officer’s leadership to lead troops in combat (Lesage, Raîche, Riopel, Fortin, & Sebkhi, 2015) . One of the goals of this paper is to make the term “Hierarchical Aggregate Assessment (HAA)” be recognized by the scientific community.

This states the problematics origin of the HAA where the evaluation of teams with several levels of hierarchy has been mostly studied by management and information systems researchers while very few work has been done on teamwork assessment with several levels of hierarchy in the education field even if these tasks could be performed in a classroom during professional training. HAA process is not very suitable with fundamental assessment paradigms as classical test theory, generalizability theory and also test validity and reliability because a single complex assessment task in HAA could present items with different assessment criteria according to the hierarchical position of the team member, as shown in Figure 1.

The HAA field defines itself as a subfield of teamwork evaluation. Teamwork evaluation is part of both management and education domains. So the HAA field is a common field of education and business administration domains, as shown in Figure 2.

A large amount of work has been done on the evaluation of large organizations in the field of management by Management Information Systems (MIS) (Laudon & Laudon, 2000; Laudon, Laudon, & Brabston, 2011) . Almost no research has been done for the assessment of large organizations in the field of education. However, the need to assess large organizations exists in education where teamwork assessment could be implemented by presenting complex assessment tasks in collaborative mode to teams with several levels of hierarchy. Some examples of this type of complex assessment tasks could be the assessment of the classroom newspaper, the assessment of navigation patrols in teams, the assessment of education students’ teaching internships and the assessment of group counseling. The main theory and examples on assessment systems with several levels of hierarchy in the field of education are found in the work of Lesage, Raîche, Riopel, Fortin and Sebkhi (2015) .

Figure 1. The field of hierarchical aggregate assessment.

Figure 2. Similarity between management information systems and HAA.

The theories and research produced by the actual project are an extension of previous work made by Nance (2000: p. 298) . Nance (2000: p. 298) is using a similar aggregation process as the Cluster application to form teams with several levels of hierarchy for educational purposes. This to manage project teams in software engineering courses. The actual research also finds its origin in the work of Freeman and McKenzie (2000, 2002) on the development of the “SPARK” software application that is an Internet distance assessment system for the management of self-assessment and peers assessment produced with assessment grids. Other teamwork assessment applications are MLE developed by Marshall-Mies, Fleischman, Martin, Zaccaro, Baughman and McGee (2000) , OTAS (Undre, Sevdalis, Healey, Darzi, & Vincent, 2007) and MEGA CODE developed by Kaye and Mancini (1986) . The SPARK, OTAS, MLE and MEGA CODE application are used in the field of education but they assess only one level of hierarchy. Only the application made by Nance (2000: p. 198) assess students with two levels of hierarchy but the hierarchy and team grouping is not managed by the software as in the “Cluster” application. Instead, the teams are grouped manually and the collaborative work is based on E-Mail. The domain of HAA first situates itself in the field of management and its computerization is in the field of computer science. However the actual research also wants to situate this process in the field of education through competency approach with the presentation of complex assessment tasks in collaborative mode in authentic context.

This paper will prove that a link exists between the fields of management and education by the assessment of teams with multiple levels of hierarchy. A lot of work can be done now in these two fields with 1) the production of Management Information Systems (MIS) that could assess the knowledge of the members of large organizations, 2) the production of complex assessment tasks in collaborative mode that could evaluate performance and productivity of students, 3) the production of simulation scenarios to assess managers in formation; and 4) the production of electronic portfolios for students, teachers and managers. The future might predict that research in the previous domains could produce significant new knowledge.

2. Management Information Systems (MIS)

A management information system (MIS) can be “technically defined as a connected set of components that stores (or captures), processes and distributes information to support decision making and control in organizations. In addition to supporting decision making and the coordination and control, information systems can help managers and employees to analyze problems, visualize complex concepts and create new products” (Laudon, Laudon, & Brabston, 2011: p. 13) . The system “uses computer equipment and software, databases, manual procedures, models for analysis, planning, control and decision making” (Davis, Olson, Ajenstat, & Peaucelle, 1986: p. 6) . These systems may contain information about the function, department and the hierarchical position of the members of the organization that are stored in hierarchical databases (Burch & Grudnitski, 1989: p. 398; Davis & Olson, 1985: pp. 522-523; Davis, Olson, Ajenstat, & Peaucelle, 1986: pp. 64-78; Laudon & Laudon, 2000: p. 234; Laudon, Laudon, & Brabston, 2011: p. 177) . Some authors such as Kanter (1984: p. 98) indicates that the employee file can be sorted by order of position or assignment to identify employees who have the same hierarchical position. A database diagram illustrating an employee’s position is shown in Figure 3. A HAA software application is therefore a Management Information System (MIS) where the employees to manage are students who have a hierarchical position.

3. Hierarchical Aggregate Assessment (HAA) Concept

The HAA concept is the assessment of teams by the presentation of exams or assessment tasks to be done in

Figure 3. The record of an employee in a management information system database (from Laudon, Laudon, & Brabston, 2011: p. 177 ).

teams with multiple levels of hierarchy. This concept is part of the integration of computers to support teaching, the E-Learning, and the assessment fields of education. The HAA concept is also based on an aggregation process, course curriculum, portfolio assessment, assessment methods and a link with Management Information Systems (MIS), as shown in Figure 4.

3.1. Definition

The concept of HAA is based on the assessment of teams with several levels of hierarchy in education. This assessment process is done on a treelike organization similar as the ones in management information systems. The process is done in three phases, the first phase consists in the team formation and the attribution of hierarchical levels to team members that is the aggregation process, the second phase is the presentation of a test or an assessment task done in team to the student and the third phase is the team dislocation and the return to the initial phase until the course is done. This iterative process consists of the course curriculum management. While the process iterates, assessment data is collected through the process as summative and formative assessment data that can be used to determine the course success or to guide student for improvement, as shown in Figure 5.

HAA implements a multi-level three dimensional formative and summative assessment process applied on teams with multiple levels of hierarchy. In fact, individual assessment could be performed on team members, assessments could be done on team members’ performance and other assessments could be applied on the whole team performance and also the whole group final production or performance that includes the combination of all the work done by each team that took part in the project. Hierarchical aggregate could also be performed in distance by collaborative work with distance learning and assessment Internet applications so that all the teams or the team members could be at different locations, as shown in Figure 6.

Figure 4. Concept of HAA.

Figure 5. Aggregation and complex assessment tasks presentation process.

Figure 6. HAA three dimensional view of assessment.

3.2. Aggregation Process

HAA is a teamwork assessment process that groups students in teams with several levels of hierarchy and assign them a hierarchical position as team member, team leader or team manager (team or group administrator) to present them complex assessment tasks in a collaborative mode. When the assessment task is completed, the actual teams are dismantled and the team members are grouped in new teams with new hierarchical positions to perform another assessment task. This assessment process is able to assess different knowledge, skills and abilities depending on the hierarchical position of the team member. A team member is not assessed with the same criteria as team leaders and team (or group) managers.

The process of HAA brings together team members into teams that include multiple levels of hierarchy where these people can occupy the hierarchical positions of president, team manager (administrator), team leader as well as team member. The structure of the team is in the form of a pyramid or an inverted tree representing an organizational chart in which each branch is an aggregation of team members. The process of HAA is the action of grouping team members together in a hierarchical organizational structure on several levels and then make an assessment process for each member of the team that is a leaf of the tree or a node of the organizational structure. The HAA process is able to assess team members and also the homework or productions done by the teams and the whole group, as shown in Figure 7.

In the HAA process, the teams with several levels of hierarchy can be formed manually by the teacher or by computerized algorithms that could use cluster analysis methods as shown in Figure 8. Cluster analysis is “a

Figure 7. HAA process capabilities for simultaneous assessment of multiple skills.

Figure 8. Formation of teams with several levels of hierarchy by cluster analysis.

statistical technique that sorts observations into similar sets or groups” (Ketchen & Shook, 1996: p. 441) by “the partitioning of data into meaningful subgroups, when the number of subgroups and other information about their composition may be unknown” (Fraley & Raftery, 1998: p. 579) . It implies “any statistical technique for grouping a set of units into clusters of similar units on the basis of observed qualitative and/or quantitative measurements, usually on several variables” (Illingworth, 1996: p. 61) . Cluster analysis uses generally hierarchical cluster analysis methods where “clusters are formed in sequence, either by amalgamation of units into clusters and clusters into larger clusters, or by subdivision of clusters into smaller clusters and single units” (Illingworth, 1996: p. 61) . Cluster analysis is based on the unique facts of the observed values while discriminant analysis uses observations and other information to classify and infer the nature of the data (Fraley & Raftery, 1998: p. 578) . There are two methods of cluster analysis, supervised clustering and unsupervised clustering. Supervised clustering uses a certain adapted classification while unsupervised clustering uses no classification (Eisen, Spellman, Brown, & Botstein, 1998: p. 14863) . Clustering methods “range from those that are largely heuristic to more formal procedures based on statistical models. They usually follow either a hierarchical strategy or one in which observations are relocated among tentative clusters” (Fraley & Raftery, 1998: p. 579) .

HAA Computerized Implementation Example with the “Cluster” Internet Application

The “Cluster” Internet application resides at the Internet address http://eval.uqam.ca/cluster/ and is developed by the CDAME (Collectif pour le Développementet les Applications enMesure et Évaluation) research center in the faculty of education of the Université du Québec à Montréal (UQÀM). The application is able to manage 1) student data, 2) course material, 3) team formation and dissolution, 4) courses, 5) formative and summative assessments; and 6) hierarchical relationships between team members who may have several levels. The application has two mutually exclusive operating modes: student mode and the administrator or assessor mode. The assessor or administrator mode is usually given to the teacher or the person assigned as an assessor. This mode allows the teacher a) to build and modify courses, b) to build and to modify assessment tasks, c) to mark homework and teamwork production or performances, d) to create and modify student data, e) to create and dissolve student teams, f) to assign hierarchical positions to students; and g) to configure the application. The student mode is only used by students or candidates on distance courses given with the “Cluster” Internet application. Student mode allows candidates on courses to i) study the course material, ii) check out the curriculum record sheet to know what course modules are done and their progression through course modules, iii) perform HTML examinations, iv) submit homework, v) be part of a team to perform a complex evaluation task in teams, vi) occupy a hierarchical position in the team as a team member, team leader and group administrator; and vii) fill in forms of self-assessment and peer assessment. All of the features of the Cluster Internet application are shown in Figure 9.

The “Cluster” Internet application is able to assess different knowledge, skills, productions and performances simultaneously in the same assessment task in team. Hence, a student participating in an assessment task in team can occupy team member, team leader and group administrator hierarchical positions. When the student completes an assessment task, he must complete the self-assessment and the peers’ assessment forms. It is therefore necessary that the self-assessment and peers’ assessment forms have different assessment criteria based on the hierarchical position of the assessed student that could be team member, team leader or group administrator. The team member assessment form is shown in Figure 10. The form contains three radio buttons named “Assessment as team member”, “Assessment as team leader” and “Assessment as group manager”. When the student fills the assessment form, he must select the radio button corresponding to his assigned hierarchical position. The assessment criteria are different depending on the hierarchical position radio button selected.

The application’s aggregation function whose tree data structure is implemented into the application’s database allows the grouping of students into teams with multiple hierarchical levels. This feature allows the system to assign the student hierarchical functions such as team member, team leader and group manager. The aggregation function is accessible from the main menu of the application that is shown in Figure 11.

The user interface form of the “Cluster” Internet application that implements aggregation process grouping teams of students with multiple levels of hierarchy and assigns team members as team leader, team member and group manager is shown in Figure 12. This form enables the teacher or the assessor to begin the aggregation process to group students in teams.

Figure 9. Cluster internet application features and functionalities.

Figure 10. Team member assessment form.

Figure 11. User interface aggregation menu for team formation.

Figure 12. Aggregation process and team formation screen.

3.3. Curriculum

Course curriculum in HAA paradigm is different of traditional courses curriculum because it combines individual assessment and teamwork assessment and the team members can be assessed on different knowledge abilities and skills in accordance to their hierarchical position. Course curriculum in HAA is divided into course modules. Each course module can be confirmed by individual assessment, teamwork assessment or both. The score of the course module is computed from scores of individual and teamwork assessment and the same process is valid to compute the overall course score. The curriculum process is a succession of course modules presentation. All of the courses modules are presented to the student and the course ends when all the course modules are presented, as shown in Figure 13. In each course module, individual assessment and teamwork assessment tasks could be presented to the student. For each teamwork assessment task, the aggregation process occurs and the student is assigned to either team member, team leader or course manager hierarchical positions. So in all the course curriculum, while the process iterates, the student could be appointed randomly or by the teacher to team member, team leader or group manager hierarchical positions. For example, on a certain course

Figure 13. Course curriculum process implemented in the “Cluster” Internet application.

curriculum, the student could be appointed five times team member, two time team leader and one time group manager. The random aspect of the aggregation process based on cluster analysis or the teacher’s judgement on a team member’s performances could result that this member could only be appointed team member, or be appointed both team member or team leader or also only be appointed group manager and team leader for the entire course duration.

The student implication through the assessment process implemented in the “Cluster” Internet application is divided in two phases. The first phase is conventional assessment where the student performs individual assessment tasks as exams or homework to submit. The second is HAA where the student performs assessment tasks in teams. The overall course score is the sum of individual assessment and teamwork assessment. The student is grouped in a team and assigned a hierarchical position as team member, team leader or team manager. The team performs the assessment task and at the end of the assessment task, the team members complete formative assessment forms as self-assessment and peers assessment forms shown in Figure 9 where assessment criteria differs regarding the member’s hierarchical position. The teacher or assessor collects the team members’ formative assessment scores and produce a mark as summative assessment for each student, for the overall team performance and for the overall group performance. After the assessment task is completed, the teams are dissolved and another teamwork assessment task is performed. The students are grouped again in teams but their hierarchical position can change. This succession of individual tasks and teamwork assessment tasks is repeated until the end of course curriculum or academic program graduation as shown in Figure 14.

The “Cluster” hierarchical aggregate application is a rather complex Internet based software application. The application menus and the multiple data entry screens described in the previous sections are only a theoretical concepts presentation that is not enough to provide a complete understanding of the application. An example will be presented in the next section that is the navigation patrol in teams example course “PO 405-Map Using Training”. The map using training topography course is a simple course example that is including an assessment task where the students has several levels of hierarchy.

Figure 14. Student implication in the assessment process implemented in the “Cluster” Internet application.

Practical Example of Course Curriculum Implementation with the “Cluster” Internet Application

To obtain a well-designed distance assessment course within the “Cluster” Internet application, the teacher or the course administrator must consider two fields of the education domain that are the curriculum studies and the HAA. Before using or entering any course material in the application, the course administrator must have done a complete analysis and a planning of the course material to present in the assessment tasks. The course administrator must produce or obtain three documents. These documents are: a syllabus or a course schedule (shown in Figure 15 & Figure 16), a list of text files, multimedia files and course material to present over the Internet (shown in Figure 17 & Figure 18) and an organizational chart of the different teams of students that will execute the assessment tasks needed to succeed the course (shown in Figure 20).

Figure 16 shows a training schedule for an online training Internet version of the map using training course shown in Figure 15 with the “Cluster” Internet application. This Internet course only asks for one day of training in presence in the training sectors of a military base instead of two days of training needed by the traditional version of the course. This means that the cadet unit saves half of the budget by holding an Internet based course, the theoretical lessons being studied at home instead of in class at the unit. Therefore, a qualified course assessor not available or unable to come at the training location will be able to assess students’ performance at distance

Figure 15. Traditional in presence course training schedule.

Figure 16. Internet based training course schedule.

with the actual software application.

The “EO 405.01-Introduction to map using” modules course material includes the learning of course notes document “EO40501.pdf” and the study an electronic version of the Borden Canadian forces base (CFB Borden) included in the electronic document “cfborden.pdf”. The module also includes a self-correcting multiple choice questions HTML form test on the basic notions of map using. The HTML form is included in the electronic document “Ec40501a.html”, as shown in Figure 18. The “Cluster” application also allows assessment tasks to have prerequisites. The assessment task “EC 405.01-Quiz” has for prerequisite the study of the Borden military base (CFB Borden) map. Likewise, the students must have learned the elementary topography notions stated in the electronic document “Eo40501.pdf” before studying the Borden military base (CFB Borden) map. A prerequisite database management system included in the course database is implemented into the software application

Figure 17. “OREN 405” course modules.

Figure 18. Module “EO 405.01” course material.

to deny the student random access to any course module or to browse into all course modules. Therefore, some assessment tasks belonging to a course module must be completed before the student can access the next one.

The main discovery or improvement brought by this actual doctoral project is the HAA concept. To experiment this theory with a research and development (R & D) methodology, the “Cluster” software application have been built by the CDAME laboratory. This software application is able to assess knowledge, skills, abilities and objectives of team members that could have hierarchical relationships with multiple levels of hierarchy between them. The application is able to assess at the same time knowledge and skills of team members, team leaders and team managers that could supervise many team leaders. To use the student grouping or the aggregation functionalities of the application, the course administrator must group students in teams and assign them team manager, team leader or team member appointments.

Figure 19 shows a class or an army Cadets platoon that needs to qualify its members on topography to be able to fulfill his training requirements. The software application first allow students to study course modules that could be done individually at their own pace without being grouped in teams and appointed to leadership functions. However, when the students will finish all distance individual training modules, that are the first to the fourth modules in the map using course example, the course administrator will have to group students in teams and appoint them leadership functions with the aggregation process before beginning the navigation assessment task that must be performed in teams.

3.4. Portfolio

At each course module iteration, individual assessment and teamwork assessment are done by team members. During teamwork assessment, team members have to produce formative assessment with self-assessment and peers assessment forms while summative assessment is given by the teacher in accordance with team production and scores obtained by team members’ individual assessment. Individual assessment is done by homework to submit and HTML quiz forms. At each course module iteration, team and group productions and also summative and formative assessment are produced and stored into the course assessment database. Each module assessment scores and productions are stored in the application’s database and available to consult by examiners. So the course assessment database could be considered as a team and group productions portfolio and also a database containing all the assessment and productions of the team members as shown in Figure 20.

Internet application “Cluster” has implemented this data structure located in its MySQL (2013) database and its complex assessment tasks presentation engine in collaborative mode can perform assessment procedures for

Figure 19. “PC 405” students teams grouping.

Figure 20. Portfolio data creation through course module iteration.

each team member or each node of the tree. So in one assessment task, the application can assess different objectives, skills, abilities and knowledge. The “Cluster” application can assess simultaneously team members’ performance, team performance and group performance. This feature has not been implemented in other distance learning applications such as Moodle (2013) , Blackboard (2013) and WebCT (Blackboard, 2013) and this statement defines the fundamentals of the problematics of this research.

3.5. Assessment Methods

HAA includes the standard or the conventional assessment field that provides the same type of assessment for all the students in the class. Hence standard or conventional assessment process is the assessment of the same abilities, performances, knowledge and skills in the same assessment task. So standard or conventional assessment is a particular case of the HAA field. HAA includes the standard or the conventional assessment and is the assessment of different abilities, performances, knowledge and skills in the same assessment tasks according to the hierarchical position assigned to the team member as shown in Figure 21.

3.6. Links with Management Information Systems (MIS)

In the actual paradigm, there is a major difference between distance assessment systems and Management Information Systems (MIS) software applications. A distance assessment system software application is a question bank repository stored in a database that usually presents the same questions or the same assessment tasks to all of the students to assess the same skills and knowledge and there is no hierarchical relationships or hierarchy levels between the students. A Management Information System (MIS) is a software application that stores and process management data and information on employees to produce information used for decision making. The assessment data that a Management Information System (MIS) produces and computes for the employees are usually sales and production performance data. Management information systems are able to record the hierarchical relations and positions of the employees while distance assessment applications can’t.

The link between the fields of management and education is established by the assessment of teams with several levels of hierarchy. Therefore, a Management Information Systems (MIS) is able to assess several teams of employees with multiple levels of hierarchy grouped in a tree structure on parameters as net sales, gross sales, stocks list, inventory, payroll and staff listings. Management Information Systems (MIS) have their counterpart in the education field that are HAA application. As a Management Information Systems (MIS), a HAA application is also able assess several teams of individuals with multiple levels of hierarchy grouped in a tree structure. In the education field, the individuals are students and the assessment parameters are scores, marks or information on self-assessment and peers assessment, as shown in Figure 22.

3.6.1. Reciprocity

In the HAA paradigm, there is only a slight difference between HAA applications and management information systems because both applications records the hierarchical relations and positions of the individuals. The only difference is that the management information system process management data while the HAA software appli-

Figure 21. Situation of HAA in the assessment field.

cation process course material, question banks and complex assessment tasks with several levels of hierarchy. So any management information system could be modified to record course material and question banks to present complex assessment tasks with several level of hierarchy. So the modified management information system has now been added HAA capability and is also now a HAA software application. The reciprocity equally applies to HAA systems that could be modified to evaluate student performances in sales and productivity as shown in Figure 23. HAA domain specialists will be ultra-generalists and they will be experts in the fields of assessment in education and management information systems (MIS) in the business administration field. In the education field, the HAA expert will be able to assess teams with several levels of hierarchy and to determine if management functions (sales, stocks, production ratios, earnings, etc.) could be added to E-learning

Figure 22. Teamwork assessment with multiple levels of hierarchy as a link between the fields of management and education.

Figure 23. Reciprocity between HAA and Management Information Systems (MIS).

applications. In the management information systems (MIS) field, the HAA expert will be able to insert course modules and complex assessment tasks in collaborative mode to management information systems.

3.6.2. Collaborative Work

In the fields of Management Information Systems (MIS) and HAA, the organization members or the teams with several levels of hierarchy are assessed in teams and the work, production or performances are both individual and in teams. In the HAA field, the assessment task submitted to the teams is complex assessment tasks in collaborative mode.

4. Experimentation

The HAA process described in this paper has underwent three experimentations. The first was the debugging phase done by the CDAME researchers as Alpha Tests to confirm that the software was fully operational and ready to be experimented by users. The second and third experimentations were Beta Tests done by high school students and Canadian army cadets.

The testing of the “Cluster” Internet application was first applied on the class of Mrs. Dalila Sebkhi’s high school students that were the first beta tests used to experiment the application on a large population of over 100 students (N > 100). The Alpha tests were done before by the CDAME researchers (Lesage, Raîche Riopel, & Sebkhi, 2013; Lesage, Raîche, Riopel, Fortin, & Sebkhi, 2014; Sebkhi, Raîche Riopel, & Lesage, 2013) . In this experiment, the “Cluster” Internet application was used by high school students of province of Quebec as an alternative method to teach geology courses. The results of the experiment were purely qualitative and were based on Ms. Sebkhi’s observations during the experiment where students used the application in their geology classes. Several students who used the application “Cluster” and some directors of the Montreal school board argued that the application user interface was too rigid and not friendly enough for students who were teenagers from 12 to 16 years of age. The high school students wanted the user interface to make more use of multimedia elements such as videos and animated graphics so that the course is more like a video game with avatars as in the “Mecanika” application implemented by François Boucher-Genesse (Boucher-Genesse, Riopel, & Potvin, 2011) rather than the actual “Cluster” Internet application basic drop down menus user interface. However, for some students, learning the use of the “Cluster” Internet application was simple and easy. These students didn’t had any problem to study the course material, review all the course modules and take the geology course exams placed at the end of course modules. The other students had experienced various problems when using the “Cluster” Internet application such as 1) resistance to change, 2) losses of user names and password 3) filling the HTML exams, 4) being lost in navigation when studying the course material, 5) impossibility to go back in the user interface navigation if the course material is not understood or saved and that the student wants to regain access to the course materials or to the previous sections, and 6) difficulty for teachers or course assessors to keep track of progress in the modules and examinations for a large number of students.

The third experimentation has been performed by the organization of Canadian army cadets. This application is now used by the cadets to provide distance courses on topography, navigation patrols, instructional techniques and general military knowledge. The results for the testing of the “Cluster” Internet application by the army cadets demonstrate that the increase of knowledge produced with the “Cluster” Internet application is 50.59%, an increase which is almost identical to that produced by the traditional classroom teaching methods that is of 48%. However, the success rate for the learning of topography and map using with the “Cluster” Internet application is only 22% compared to learning the map using in class is 83%. The success rate of 22% produced by distance learning can be explained by the fact that many of the cadets in the experimental group were having learning disabilities and also that some of the major drawbacks as to leave the student alone in his learning process without being in the classroom and lacking the presence of a teacher or colleagues to help him. Very often students through distance course became confused by the lack of classroom dynamics that attenuates motivation and desire to learn.

5. Conclusion

The aim of this paper was to define the HAA process and to establish a link between the education and assessment fields. The link is created by the assessment of treelike structures organizations of individuals with multiple levels of hierarchy. An Internet application named “Cluster” implements the production of complex assessment tasks in collaborative mode that assess teams with multiple levels of hierarchy. In both education and assessment fields the link is based on the processing of data through a multiple level tree structure. Some limitations concern: 1) the novelty of the established link between education and management fields, 2) resistance to change by using HAA prototype applications that are not produced at commercial levels, and 3) the limited number of researchers working on the HAA field. Future work and possible applications of the current research are a) the production of Management Information Systems (MIS) that could assess the knowledge of the members of large organizations, b) the production of complex assessment tasks in collaborative mode, c) the production of simulation scenario to assess managers in formation, d) and the production of electronic portfolio assessment systems for students, teachers and managers. Some other considerations brought by the actual research project are that the Internet application accelerates significantly the assessment process of teams with several levels of hierarchy. At the actual stage of the project, software developers didn’t have time to implement the assessment tasks coding in IMS/QTI standard for MOODLE compatibility but that issue is seriously considered for future improvements of the application to create an add-on multiple levels of hierarchy aggregation module to MOODLE so that MOODLE users could do HAA in courses already implemented in MOODLE. Finally, resistance to change has been a major obstacle to the implementation of the “Cluster” application in organizations.

Acknowledgements

This work has been funded by the Social Sciences and Humanities Research Council (SSHRC) and the Fonds de Recherche du Québec-Santé (FQRS).

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1] Blackboard (2013). Blackboard Web Site. [On Line] http://www.blackboard.com
[2] Boucher-Genesse, F., Riopel, M., & Potvin, P. (2011). Research Results for Mecanika, a Game to Learn Newtonian Concepts. In C. Steinkuehler, C. Martin, & A. Ochsner (Eds.), Games, Learning and Society Conference Proceedings (pp. 31-38). Madison, WI: ETC Press.
[3] Burch, J. G., & Grudnitski, G. (1989). Information Systems: Theory and Practice (5th ed.). New York: John Wiley.
[4] Davis, G. B., & Olson, M. H. (1985). Management Information Systems: Conceptual Foundations, Structure, and Development. New York: McGraw-Hill.
[5] Davis, G. B., Olson, M. H., Ajenstat, J., & Peaucelle, J. L. (1986). Information Systems forManagement. Volume I: The bases. Boucherville: G. Vermette Inc. Editors.
[6] Eisen, M. B., Spellman, P. T., Browm, P. O., & Botstein, D. (1998). Cluster Analysis and Display of Genome-Wise Expression Patterns. Proceedings of the National Academy of Sciences of the United States of America, 95, 14863-14868.

http://dx.doi.org/10.1073/pnas.95.25.14863

[7] Fraley, C., & Raftery, A. E. (1998). How Many Clusters? Which Clustering Method? Answers via Model-Based Cluster Analysis. The Computer Journal, 41, 578-588.

http://dx.doi.org/10.1093/comjnl/41.8.578

[8] Freeman, M., & McKenzie, J. (2000). Self and Peer Assessment of Student Teamwork: Designing, Implementing and Evaluating SPARK, a Confidential, Web Based System [On Line]. In L. Richardson, & J. Lidstone (Eds.), Flexible Learning for a Flexible Society, Proceedings of ASET-HERDSA 2000 Conference. University of Southern Queensland, Toowoomba: ASET and HERDSA.

http://ascilite.org/archived-journals/aset/confs/aset-herdsa2000/procs/freeman.html

[9] Freeman, M., & McKenzie, J. (2002). SPARK, a Confidential Web-Based Template for Self and Peer Assessment of Student Teamwork: Benefits of Evaluating across Different Subjects. British Journal of Educational Technology, 33, 551-569.

http://dx.doi.org/10.1111/1467-8535.00291

[10] Illingworth, V. (Ed.) (1996). Dictionary of Computing. New York: Oxford University Press.
[11] Kanter, J. (1984). Management Information Systems (3rd ed.). Upper Saddle River, NJ: Prentice Hall.
[12] Ketchen Jr, D. J., & Shook, C. L. (1996). The Application of Cluster Analysis in Strategic Management Research: An Analysis and Critique. Strategic Management Journal, 17, 441-458.

http://dx.doi.org/10.1002/(SICI)1097-0266(199606)17:6<441::AID-SMJ819>3.0.CO;2-G

[13] Laudon, K. C., & Laudon, J. P. (2000). Management Information Systems: Organization and Technology in the Networked Enterprise (6th ed.). Upper Saddle River, NJ: Prentice Hall.
[14] Laudon, K. C., Laudon, J. P., & Brabston, M. E. (2011). Management Information Systems: Managing the Digital Firm (5th Canadian ed.). Toronto, ON: Pearson Education Canada Inc.
[15] Lesage, M., Raiche, G., Riopel, M., & Sebkhi, D. (2013). Hierarchical Aggregate Assessment (HAA) Internet Application Development with Research Development (R & D) Methodology. International Francophone Association of Scientific Research in Education (AFIRSE).
[16] Lesage, M., Raiche, G., Riopel, M., Fortin, F., & Sebkhi, D. (2014). An E-Assessment Website to Implement Hierarchical Aggregate Assessment. World Academy of Science, Engineering and Technology, International Science Index, 86, 925- 933.
[17] Lesage, M., Raiche, G., Riopel, M., Fortin, F., & Sebkhi, D. (2015). The Internet Implementation of the Hierarchical Aggregate Assessment Process with the “Cluster” Wi-Fi E-Learning and E-Assessment Application: A Particular Case of Teamwork Assessment. In B. Gradinarova (Ed.), E-Learning (pp. 83-125). Croatia, Rijeka: InTech Europe.
[18] Marshall-Mies, J. C., Fleishman, E. A., Martin, J. A., Zaccaro, S. J., Baughman, W. A., & McGee, M. L. (2000). Development and Evaluation of Cognitive and Metacognitive Measures for Predicting Leadership Potential. The Leadership Quarterly, 11, 135-153.

http://dx.doi.org/10.1016/S1048-9843(99)00046-6

[19] Moodle (2013). Moodle Web Site. [On Line] http://www.moodle.org
[20] MySQL (2013). MySQL Web Site. [On Line] http://www.mysql.com
[21] Nance, W. D. (2000). Improving Information Systems Students’ Teamwork and Project Management Capabilities: Experiences from an Innovative Classroom. Information Technology and Management, 1, 293-306.

http://dx.doi.org/10.1023/A:1019137428045

[22] Sebkhi, D., Raiche, G., Riopel, M., & Lesage, M. (2013). A First Experimentation of an Hierarchical Aggregate Assessment (HAA) Internet Application with High-School Students through Teaching Internships. International Francophone Associ- ation of Scientific Research in Education (AFIRSE).
[23] Undre, S., Sevdalis, N., Healey, A. N., Darzi, S. A., & Vincent, C. A. (2007). Observational Teamwork Assessment for Surgery (OTAS): Refinement and Application in Urological Surgery. World Journal of Surgery, 31, 1373-1381.

http://dx.doi.org/10.1007/s00268-007-9053-z

Copyright © 2024 by authors and Scientific Research Publishing Inc.

Creative Commons License

This work and the related PDF file are licensed under a Creative Commons Attribution 4.0 International License.