X-Pro Milestone: Comparative Evaluation against Similar Software Tools, Current Status, and Planned Future Directions ()
1. Introduction
Studies show that computerized (and even partially computerized) exams rely on the availability of high-quality item banks. Those item banks serve as repositories from which we select items to be applied to tests (Molina, Pareja, & Sanmartín, 2008) . However, further studies also show that the availability of item banks is rather limited because the development of a high-quality item bank can involve investing a great amount of time and effort. Therefore, we vitally need handy computer software tools that facilitate instantiation and maintenance of high-quality item banks.
It is well-known that, in the education sector, class-size is continuously growing higher every day (Ministry of Higher Education, 2010; Bani-Ahmad & Audeh, 2010a) . Taking the country of Jordan as an example, Figure 1(a) depicts the change in student numbers along with the growth in the number of instructors between academic years 1999/2000 and 2009/2010 in Jordanian private and public universities (Bani-Ahmad & Audeh, 2010a) . The figure indicates that student numbers grows very fast compared to instructor number. As indicated in (Bani-Ahmad & Audeh, 2010a) , student-per-ratio in some Jordanian countries is more than sixty. This growth is not usually supported by an equivalent and proper growth of educational resources and technology infrastructure (Bani-Ahmad, 2015; Bani-Ahmad & Audeh, 2010b) . This observation is truer for developing countries probably due to the general phenomenon of digital divide (Chinn & Fairlie, 2004) .
A digital divide is defined as the economic and social inequality with regard to 1) access to, 2) use of, or 3) impact of information and communication technologies (ICT) (Chinn & Fairlie, 2004; I nternational Telecommunication Union (ITU), 2015) . Studies indicate that there is, as indicated in Figure 1(b), a significant and dramatic digital divide between developed countries (e.g. US and most of Europe) and the third world (e.g. Middle East countries) in terms of availability of technological devices (namely; computers) and access to the Internet (Chinn & Fairlie, 2004; International Telecommunication Union (ITU), 2015) . This phenomenon makes the usage of online exams be prohibitively impossible especially in developing countries where efficient and effective ICT-related infrastructure is not available and not affordable.
According to (Jenkins, 2017) , “Class size is really irrelevant in this day and age in education. It’s not about how many children you have in the classroom. It’s about how you’re leveraging technology to deliver one-on-one instruction". This draws the link that relates the digital-divide phenomena to the problem addressed in (Jenkins, 2017) . In fact, the digital divide phenomenon makes the usage of fully computerized (i.e. online) examination tools (that allows students to set for tests and exams using a computer, usually networked) prohibitively impossible with the absence of proper information technology infrastructure (Ministry of Higher Education, 2010; Maharey, 2011) .
According to (Ehrenberg et al., 2001) , some of the disadvantages of large class size are: 1) it can increase the disruptive behavior in classrooms; and reduce the amount of time the instructor spend working with each individual student; 2) it can reduce the material the teacher can cover and can eliminate many methods of assessing students due to difficulty of grading; and, therefore, it can reduce the quality of teaching methods that the teacher can employ in his/her classroom.
Proper use of an item bank and of good item banking software tool significantly reduces the effort and expense of preparing new items for each exam. This is true whether the exam is used by a teacher of a relatively small class; a university lecturer who has large sections; or even the director of multifaceted programs for certification, licensing, accountability, or norm-referenced survey testing (Ward & Murray-Ward, 1994) . Serving this purpose, this article is about X-Pro Milestone; an item-bank software tool that saves many work hours that the instructor can spend improving his/her teaching skills. This software facilitates producing pre-formatted printable exams. In this paper,
1) The latest release of this software is presented and the latest improvements are explained. We also cover the special Descriptive Markup Language (DML) used in constructing item banks; termed XDML.
2) The software is comparatively evaluated against a set of similar software tools available in the market. This comparison is based on a set of features. We demonstrate that X-Pro Milestone outperforms other software based on a wide spectrum of features and functionalities that are not available in other off- the-shelf software tools. We also show that the availability of such sound item banking software could help to extend the application of item-bank-based exams beyond the boundaries of large-scale expensive testing programs.
3) We also cover the current status of this software tool and draw the future planned short-term and long-term future directions of the X-Pro Project.
2. Course Item-Bank Hierarchical Structure in X-Pro Milestone
A typical X-Pro syllabus-specific item bank contains a set of test items (questions). Those items are of four possible types: multiple-choice (MC), true-false (TF), short-answer (SA), and free-response (FR) questions. Those question types are also organized in a hierarchical structure as shown in Figure 2. As illustrated in the figure, each test item belongs to a question pool where, in turn, belongs to a section.
Each question pool includes a set of test-items covering a specific topic or tackling a specific course objective. It is assumed that the instructor, when preparing his/her course item bank for a given course, he or she subdivides the syllabus into a set of well-defined course objectives or fine topics. Sections in Figure 2 represents book chapters. Book here refers to the textbook used in the current course bank.
To produce exam instances based on item banks, items are randomly pulled from the item bank and are later assigned to exam forms for publication (either as a paper-based exam or some form of e-assessment or online exams).
To tackle the problem of performance interference (or cheating), the user is recommended to produce many-enough exam forms. Those forms are generated
![]()
(a) ![]()
(b)
Figure 1. (a) Student and instructor numbers in Jordanian Universities (years 1999-2010) (Bani-Ahmad, 2017) , (b) the digital divide phenomenon: Internet users per 100 inha- bitants (Willits, 1992) .
![]()
Figure 2. The hierarchical structure of a typical X-Pro Milestone syllabus-specific item bank.
by randomly permuting 1) choice-list items (for multiple-choice questions), 2) test items, or 3) both. The randomization should produce uniformly distributed correct-choices over the set of possible positions within choice-lists to reduce the negative effect of randomly responding to multiple-choice question. In X-Pro Milestone, the user has the option to have different set of test-items per produced exam-form. The set of question pools from which test items are selected are exactly identical for all exam-forms.
Each produced exam-form is accompanied by its specific answer-key. The list of choices is to be properly permuted such that the probability of observing the correct answer at one position within the list of choices is equal to the probability of observing it at any other position (the computer uniform random number generator is to be used for that purpose). The goal of this requirement is to minimize the chance that an examiner receives high score solely by clustering his or her answers at one given position, not because of his/her real achievement.
Notice that one and exactly one test item is selected from each randomly selected question pool is that we would like to have the produced exam properly cover a wide spectrum of the fine topics covered by the course syllabus. We argue that this guarantees that the produced exam effectively covers a relatively wide range of topics.
Notice that a question pool includes questions covering a specific and well- defined course objective or sub-topic. This means that questions belonging to the same question pool are somehow topically very similar and cannot be included in the same exam instance.
Figure 3 shows a screenshot of the main screen of the GUI of X-Pro Milestone. The left set of tab-pages displays the basic functional tab-pages of X-Pro Milestone: one tab page for each type of the four supported types of questions. The “Figures and Attachments” helps the user manage the figures used within the current item bank; this includes: importing, removing, previewing, and adding notes to any figure in the course bank. The function of the “Pre-parse Input” tab-page is to check the item bank for any syntax or structural errors that violate
![]()
Figure 3. The English-language GUI interface of X-Pro Milestone.
the DML of the software. The pre-parse step is required before being able to produce exams using the “Produce a New Exam” tab-page. Once a new exam is produced, the user can preview and manage it through the “Preview Exam- Forms” tab-page. All available course item banks can be found in the “List of Existing Question-Banks” tab-page.
3. X-Pro Milestone: User Interface and Descriptive Markup Language
X-Pro Milestone Supports four types of questions as mentioned earlier. Each type of has the same generic hierarchical structure. That is; a number of sections, each section is composed of a number of question pools. Each question pool, in turn, includes one or more test item as shown in Figure 4 and Figure 5. X-Pro Milestone uses special tags to indicate the beginning of sections and question pools and test items. These form the basic tags in the X-Pro Descriptive Markup Language (XDML). The following is a list of the basic X-Pro Milestone tags to indicate sections, pools, and questions.
• The [[New Section Start]] starts a new section.
• The [[Begin Pool]] tag starts a new pool.
• The [[Begin Question]] tag starts a new question.
![]()
Figure 4. X-Pro Milestone course bank structure.
![]()
Figure 5. The syntax of a typical multiple-choice question in X-Pro Milestone.
Figure 5 shows a sample multiple-choice question. The [[Begin Question]] indicates the beginning of a new multiple-choice question. The [[Begin Choice List]] tag indicates that the lines after are the list of choices of that specific question. The set of lines after the [[Begin Question]] and before the [[Begin Choice List]] tags represent the multiple-choice question stem. This means that the question stem may be composed of multiple lines. However, the [[Begin Choice List]] tag is optional. In this case, the question stem is assumed to be composed of one single line; that is the line just after the [[Begin Question]] tag. The lines following that line represent the list of choices. Each line in the list of choices represents one choice. A choice my not span multiple lines.
Each multiple-choice question can have up to six choices (the minimum is two). Those choices will be later permuted to produce exam forms. To force one specific choice to remain in-place (will not be permuted), the “do not permute” tag next to that choice is used; that is the
tag. This help the user to have choices like “All above” or “both A and C are correct” choices.
The “
” next to a choice indicates that this choice is a correct choice (used later by the application to produce the answer key table). The
is referred to as the “correct answer” tag. A multiple-choice question may have one or more correct choice (one is the minimum).
The [[#Inline_Choices#]]} tag next to the [[Begin Question]] tag indicates that the list of choices in the produced exam form will be put in-line (next to each other) as shown in Figure 6. If you remove the [[#Inline_Choices#]]} tag, each choice is placed in a different line.
Figure 7 shows a sample of a few short-answer questions. In general, each short-answer question appears in a single line. Each question is of two parts (see Figure 7): 1) the question text, and 2) the correct answer placed between the “[[“ and “]]” tags (referred to as the short-answer opening and closing tags).
Figure 8 shows a sample of few true-false questions. Similar to short-answer question, each true-false question appears in a single line and consists of two parts (as illustrated in Figure 8): 1) the question text, and 2) the correct answer that will be either [[True]] or [[False]].
Figure 9 shows a sample Free-Response essay question. Each Free-Response essay question is of two parts: 1) the question text that may span multiple lines
![]()
Figure 6. A sample multiple-choice question with its choice-list displayed inline.
![]()
Figure 7. A sample short-answer question in X-Pro Milestone.
![]()
Figure 8. A sample True/False question in X-Pro Milestone.
![]()
Figure 9. A sample Free-Response essay question in X-Pro Milestone.
(appears after the [[Begin Question]] and before the [[Begin answer]] tags). 2) the correct answer that appears after the [[Begin Answer]] tag. Again, the answer may also span multiple lines. Notice that the user can add the number of lines to be left to students in the answer form to place their response to the question. To do so, the user needs to add this number next to [[Begin Answer]] tag. If the user does not enter this number, the software automatically computes this space depending on the key answer text length.
4. X-Pro Milestone User Interface
The results of the analysis of the information associated with the development of item banks have shown a well-structured underlying data scheme (Molina, Pareja, & Sanmartín, 2008; Molina, Sanmartín, & Pareja, 1997) . Further studies also show that databases and database management systems excel at processing highly structured data. Notice that item banks in X-Pro Milestone are not implemented as a database (Willits, 1992) . Experience showed us that expanding X-Pro Milestone features and capabilities requires implementing item banks as a flat text file with proper special tag-based language as illustrated in Figure 10.
5. Producing Exam İnstances Using X-Pro Milestone
Before the user can produce exam instances from his/her course item bank as shown in Figure 12, he/she needs to pre-parse the course item bank to check if the course has any syntax or structural error. Example errors are: finding a multiple choice question with no choice list or only one single choice.
![]()
Figure 10. The “multiple-choice questions” tab page.
![]()
Figure 11. The “figures and attachments” tab page.
After the user prepares the course item bank with no errors, the “Produce a New Exam” tab page will be enabled and its content will be refreshed. The user can now produce exam instances. This is done in five steps (Figure 13): 1) choose the set of sections to be included and the number of pools to be randomly selected, 2) set exam information; 3) set exam instructions, 4) con- figure exam signup-sheet settings. The final step is to set the form generator output settings. This includes (examples): 1) the number of forms required, 2) the form ID style, 3) wither the user wants to hide the form ID from in the produced forms, 4) the permutation options, 5) wither the user wants to have the same set of questions in all produced forms or not, notice that regardless of the users’ choice of this option, the final set of questions will be selected from the same set of pools. 6) the multiple-choice answer table and answer-key styles and 7) where to place the answer key of the produced exam form (first page or last page), and 8) the number of extra blank sheets, the font type and size, and the space between MC questions.
Let S[] be the set of sections selected by the current user to produce a new exam instance. Let S[k] be the number of question pools that the user wants to include from section k. Let further S[k*] be the total number of question pools available in section k. Notice that S[k*] ≥ S[k], meaning that the user can not choose more than the available question pools. Notice that the number of questions to selected from section k will be S[k]; exactly one question is to be randomly selected from question pool number k.
The following pseudo code describes who X-Pro Milestone prepares exam instance from a given question bank. Pk[] is the set of question pools randomly selected from section k that has S[k*] available question pools. Notice that |Pk[]| = S[k]. Q[] is the set of questions selected to be included in the new exam instance.
Q[] = nil;
For each section k in S[k]
・ Pk[]←Randomly select S[k] Question pools from the available S[k*] pools
・ For each question pool j in Pk[]
・ Q*←Randomly select one question from pool j
・ Q[].Add(Q*)
After successfully producing the exam, the user can preview the produced exam forms and answer keys (Figure 14 and Figure 15). Through this page, the user can preview, print, save, or even archive produced forms for future refe- rencing.
Having talked about exam archiving, archived exams can be manipulated us- ing the archived-exams manipulation tool shown in Figure 16.
6. Extra Features of X-Pro Milestone
6.1. Course İtem Bank Manipulation
As shown in Figure 17, the user can create and new course-bank folder or open existing item bank (for editing). (iii) backup or archive an already existing item bank folder, and (iv) delete or rename existing item bank folder. The user can also export a tarball of the course folder and import it later into some other machine. Finally, the user can also view the set of archived exams of a specific course item bank.
![]()
Figure 14. The “Preview and Export Exam-Forms” tab page.
![]()
Figure 15. A sample answer key of multiple-choice questions.
![]()
Figure 16. The “archived-exams manipulation” tool.
![]()
Figure 17. The “item-bank” right-mouse-click menu item.
6.2. Insert a Figure from the Pre-Fed Set of Figures
Figures can be inserted into the text of any item (and even in the choice list of a multiple-choice item). A figure can be an image (e.g., a chart or a diagram) or text file (e.g., computer code). X-Pro Milestone supports three styles of inserting figures: Inline figure, repeatable appendix figure, and unique appendix figure (Figure 18).
A figure can be inserted inline with text. In this case, the figure content will appear with the question in the produced exam form. In other words, it will not appear at the end of the exam form as an appendix.
Appendix figures are of two forms; repeatable and unique. A figure can be inserted as a repeatable figure at the end of the exam form as an appendix. That is; if the figure is referenced inside n different questions in a given exam-form, then the same figure content will appear n times in the appendix of that exam- form and it will be assigned n different figure IDs. Notice that the figure refe- rence inside the question will be replaced by a proper reference to the figure ID as it appears in the appendix part of the form. e.g., it will be replaced by “Figure 5”, or “figure x” in general. The sequence or ID number “x” here is automatically generated by the software; the user does not have to worry about this issue.
A figure can be inserted as a unique (not repeatable) figure at the end of the exam form as an appendix. That is; if the figure is referenced inside n different questions in a given exam-form, then this figure content will appear only once in the appendix of the exam-form. Again, that the figure reference inside the question text will be replaced by a proper reference ID to the figure as it appears in the appendix part of that exam-form. Again, this ID is automatically gene- rated by the software and the user does not have to worry about this issue.
6.3. Insert Structural, Styling, and Symbol Tags
As shown in Figure 19(a), the user can insert any of the X-Pro Milestone special item-bank structural and styling tags. Example styling tags are: bold, italic, underlined, subscript, superscript. Figure 19(b) shows the user interface tab page through which the user inserts symbol tags, e.g., Math symbols (∏, π), and Greek letters (α, β, µ).
![]()
Figure 18. Inserting Figures using X-Pro Milestone.
Examples: Inserting mathematical equation, computer code and simple tables
Figure 20 shows an example on how to insert mathematical equations into a X-Pro Milestone item bank. To the left of this figure, you see a screen-shot of the question as it appears in the exam form. The tag that represents the X-Pro syntax that represents the equation
![]()
![]()
![]()
(a) (b)
Figure 19. (a) Inserting X-Pro and styling tags, (b) inserting symbols and special characters.
![]()
Figure 20. Example on how to insert a mathematical equation into a X-Pro Milestone item bank.
Is the following:
![]()
Here we use the superscript [@sup], [/@sup] tags and the theta symbol tag [theta]. Notice that [Theta] is different than [theta].the first is the uppercase form of the letter theta.
If the user wants to insert computer code within the item bank question text, then the user needs to surround his/her lines of code with the styling tags [@- code] and [/@code] as shown in Figure 21.
X-Pro Milestone supports different styles for numbing and bulleting (as shown in Figure 22). It also allows the user to insert simple tables (Figure 23). If the user wants to insert a simple table within the item bank question text, then the user needs to surround his/her lines that represent the table rows (each line represents a table row) with the styling tags [xtable] and [/xtable].
![]()
Figure 21. Inserting computer code in X-Pro question stems.
6.4. Item Backward Reachability
If the user finds some typos or mistakes in any of the questions of the produced exam instance, then he/she can locate back and directly navigate to the erron- eous question within the item bank. This is achieved by clicking the “<<<” mark that is located just after question text in the web browser or in the “Preview and Export Exam-Forms” tab page (Figure 24). This will take the user directly to the location of that question in the course item bank.
As indicated in Figure 24 also, the correct answer to the multiple-choice question is displayed in red color (in preview-mode only, when the exam-form is printed out, all the choices will look and feel similar). This feature is needed if the user would like to double-check the correctness of the question.
6.5. Extra Tools: Score Transformation and Archived Exams Tools
X-Pro Milestone is also equipped with a simple score transformation tool as shown in Figure 25. A linear score transformation is a transformation of the form (Stockburger, 2017) .
![]()
where
• X’ is the new transformed score (the “new scores” column in Figure 25)
• “a” and “b” are the transformation parameters
• X is the raw (original) score(the “row scores” column in Figure 25)
Notice that if a measurement system approximated an interval scale before the linear transformation, it will approximate it to the same degree after the linear transformation. Other properties of the distribution, however, are unaffected. For example, if a distribution was positively skewed before the transformation, it will be positively skewed after (Stockburger, 2017) . Next is the user interface of the score transformation tool.
![]()
Figure 24. Tracking back questions in X-Pro Milestone by clicking on the “<<<” mark.
![]()
Figure 25. The linear score transformation tool of X-Pro Milestone.
Archived exams can also be manipulated using the archived-exams manipu- lation tool shown in Figure 16. Through this tool, archived exam forms can be viewed, printed, copied, and deleted.
7. Comparing X-Pro to Similar software
(http://en.softonic.com/s/question-bank-software). This search gave back 9 hits all listed in Table 1 (software tools for Windows environment).
Next, we compare X-Pro Milestone to the most mature of which. The comparison is made based on a set of criteria groups; these are:
Group 1: Software user interface
1) User friendliness: the UI of the software need to be simple and intuitive. This helps the learning curve of the software to be fast.
2) Flat GUI design: Flat GUI interfaces helps the user make maximum use of the features of the software tool. The design needs to be logical and intuitive.
3) GUI language: For the tool to be responsive to the user needs, the GUI languages need to cover the possible languages of the users.
Group 2: Item content, Item entry and storage
1) Flat Question-Bank (QB) design: The structure of a typical item bank need to be hierarchical and intuitive. It also needs to be based on the structure of the text-book (of the course) and/or the course syllabus itself.
2) Database (DB) based: It is preferred that item-banks be kept in flat files and not as database file for simplicity and portability.
![]()
Table 1. IB software tools for Windows environment
3) Portable item bank: item-banks need to be portable; the software needs to provide proper import and export easy-to-use commands.
4) Multi-language support: Exams can be in any language, for that, the software tool must use Unicode character set.
5) Attachments and figures format: printable exams may contain shapes and figure. For that, the software tool should allow entering image-based and text-based figures. Each figure is usually accompanied by important metadata; the software tool should provide simple and intuitive mechanisms for manipulating shapes and figures.
6) Attachments and figures management: The software tool needs to provide featured management menu.
7) Figure insertion location: the software should allow inserting figure within the question stem (inline figure) or within an appendix at the last page of the exam.
Group 3: Item retrieval for reviewing items, formatting test forms
1) Question selection (to produce exams): Question selection need to be automatic. Further, selected question sets should produce comprehensive exams that perfectly cover the course syllabus.
2) Media: be it online (for online exams) or print (for paper-based exams).
3) Informative, rich exam form header: This indicates is the exam header is rich and informative or simple and very basic. Rich exam headers are preferred (without increasing the user-friendliness of the software interface).
4) Printable answer key: The answer-key of produced exams need also to be printable. This is important when sufficient IT infrastructure is not available.
5) Widow and orphan questions: In print media, no question should split into two pages when printing.
6) Archiving produced exams: The software tool should provide extra tool(s) for archiving produced exam and for manipulating previously archived exams.
7) Provision of special permutation option: in multiple-choice questions, some option should be permuted, e.g.; “all above” and “none of above”. The software tool must provide a mechanism to control permutation options.
8) Space control options: the software tool needs to provide mechanisms to control space in exam forms. E.g.; space between questions, space left for the answer in free-response questions, provision of blank sheets, and allowing the user to place the choices in multiple-choice questions to be rendered inline or not (all choices in one single line).
![]()
Table 2. Comparative evaluation between X-Pro Milestone and similar software tools.
9) Performance interference prevention mechanisms: the software tool needs to provide mechanisms to prevent cheating in exams. Some are: permuting questions and/or choices and hiding form ID.
Group 4: Maintenance of item history, editing, and updating items
1) Item backward reachability: If the user finds some typos or mistakes in any of the questions of the produced exams, the software should help to locate and directly navigating back to the erroneous question within the item bank.
Group 5: Extra tools and features
1) Score transformation tool: One important extra tool to provide is the linear transformation tool.
2) Producing sign-up sheets: The software tool should be capable of producing sign-up sheets for each produced exam. Preferably, student names can be pre-feed to these sign-up sheets already.
3) Protection of item banks: the software needs to provide mechanisms to protect the content of item banks.
In Table 2 we start comparatively evaluate the five tools based on the set of criteria listed above. We should notice that X-Pro Milestone out-performs all other software tools based on each and every criterion.
8. Conclusion and Future Directions
Use of item bank can provide much relief for the chores related preparing exams (be them online or paper-based). Use of item banks may also enhance the quality of the items as well as the quality of the exams overall. Item banking software tools provide for item entry, storage, maintenance and retrieval and exam creation. In this article, we presented latest advances in the X-Pro Project. We also presented the recently added features to the latest release of the X-Pro Milestone; which is a syllabus-specific item-bank system that can produce paper-based exam forms and answer keys.
We also comparatively evaluated X-Pro Milestone against a set of similar software tools reachable and downloadable over the web. The software is comparatively evaluated against the set of similar software tools against a set of more than twenty features. We demonstrated that X-Pro Milestone outperforms other software based on a wide spectrum of features and functionalities that are not available in other software tools.
The next step of this project, focus on improving the improving usability feature of X-Pro Milestone. We also plan to allow the user to indicate the difficulty level of each question in his/her item bank. This should help the user produce exam instances with different levels of difficulty. The X-Pro Project website URL is http://sulieman.net/xpro. A trial version of the software can be downloaded through the link http://sulieman.net/xpro/get.
Acknowledgements
We would like to sincerely thank the current users of X-Pro Milestone for their feedback and constructive comments. Special thanks to the users at the University of Tabuk-Saudi Arabia for their positive messages.