Reliability and Validity of Online Individualized Multimedia Instruction Instrument for Engineering Communication Skills ()
1. Introduction
Engineering field is accountable for major industrial, technological, and economical progress in human history. The engineer who design and build things plays a major role in keeping the society running smoothly. Well qualified engineers are always needed in a growing society. The format of engineering communications can vary widely, from summaries of calculations, to short technical messages, to oral presentations, to drawings describing data or machinery.
In Jordan, a number of engineering branches and specializations are taught by the public and private universities. Each university tries to offer unique specializations to attract students enrolment. Nevertheless, studying abroad continued even after the establishment of the engineering colleges in Jordan (Aqlan et al., 2010). These issues due to the reality where engineering education traditionally relies more on technical skills and less communication skills (Corrello 2012). Engineering instructors seem to conduct inappropriate teaching techniques in order to develop engineering student communication skills (Nasir et al., 2018; Baharudin et al., 2018). The study sought to examine the reliability and validity of an instrument use to measure Multimedia Instruction (MI) constructs. MI is used to deliver communication skills course for engineers. A research question of whether the measurement scale for MI is construct-valid was formed to guide the study.
2. Past Study
2.1. Multimedia Instruction (MI)
Multimedia learning is the learning that “occurs when people build mental representations from words (such as spoken text or printed text) and pictures (such as illustrations, photos, animation, or video)” (Mayer, 2009). Mayer had investigated a number of instructional design principles, which had suggested ways of creating multimedia presentations intended to promote multimedia learning. Mayer’s Cognitive Theory of MI acknowledged that for successful learning to occur, information must be synthesized into a coherent model of knowledge and integrated into long-term memory all within the working memory store.
Multimedia has remarkable impact on learning (Zainal et al., 2018; Gabarre et al. 2018; Ahmad et al. 2016; Azizul & Din, 2016). It has emerged in various form of recourses and equipment which can be use to aid the instructor and learner effort in ensuring effective learning environment (Ahmad et al., 2016). When meeting all these conditions, there would be an opportunity to increase efficiency and effectiveness of engineering communication skills specifically in Al al-Bayt University in Jordan.
The usage of MI in engineering education has changed the practice of structured engineering. MI in the circumstance of structured engineering enhances the engineering process. It can be used in the design and construction of a building including safety management in construction work, and computer-aided design and construction. The MI through video conferencing, shared-screen computing and remote multimedia links on construction projects could have a significant impact on inter-professional communication in engineering education (Di Gironimo et al., 2013). In short, this approach for learning with high-quality instructional materials can reduce lecture time and learners as well as instructors efforts (Khalid & Quick, 2016).
2.2. Learning Styles
According to Romanelli at el., (2009) “learning styles” are “characteristic cognitive, effective, and psychosocial behaviors that serve as relatively stable indicators of how learners perceive, interact with, and respond to the learning environment”. Mismatches exist between common learning styles of engineering students and traditional teaching styles of engineering professors. In consequence, students become bored and inattentive in class, and do poorly on tests. According to Felder (2002), learning-style model classifies students according to the ways they receive and process information on a number of scales pertaining. A model intended to be particularly applicable to engineering education. Sensing and intuitive learners, visual and verbal learners, active and reflective learners sequential and global learners are describe in engineering student learning styles model (Felder & Silverman, 1988; 2002). Din (2010, 2017) also acknowledged that learning style can be divided into social and sensory learning style.
3. Methodology
This study uses a quantitative research approach via a survey distributed at a University in Jordan. It is about measuring engineering students’ level of MI. It was constructed based on reviewed literature specifically grounded on Mayer’s Cognitive Theory of Multimedia Learning (Mayer 2010; 2009).
3.1. Respondents
The sample was 166 engineering learners from Al-Bayt University in Jordan who enrolled in the first semester of an academic year. They were selected purposively from four major engineering courses: communication skills course; the provisions of the building; skills practice of the profession course; technical skills course. The sample size was still within the acceptable range (Hoyle, 1995).
3.2. Instrument
The MI item was developed specifically based on Mayer’s Cognitive Theory of Multimedia Learning (Mayer 2010; 2009). The instrument consists of thirty-one items; five Items for each indicator (modality, contiguity, personalization, redundancy, signaling), while six items were for coherence measure. The measurements scale is a Likert-type scale, which has 1 to 5 scales; 1 equals “strongly disagree” and 5 equals “strongly agree.” 1 represents the lowest and most negative impression on the scale, 3 represents an adequate impression, and 5 represents the highest and most positive impression. In addition, a response category for “Not Applicable” was added for each Likert item. Table 1 shows the contents of the MI measure after the content validation.
CFA was conducted on the hypothesized six-factor structure model using AMOS model-fitting program. The program adopted maximum likelihood estimation to generate estimates in the full-fledged measurement model. At the beginning, this model indicates the latent variable, MI, to be measured by six
observed variables (modality, contiguity, personalization, coherence, redundancy, and signaling). The construct MI was indicated by six measured indicators and was identified. It had more degrees of freedom than the paths to be estimated.
To assess the fit of the measurement model, the analysis relied on a number of descriptive fit indices, which included the 1) relative chi-square (χ2/df), 2) comparative fit index (CFI), 3) Tucker-Lewis coefficient (TLI), and 4) root mean square error approximation (RMSEA). Hair et al. (2006) suggest the use of relative chi-square (chi-square/df) as a fit measure.
4. Results
The estimated six-factor model for MI using the data drawn from 153usable and completed responses. A value of approximately .08 or less for RMSEA shows a reasonable error of estimation. The items from each scale were assumed to load only on the respective latent variable. The results indicate that the parameters ranging from .11 to .97. In the MI case, all of the coefficients were acceptable (>.7) except for the Coherence indicator which was .11 ( Hair et al. 2006; Arbuckle 1997; James et al. 2006). The CFI (.922) exceed the threshold of .90 indicating a good fit ( Hair et al. 2006, Arbuckle 1997, James et al. 2006), while the TLI (.870) fit indicators did not exceed the threshold of .90, indicating a poor fit ( Hair et al. 2006; Arbuckle 1997; James et al. 2006). The root-mean square error of approximation (RMSEA = .160) was >.08, Chi-square (χ2) was 44.083 with degree of freedom (9) and p value = 0 (normally acceptable at p > .05) reflecting a possible fit problem ( Hair et al. 2006; Arbuckle 1997; James et al. 2006).
Since the hypothesized model was found to be contaminated (p value = 0 and TLI (.870) is less than .9), the model was revised. The revised model was achieved after examining the modification indices in order to correlate the measurement errors of the signaling with contiguity, as well as correlate the measurement errors of the redundancy with personalization, and contiguity and modality factors. After a number of iteration to fit the model the results reflect a possible fit problem, yet no possible modifications could be made. As suggested by Aryee and Lee (2005) the researcher decided to delete a factor with the lowest loading where the “coherence” factor was dropped.
To validate the likelihood of the revised five-factor model, another CFA was applied on the same sample. The magnitude of the factor loadings in the revised model was substantially significant with CFI = .997, TLI =. 986 and chi-square = 3.971. The parameters were free from offending estimates, ranging from .72 to. 99. The CFI (.998) and TLI (.983) fit indicators exceeded the threshold of .90, indicating a very good fit ( Hair et al. 2006, Arbuckle 1997). The root-mean square error of approximation (RMSEA = .72) also indicated a good fit (Hair et al. 2006).
In this revised model, the chi-square (χ2) with a value of 1.784, with degree of freedom (1) successfully met the required threshold of <5, indicating a high goodness-of-fit value. The p value of .182 (acceptable at p > .05) hence indicates that the test failed to reject the hypothesized model. The procedures established the model in Figure 1 as the validated confirmatory measurement model. The cronbach alphas for the five sub-constructs after CFA are range from .814 to .879 (Modality = .839, Contiguity = .814, Personalization = .879, Redundancy = .873 and Signaling = .813) while cronbach alpha for the whole section measures = .927. It is worth to mention cronbach alpha for coherence” factor which was dropped was .651. Overall result indicates that the test failed to reject the hypothesized model. Thus, the procedures established that the model in Figure 1 was a validated confirmatory measurement model.
5. Discussion
The study was able to validate the MI model, which is measured by five observed variables: modality, contiguity, personalization, redundancy, and signaling. As
![]()
Figure 1. Revised CFA measurement model for MI.
proposed in literature, the study offered evidence that (five out of the six dimensions excluding the coherence indicator) the five-dimension measurement model did generate from data collected from Al al-Bayt university engineering learners. The result did not establish any basis, which can be used to claim that the MI model is incorrect. Thus, MI measurement model can be explained by five factors namely, redundancy, contiguity, personalization, modality, and signaling. This result was consistent with several literatures on MI (Gerjets at el. 2004; Ibrahim & Callaway, 2012a; 2012b; Sorden, 2012; Clark & Mayer, 2011).
This conception represents a major adjustment in the way engineering faculties have usually developed engineering communication skills. The results of the present study are relevant to give insights for theorists, learners, academic staff and knowledge management system designers and developers towards the goal of achieving effective learning and teaching environment for engineering communication skills. In addition, these findings would assist engineering learners with differentiated learning style preferences to learn and practice engineering communication skills knowledge by integrating MI theories into the learning environment via Blackboard Course Management System especially in Jordan higher learning education.
Acknowledgements
This publication and research were funded by the Faculty of Education and Universiti Kebangsaan Malaysia under the project code GG-2019-046 and PP-FPEND-2019, DCP-2017-202/3, GG2018-011 respectively. The authors would also like to express special thanks to all personalized education research team members, students, researchers, graduate and research assistants for their help and contribution in conducting this project.