A Procedure for Diagnostically Modeling Extant Large-Scale Assessment Data: The Case of the Programme for International Student Assessment in Reading


Cognitive diagnosis models (CDMs) are psychometric models developed mainly to assess examinees’ specific strengths and weaknesses of a set of skills or attributes within a domain. Recently, several methodological developments have been added to the CDM literature, which include the development of general and reduced CDMs, various absolute and relative fit measures at both the test and item levels, and a general Q-matrix validation procedure. Building on these developments, this research proposes a systematic procedure to diagnostically model extant large-scale assessment data. The procedure can be divided into four phases: construction of initial attributes and Q-matrices, construction of final attributes and Q-matrix, evaluation of reduced CDMs, and crossvalidation of the selected model. Working with language experts, we use data from the PISA 2000 reading assessment to illustrate the procedure.

Share and Cite:

Chen, J. & Torre, J. (2014). A Procedure for Diagnostically Modeling Extant Large-Scale Assessment Data: The Case of the Programme for International Student Assessment in Reading. Psychology, 5, 1967-1978. doi: 10.4236/psych.2014.518200.

Conflicts of Interest

The authors declare no conflicts of interest.


[1] Akaike, H. (1974). A New Look at the Statistical Identification Model. IEEE Transactions on Automated Control, 19, 716-723.
[2] Chen, J., & de la Torre, J. (2012). An Extension of the G-DINA Model for Polytomous Attributes. Paper Presented at the Annual Meeting of American Educational Research Association, Vancouver.
[3] Chen, J., & de la Torre, J. (2013). A General Cognitive Diagnosis Model for Expert-Defined Polytomous Attributes. Applied Psychological Measurement, 37, 419-437.
[4] Chen, J., de la Torre, J., & Zhang, Z. (2013). Relative and Absolute Fit Evaluation in Cognitive Diagnosis Modeling. Journal of Educational Measurement, 50, 123-140.
[5] de la Torre, J. (2008). An Empirically-Based Method of Q-Matrix Validation for the DINA Model: Development and Applications. Journal of Educational Measurement, 45, 343-362.
[6] de la Torre, J. (2011). The Generalized DINA Model Framework. Psychometrika, 76, 179-199.
[7] de la Torre, J., & Chen, J. (2011). Estimating Different Reduced Cognitive Diagnosis Models Using a General Framework. Paper Presented at the Annual Meeting of the National Council on Measurement in Education, New Orleans.
[8] de la Torre, J., & Chiu, C.-Y. (2010). A General Method of Empirical Q-Matrix Validation Using the G-DINA Model Discrimination Index. Paper Presented at the Annual Meeting of the National Council on Measurement in Education, Denver.
[9] de la Torre, J., & Lee, Y.-S. (2010). Item-Level Comparison of Saturated and Reduced Cognitive Diagnosis Models. Paper Presented at the Annual Meeting of the National Council on Measurement in Education, Denver.
[10] DiBello, L., Roussos, L., & Stout, W. (2007). Cognitive Diagnosis Part I. In C. R. Rao, & S. Sinharay (Eds.), Handbook of Statistics (Vol. 26): Psychometrics (pp. 979-1030). Amsterdam: Elsevier.
[11] Haertel, E. H. (1989). Using Restricted Latent Class Models to Map the Skill Structure of Achievement Items. Journal of Educational Measurement, 26, 301-321.
[12] Hartz, S. (2002). A Bayesian Framework for the Unified Model for Assessing Cognitive Abilities: Blending Theory with Practicality. Unpublished Doctoral Dissertation, University of Illinois at Urbana-Champaign.
[13] Henson, R. A., Templin, J., & Willse, J. (2009). Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables. Psychometrika, 74, 191-210.
[14] Jang, E. E. (2009). Cognitive Diagnostics Assessment of L2 Reading Comprehension Ability: Validity Arguments for Fusion Model Application to Langu Edge Assessment. Language Testing, 26, 31-73.
[15] Junker, B. W., & Sijtsma, K. (2001). Cognitive Assessment Models with Few Assumptions, and Connections with Non-Parametric Item Response Theory. Applied Psychological Measurement, 25, 258-272.
[16] Kunina-Habenicht, O., Rupp, A. A., & Wilhelm, O. (2012). The Impact of Model Misspecification on Parameter Estimation and Item-Fit Assessment in Log-Linear Diagnostic Classification Models. Journal of Educational Measurement, 49, 59-81.
[17] Lee, Y., & Sawaki, Y. (2009). Application of Three Cognitive Diagnosis Models to ESL Reading and Listening Assessments. Language Assessment Quarterly, 6, 239-263.
[18] Maris, E. (1999). Estimating Multiple Classification Latent Class Models. Psychometrika, 64, 187-212.
[19] OECD (1999). Measuring Student Knowledge and Skills: A New Framework for Assessment. Paris: Organization for Economic Cooperation and Development.
[20] OECD (2006a). Assessing Scientific, Reading and Mathematical Literacy: A Framework for PISA 2006. Paris: Organization for Economic Cooperation and Development.
[21] OECD (2006b). PISA Released Items: Reading.
[22] Schwarzer, G. (1976). Estimating the Dimension of a Model. Annals of Statistics, 6, 461-464.
[23] Tatsuoka, K. K. (1983). Rule Space: An Approach for Dealing with Misconceptions Based on Item Response Theory. Journal of Educational Measurement, 20, 345-354.
[24] Tatsuoka, K. K. (1990). Toward an Integration of Item-Response Theory and Cognitive Error Diagnosis. In N. Frederiksen, R. Glaser, A. Lesgold, & M. Safto (Eds.), Diagnostic Monitoring Skills and Knowledge Acquisition (pp. 453-488). Hillsdale, NJ: Erlbaum.
[25] Templin, J., & Henson, R. A. (2006). Measurement of Psychological Disorders Using Cognitive Diagnosis Models. Psychological Methods, 11, 287-305.
[26] von Davier, M. (2008). A General Diagnostic Model Applied to Language Testing Data. British Journal of Mathematical and Statistical Psychology, 61, 287-307.

Copyright © 2022 by authors and Scientific Research Publishing Inc.

Creative Commons License

This work and the related PDF file are licensed under a Creative Commons Attribution 4.0 International License.