Comparison of GDINA, DINA, DINO Model Fit in Cognitive Diagnosis at Item and Test Level Based on Real Data

Year 2022, Volume: 42 Issue: 2, 1083 - 1108, 29.08.2022

Seyhan Sarıtaş Akyol , Mehtap Çakan

https://doi.org/10.17152/gefad.1005950

Abstract

Cognitive diagnostic models enable the creation of individuals' attribute profiles through
particular categorical latent variables. In order to collect evidence of whether the inferences
drawn from these models are valid or not, it is of great importance to reveal the model-data fit
and to identify the models that fit. This study aims to determine whether reduced DINA and DINO
models can be used instead of the saturated GDINA model determined within cognitive diagnostic
modelling framework by using Wald statistics for each item level and to specify the model that
best fits the data by comparing it at the item and test level. The data set of the study was obtained
from 478 students and the answer pattern of the Symbolic Number Comparison test comprising
36 items. Analyses were performed in R 3.5.3 program and the GDINA package in the R program
was used. In the first stage, the saturated GDINA model was taken as the basis, and the absolute
and relative fit indices were evaluated together and it was seen that the saturated model gave the
best fit. Afterward, using Wald statistics, instead of the saturated GDINA model, the availability
of reduced DINA and DINO models for each item-level with no significant loss in model-data fit
was examined.

Keywords

Wald test, Model comparison, Model fit, CDM

Bilişsel Tanı Çerçevesinde GDINA, DINA, DINO Model Uyumlarının Madde ve Test Düzeyinde Gerçek Veriye Dayalı Olarak Karşılaştırılması

Abstract

Bilişsel tanı modelleri belirli kategorik gizil özellikler aracılığıyla bireylerin nitelik profillerinin oluşturulmasına imkân vermektedir. Bu modellerden elde edilen çıkarımların geçerli olup olmadığına yönelik kanıt toplamak için model veri uyumunun ortaya konması ve uygun modellerin belirlenmesi büyük önem taşımaktadır. Bu çalışmada, bilişsel tanı yaklaşımı kapsamında belirlenen doymuş GDINA modeli yerine indirgenmiş (reduced) DINA ve DINO modellerinin çalışma verisi için kullanılıp kullanılamayacağı her madde düzeyi için Wald istatistiğinin kullanılarak belirlenmesi ve veriye en iyi uyum sağlayan modelin madde ve test düzeyinde karşılaştırılarak belirlenmesi amaçlanmıştır. Veri seti olarak 478 öğrenciden elde edilen, 36 maddeden oluşan Sayı Karşılaştırma testine ait cevap örüntüsü kullanılmıştır. Analizler R 3.5.3 programında yürütülmüş R programındaki GDINA paketinden yararlanılmıştır. Çalışmada ilk olarak, temele doymuş GDINA modeli alınarak mutlak ve göreli uyum indeksleri birlikte değerlendirilmiş ve doymuş modelin en iyi uyumu verdiği görülmüştür. Devamında Wald istatistiği kullanılarak doymuş GDINA modeli yerine her madde düzeyi için model veri uyumunda
manidar bir kayıp olmadan indirgenmiş DINA ve DINO modellerinin kullanılabilme durumu incelenmiştir.

Keywords

Wald testi, Model karşılaştırma, Model uyumu, BTM

References

• Başokçu, O. T. (2012). DINA model parametreleri kullanılarak tahminlenen madde ayırıcılık indekslerinin incelenmesi. Eğitim ve Bilim, 37(163).
• Basokcu, T. O., Ogretmen, T., & Kelecioglu, H. (2013). Model data fit comparison between DINA and G-DINA in cognitive diagnostic models. Education Journal, 2(6), 256-262. http://doi.org /10.11648/j.edu.20130206.18
• Bradshaw, L., Izsak, A., Templin, J., & Jacobson, E. (2014). Diagnosing teachers’ understandings of rational numbers: Building a multidimensional test within the diagnostic Classification framework. Educational Measurement: Issues and Practice, 33, 2–14.
• Chen, J., de la Torre, J., & Zhang, Z. (2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50, 123- 140.
• de la Torre, J. (2008). An empirically based method of Q‐matrix validation for the DINA model: Development and applications. Journal of educational measurement, 45(4), 343-362.
• de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76, 179-199.
• de la Torre, J., & Douglas, J. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333-353.
• de la Torre, J., & Lee, Y. S. (2013). Evaluating the Wald test for item‐level comparison of saturated and reduced models in cognitive diagnosis. Journal of Educational Measurement, 50(4), 355-373.
• Demir, E. K., & Koç, N. (2018). DINA model ile geliştirilen bir testin psikometrik özelliklerinin belirlenmesi. Abant İzzet Baysal Üniversitesi Eğitim Fakültesi Dergisi, 18(1), 130-156.
• Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333–352.
• Han, Z., & Johnson, M. S. (2019). Global-and item-level model fit indices. In Handbook of Diagnostic Classification Models (pp. 265-285). Springer, Cham.
• Hartz, S., Roussos, L., & Stout, W. (2002). Skills diagnosis: Theory and Practice. User Manual for Arpeggio software [Computer software manual]. Princeton, NJ: Educational Testing Service.
• Hu, J., Miller, M. D., Huggins-Manley, A. C., & Chen, Y. H. (2016). Evaluation of Model Fit in Cognitive Diagnosis Models, International Journal of Testing, 16:2, 119-141, DOI: 10.1080/15305058.2015.1133627.
• Jiao, H. (2009). Diagnostic classification models: Which one should I use? Measurement: Interdisciplinary Research & Perspective, 7(1), 65–67.
• Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258– 272. DOI:10.1177/01466210122032064
• Kalkan, Ö. K. (2016). Bilişsel tanı modellerinin değişen koşullar altında karşılaştırılması: DINA, RDINA, HODINA ve HORDINA Modelleri. Doktora Tezi, Hacettepe Üniversitesi Eğitim Bilimleri Enstitüsü, Ankara.
• Koyuncu, M. S. (2020). Bilişsel tanı modellerinde yapısal eşitlik modeli ile q-matris doğruluğunun belirlenmesi. Doktora Tezi, Gazi Üniversitesi Eğitim Bilimleri Enstitüsü, Ankara.
• Kunina-Habenicht, O., Rupp, A. A., & Wilhelm, O. (2009). A practical illustration of multidimensional diagnostic skills profiling: Comparing results from confirmatory factor analysis and diagnostic classification models. Studies in Educational Evaluation, 35(2), 64-70.
• Lee, Y. S., Park, Y. S., & Taylan, D. (2011). A cognitive diagnostic modeling of attribute mastery in Massachusetts, Minnesota, and the US national sample using the TIMSS 2007. International Journal of Testing, 11, 144–177.
• Leighton, J. P., & Gierl, M. J. (2007). Defining and evaluating models of cognition used in educational measurement to make inferences about examinees’ thinking processes. Educational Measurement: Issues and Practice, 26, 3-16.
• Li, F., Cohen, A. S., Kim, S. H., & Cho, S. J. (2009). Model selection methods for mixture dichotomous IRT models. Applied Psychological Measurement, 33(5), 353-373.
• Liu, Y., Tian, W., & Xin, T. (2016). An Application of M2 Statistic to Evaluate the Fit of Cognitive Diagnostic Models. Journal of Educational and Behavioral Statistics, 41, 3-26.
• Ma, W., Iaconangelo, C., & de la Torre, J. (2016). Model similarity, model selection and attribute classification. Applied Psychological Measurement, 40, 200-217. https://doi.org/10.1177/0146621615621717
• Ma, W., & de la Torre, J. (2020). GDINA: an R package for cognitive diagnosis modeling. Journal of Statistical Software, 93(14), 1-26.
• Macready, G. B., & Dayton, C. M. (1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 2, 99–120.
• Maydeu-Olivares, A., & Joe, H. (2005). Limited and full information estimation and goodness of-fit testing in 2n contingency tables: A unified framework. Journal of the American Statistical Association, 100, 1009–1020. Sarıtaş Akyol & Çakan 1105
• Maydeu-Olivares, A., & Joe, H. (2006). Limited information goodness-offit testing in multidimensional contingency tables. Psychometrika, 71, 713–732.
• Olkun, S. (2015). 6-11 Yaş Türk Çocukları Örnekleminde Diskalkuliye Yatkınlığı Ayırt Etmede Kullanılacak Bir Ölçme Aracı Geliştirme Çalışması. 111K545 Nolu TÜBİTAK Projesi, Ankara, Türkiye.
• Rojas, G., de la Torre, J., & Olea, J. (2012, April). Choosing between general and specific cognitive diagnosis models when the sample size is small. Paper presented at the meeting of the National Council on Measurement in Education, Vancouver, Canada.
• Rupp, A., Templin, J, & Henson, R. (2010). Diagnostic Measurement: Theory, Methods, and Applications. New York: The Guildford.
• Rupp, A. A., & Templin, J. L. (2008). Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement: Interdisciplinary Research and Perspectives, 6(4), 219–262.
• Sen, S., & Bradshaw, L. (2017). Comparison of relative fit indices for diagnostic model selection. Applied psychological measurement, 41(6), 422-438. https://doi.org/10.1177/0146621617695521
• Tatsuoka, K. K. (1983). Rule Space: An Approach for Dealing with Misconceptions Based on Item Response Theory. Journal of Educational Measurement, 20(4), 345–354. Retrieved from http://www.jstor.org/stable/1434951
• Templin, J. & Henson, R. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11 (3), 287-305.
• von Davier, M. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61, 287–307.
• von Davier, M., & Lee, Y. S. (2019). Handbook of Diagnostic Classification Models. New York, NY: Springer.