The Efficacy of the IRTree Framework for Detecting Missing Data Mechanisms in Educational Assessments

Year 2024, Volume: 15 Issue: 3, 209 - 220, 26.10.2024

Yeşim Beril Soğuksu

https://doi.org/10.21031/epod.1514741

Abstract

The effectiveness of methods for handling missing data in educational assessments depends on understanding the underlying missing mechanisms. This study investigates the performance of the IRTree framework in detecting missing data mechanisms using a Monte Carlo simulation. Omitted responses were simulated at varying proportions according to three mechanisms: MCAR, MAR, and MNAR, across tests with different lengths and sample sizes. The IRTree was employed to model the omitted responses and detect the mechanisms based on the correlations between the propensity to omit and proficiency. Results indicate that the IRTree accurately identifies all three missing data mechanisms, with no relationship between propensity to omit and proficiency under MCAR, and negative correlations for MAR, reaching up to -0.3, and for MNAR, as high as -0.8. Furthermore, the detection of MAR and MNAR mechanisms became more pronounced with higher proportions of omitted responses, longer tests, and larger sample sizes. IRTree framework not only enables educators and researchers to accurately understand the nature of missing data but also guides them in using appropriate methods for handling it.

Keywords

IRTree, missing data, missing mechanism, simulation, R language

References

• Alagöz, Ö. E. C., & Meiser, T. (2023). Investigating heterogeneity in response strategies: A mixture multidimensional IRTree approach. Educational and Psychological Measurement, 84(5), 957-993. https://doi.org/10.1177/00131644231206765
• Alarcon, G. M., Lee, M. A., & Johnson, D. (2023). A Monte Carlo study of IRTree models' ability to recover item parameters. Frontiers In Psychology, 14, 1003756. https://doi.org/10.3389/fpsyg.2023.1003756
• Allison, P. D. (2002). Missing data. Sage Publications.
• Baker, F. B. (2001). The basics of item response theory. ERIC Clearinghouse on Assessment and Evaluation.
• Bock, R. D., & Aitkin M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46(4), 443-459. https://doi.org/10.1007/BF02293801
• Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6(4), 431-444. https://doi.org/10.1177/014662168200600405
• Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 397-460). MA: Addison-Wesley.
• Böckenholt, U. (2012). Modeling multiple response processes in judgment and choice. Psychological Methods, 17(4), 665-678. https://doi.org/10.1037/a0028111
• Böckenholt, U. (2017). Measuring response styles in Likert items. Psychological methods, 22(1), 69–83. https://doi.org/10.1037/met0000106
• Chalmers, R. P. (2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06
• Cheema, J. R. (2014). A review of missing data handling methods in education research. Review of Educational Research, 84(4), 487-508. https://doi.org/10.3102/0034654314532697
• Collins, L. M., Schafer, J. L., & Kam, C. M. (2001) A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6(4), 330-51.
• Damiani, V. (2016). Large-scale assessments and educational policies in Italy. Research Papers in Education, 31(5), 529–541.
• De Ayala, R. J., Plake, B. S., & Impara, J. C. (2001). The impact of omitted responses on the accuracy of ability estimation in item response theory. Journal of Educational Measurement, 38(3), 213–234. https://doi.org/10.1111/j.1745-3984.2001.tb01124.x
• De Boeck, P., & Partchev, I. (2012). IRTrees: Tree based item response models of the GLMM family. Journal of Statistical Software, 48, 1-28. https://doi.org/10.18637/jss.v048.c01
• Debeer, D., Janssen, R., & De Boeck, P. (2017). Modeling skipped and not-reached items using IRTrees. Journal of Educational Measurement, 54(3), 333-363. https://doi.org/10.1111/jedm.12147
• DeMars, C. (2010). Item response theory: Understanding statistics measurement. Oxford University Press.
• Dibek, M. I. (2019). Examination of the extreme response style of students using IRTree: The case of TIMMS 2015. International Journal of Assessment Tools in Education, 6, 300-313. https://doi.org/10.21449/ijate.534118
• Enders, C. K. (2010). Applied missing data analysis. The Guilford Press.
• Feinberg, R. A., & Rubright, J. D. (2016). Conducting simulation studies in psychometrics. Educational Measurement: Issues and Practice, 35(2), 36-49. https://doi.org/10.1111/emip.12111
• Glas, C. A. W., & Pimentel, J. L. (2008). Modeling nonignorable missing data in speeded tests. Educational and Psychological Measurement, 48(6), 907-922. https://doi.org/10.1177/0013164408315262
• Glas, C. A. W., Pimentel, J. L., & Lamers, S. M. A. (2015). Nonignorable data in IRT models: Polytomous models with covariates. Psychological Test and Assessment Modeling, 57(4), 523-541.
• Graham, J. W. (2012). Missing data analysis and design. Springer. Hambleton, R. K., Swaminathan, H. & Rogers, H. J. (1991). Fundamentals of item response theory. California: Sage Publications.
• Harwell, M., Stone, C. A., Hsu, T. C., & Kirisci, L. (1996). Monte Carlo studies in item response theory. Applied Psychological Measurement, 20(2), 101-125. https://doi.org/10.1177/014662169602000201
• Holman, R., & Glas, C. A. (2005). Modelling non-ignorable missing-data mechanisms with item response theory models. British Journal of Mathematical and Statistical Pyschology, 58, 1-17. https://doi.org/10.1111/j.2044-8317.2005.tb00312.x
• Huang, H. Y. (2020). A mixture IRTree model for performance decline and nonignorable missing data. Educational and Psychological Measurement, 80(6), 1168-1195. https://doi.org/10.1177/0013164420914711
• Huisman, M. (2000). Imputation of missing item responses: Some simple techniques. Quality & Quantity, 34, 331–351. https://doi.org/10.1023/A:1004782230065
• Jeon, M., & De Boeck, P. (2016). A generalized item response tree model for psychological assessments. Behavior Research Methods, 48, 1070-1085. https://doi.org/10.3758/s13428-015-0631-y
• Jeon, M., De Boeck, P., & van der Linden, W. (2017). Modeling answer change behavior: An application of a generalized item response tree model. Journal of Educational and Behavioral Statistics, 42(4), 467-490. https://doi.org/10.3102/1076998616688015
• Jeon, M., Rijmen, F. & Rabe-Hesketh, S. (2014). Flexible item response theory modeling with FLIRT. Applied Psychological Measurement, 38, 404-405. https://doi.org/10.1177/0146621614524982
• Jin, K.-Y., Wu, Y.-J., & Chen, H.-F. (2022). A new multiprocess IRT model with ideal points for likert-type items. Journal of Educational and Behavioral Statistics, 47(3), 297-321. https://doi.org/10.3102/10769986211057160
• Köhler, C., Pohl, S., & Carstensen, C. (2017). Dealing with item nonresponse in large‐scale cognitive assessments: The impact of missing data methods on estimated explanatory relationships. Journal of Educational Measurement, 54, 397-419. https://doi.org/10.1111/jedm.12154
• Leventhal, B. C. (2019). Extreme response style: A simulation study comparison of three multidimensional item response models. Applied Psychological Measurement, 43(4), 322–335. https://doi.org/10.1177/0146621618789392
• Little, R. J. A., & Rubin, D. B. (1987). Statistical analysis with missing data. John Wiley & Sons.
• Little, R. J. A. (1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83, 1198–1202.
• Little, T. D., Lang, K. M., Wu, W., & Rhemtulla, M. (2016). Developmental psychopathology. In D. Cicchetti (Ed.), Missing Data (pp. 760-797). John Wiley & Sons.
• Martens, K., Niemann, D., & Teltemann, J. (2016). Effects of international assessments in education – a multidisciplinary review. European Educational Research Journal, 15(5), 516-522. https://doi.org/10.1177/1474904116668886
• McKnight, P. E., McKnight, K. M., Sidani, S. & Figueredo, A. J. (2007). Missing data: A gentle introduction. Guilford Press.
• Newman, D. A. (2014). Missing data: Five practical guidelines. Organizational research methods, 17(4), 372-411. https://doi.org/10.1177/1094428114548590
• Park, M., & Wu, A. D. (2019). Item response tree models to investigate acquiescence and extreme response styles in Likert-type rating scales. Educational and Psychological Measurement, 79(5), 911–930. https://doi.org/10.1177/0013164419829855
• Peugh, J. L., & Enders, C. K. (2004). Missing data in educational research: A review of reporting practices and suggestions for improvement. Review of Educational Research, 74(4), 525-556. https://doi.org/10.3102/00346543074004525
• Pigott, T. D. (2010). A review of methods for missing data. Educational Research and Evaluation: An International Journal on Theory and Practice, 7(4), 353-383. https://doi.org/10.1076/edre.7.4.353.8937
• Plieninger, H. (2021). Developing and applying Ir-Tree models: Guidelines, caveats, and an extension to multiple groups. Organizational Research Methods, 24(3), 654-670. https://doi.org/10.1177/1094428120911096
• Pohl, S., Gräfe, L. & Rose, N. (2014). Dealing with omitted and not-reached items in competence tests: Evaluating approaches accounting for missing responses in item response theory models. Educational and Psychological Measurement, 74(3), 423–452. https://doi.org/10.1177/0013164413504926
• Quirk, V. L., & Kern, J. L. (2023). Using IRTree models to promote selection validity in the presence of extreme response styles. Journal of Intelligence, 11(11), 216. https://doi.org/10.3390/jintelligence11110216
• Rose, N., von Davier, M., & Nagengast, B. (2015). Modeling omitted and not-reached items in IRT models. Psychometrika, 82, 795-819. https://doi.org/10.1007/s11336-016-9544-7
• Rose, N., von Davier, M., & Xu, X. (2010). Modeling nonignorable missing data with item response theory (IRT) (ETS Research Report No. RR-10-11). Educational Testing Service.
• Roth, P. L. (1994). Missing data: A conceptual review for applied psychologists. Personnel Psychology, 47(3), 537–560. https://doi.org/10.1111/j.1744-6570.1994.tb01736.x.
• Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581-592. https://doi.org/10.1093/biomet/63.3.581
• Spratto, E. M., Leventhal, B. C., & Bandalos, D. L. (2021). Seeing the forest and the trees: Comparison of two IRTree models to investigate the impact of full versus endpoint-only response option labeling. Educational and Psychological Measurement, 81(1), 39-60. https://doi.org/10.1177/0013164420918655
• Sulis, I., & Porcu, M. (2017). Handling missing data in item response theory. Assessing the accuracy of a multiple imputation procedure based on latent class analysis. Journal of Classification, 34, 327–359. https://doi.org/10.1007/s00357-017-9220-3
• Tabachnick, B. G., & Fidell L. S. (2007). Using multivariate statistics. Allyn and Bacon.