The Impact of Missing Data on the Performances of DIF Detection Methods

Year 2023, Volume: 14 Issue: 1, 95 - 105, 25.03.2023

Rabia Akcan , Kübra Atalay Kabasakal

https://doi.org/10.21031/epod.1183617

Abstract

This study analyzed the impact of missing data techniques on performances of two differential item functioning (DIF) detection methods (Mantel Haenszel and Multiple Indicator and Multiple Causes) under missing completely at random missing data mechanism. Percentage of missing data was set at 5% and 15%. Zero imputation, listwise deletion and fractional hot-deck imputation were used to handle missing data. The data set of the study consisted of 17 items in the S12 item cluster of Programme for International Student Assessment (PISA) 2015 science test. Results showed that fractional hot-deck imputation produced the best results in identifying DIF items in all conditions and it had also the closest DIF values to the values obtained from complete data set. It was also found that multiple indicator and multiple causes method was more adversely affected than Mantel Haenszel by the presence of missing data.

Keywords

Differential item functioning, Mantel Haenszel, MIMIC, missing data

References

• Banks, K. (2015). An introduction to missing data in the context of differential item functioning. Practical Assessment, Research & Evaluation, 20(12), 1-10. https://eric.ed.gov/?id=EJ1059748
• Banks, K., & Walker, C. (2006, April). Performance of SIBTEST when focal group examinees have missing data. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.
• Camilli, G., & Shepard, L. A. (1994). Methods for identifying biased test items. London Sage.
• Clauser, B. E., & Mazor, K. M. (1998). Using statistical procedures to identify differentially functioning test items. Educational Measurement Issues and Practice, 17(1), 31-44. https://eric.ed.gov/?id=EJ564712
• Doğanay Erdoğan, B. (2012). Çoklu atama yöntemlerinin Rasch modelleri için performansının benzetim çalışması ile incelenmesi [Assessing the performance of multiple imputation techniques for Rasch models with a simulation study] (Publication No. 314412) [Doctoral dissertation, Ankara University]. Council of Higher Education Thesis Center.
• Dong, Y., & Peng, C. Y. (2013). Principled missing data methods for researchers. Springer Plus, 2(1), 222. https://doi.org/10.1186/2193-1801-2-222
• Emenogu, B. C., Falenchuk, O., & Childs, R. A. (2010). The effect of missing data treatment on Mantel-Haenszel DIF detection. The Alberta Journal of Educational Research, 56(4), 459-469. https://doi.org/10.11575/ajer.v56i4.55429
• Finch, H. (2011a). The use of multiple imputation for missing data in uniform DIF analysis: Power and type I error rates. Applied Measurement in Education, 24(4), 281-301. https://doi.org/10.1080/08957347.2011.607054
• Finch, H. (2011b). The impact of missing data on the detection of nonuniform differential item functioning. Educational and Psychological Measurement, 71(4), 663-683. https://doi.org/10.1177/0013164410385226
• Finch, H. (2005). The MIMIC model as a method for detecting DIF: Comparison with Mantel-Haenszel, SIBTEST, and the IRT likelihood ratio. Applied Psychological Measurement, 29(4), 278-295. https://doi.org/10.1177/0146621605275728
• Finch, H. W., & French, B. F. (2007). Detection of crossing differential item functioning a comparison of four methods. Educational and Psychological Measurement, 67(4), 565-582. https://doi.org/10.1177/0013164406296975
• Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2012). How to design and evaluate research in education McGraw-hill.
• Garrett, P. L. (2009). A monte carlo study investigating missing data, differential item functioning, and effect size (Publication No. 3401601) [Doctoral dissertation, Georgia State University]. ProQuest Dissertations Publishing.
• Hallquist, M., & Wiley, J. (2018). MplusAutomation: An R package for facilitating large-scale latent variable analyses in Mplus. Structural Equation Modeling: A Multidisciplinary Journal, 25(4), 621-638. https://doi.org/10.1080/10705511.2017.1402334
• Harrington, D. (2009). Confirmatory factor analysis. Oxford University Press.
• Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer, & H. I. Braun, Test Validity (pp. 129-145). Lawrence Erlbaum.
• Holland, P. W., & Wainer, H. (1993). Differential item functioning. Lawrence Erlbaum.
• Im, J., Cho, I. H., & Kim, J. K. (2018). FHDI: Fractional hot deck and fully efficient fractional imputation. https://CRAN.R-project.org/package=FHDI
• Im, J., Kim, J. K., & Fuller, W. A. (2015). Two-phase sampling approach to fractional hot deck imputation. In Proceedings of the Survey Research Methods Section, pages 1030-1043. http://www.asasrms.org/Proceedings/y2015/files/233957.pdf
• Jin, KY., & Chen, HF. (2020). MIMIC approach to assessing differential item functioning with control of extreme response style. Behavior Research Methods, 52, 23-35. https://doi.org/10.3758/s13428-019-01198-1
• Kalton, G., & Kish, L. (1984). Some efficient random imputation methods. Communications in Statistics-Theory and Methods ,13(16), 1919-1939. https://doi.org/10.1080/03610928408828805
• Kim, J. K., & Fuller, W. (2004). Fractional hot deck imputation. Biometrika, 91(3), 559-578. https://doi.org/10.1093/biomet/91.3.559
• Magis, D., Beland, S., Tuerlinckx, F., & De Boeck, P. (2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42(3), 847-862. https://doi.org/10.3758/BRM.42.3.847
• Montoya, A. K., & Jeon, M. (2020). MIMIC models for uniform and nonuniform DIF as moderated mediation models. Applied Psychological Measurement, 44(2), 118-136. https://doi.org/10.1177/0146621619835496
• Peugh, J. L., & Enders, C. K. (2004). Missing data in educational research:A review of reporting practices and suggestions for improvement. Review of Educational Research, 74(4), 525-556. https://journals.sagepub.com/doi/pdf/10.3102/00346543074004525
• Robitzsch, A., & Rupp, A. A. (2009). Impact of missing data on the detection of differential item functioning: The case of Mantel-Haenszel and logistic regression analysis. Educational and Psychological Measurement, 69(1), 18-34. https://doi.org/10.1177/0013164408318756
• Rousseau, M., Bertrand, R., & Boiteau, N. (2004, April). Impact of missing data on robustness of DIF IRT-based and non IRT-based methods. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
• Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581-592. https://doi.org/10.1093/biomet/63.3.581
• Scheuneman, J. (1979). A method of assessing bias in test items. Journal of Educational Measurement, 16, 143–152. https://www.jstor.org/stable/1433816
• Shih, C. L., & Wang, W. C. (2009). Differential item functioning detection using multiple indicators, multiple causes method with a pure short anchor. Applied Psychological Measurement, 33(3), 184-199. https://doi.org/10.1177/0146621608321758
• Tamcı, P. (2018). Kayıp veriyle baş etme yöntemlerinin değişen madde fonksiyonu üzerindeki etkisinin incelenmesi [Investigation of the impact of techniques of handling missing data on differential item functioning] (Publication No. 517260) [Master's dissertation, Hacettepe University]. Council of Higher Education Thesis Center.
• Uğurlu, S., & Atar, B. (2020). Performances of MIMIC and logistic regression procedures in detecting DIF. Journal of Measurement and Evaluation in Education and Psychology, 11(1), 1-12. https://doi.org/10.21031/epod.531509
• Woods, C. M. (2009). Evaluation of MIMIC-model methods for DIF testing with comparison to two-group analysis. Multivariate Behavioral Research, 44(1), 1-27. https://doi.org/10.1080/00273170802620121
• Zieky, M. (1993). Practical questions in the use of DIF statistics in test development. In P. W. Holland, & H. Wainer, Differential Item Functioning (pp. 337-347). Lawrence Erlbaum.
• Zumbo, B. D. (2007). Three generation of DIF analyses: Considering where it has been, where it is now, and where it is going. Language Assessment Quarterly, 4(2), 223–233. https://doi.org/10.1080/15434300701375832