Comparison of confirmatory factor analysis estimation methods on mixed-format data

Year 2021, Volume: 8 Issue: 1, 21 - 37, 15.03.2021

Abdullah Faruk Kılıç , Nuri Doğan

https://doi.org/10.21449/ijate.782351

Cited By: 15

Abstract

Weighted least squares (WLS), weighted least squares mean-and-variance-adjusted (WLSMV), unweighted least squares mean-and-variance-adjusted (ULSMV), maximum likelihood (ML), robust maximum likelihood (MLR) and Bayesian estimation methods were compared in mixed item response type data via Monte Carlo simulation. The percentage of polytomous items, distribution of polytomous items, categories of polytomous items, average factor loading, sample size and test length conditions were manipulated. ULSMV and WLSMV were found to be the more accurate methods under all simulation conditions. All methods except WLS had acceptable relative bias and relative standard error bias. No method gives accurate results with small sample sizes and low factor loading, however, the ULSMV method can be recommended to researchers because it gives more appropriate results in all conditions.

Keywords

Bayesian estimation, Monte-carlo simulations, CFA

References

• AERA, APA, NCME, American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement In Education (NCME). (2014). Standards for educational and psychological testing. American Educational Research Association.
• Babakus, E., Ferguson, C. E., & Jöreskog, K. G. (1987). The sensitivity of confirmatory maximum likelihood factor analysis to violations of measurement scale and distributional assumptions. Journal of Marketing Research, 24(2), 222 228. https://doi.org/10.2307/3151512
• Bandalos, D. L. (2014). Relative performance of categorical diagonally weighted least squares and robust maximum likelihood estimation. Structural Equation Modeling: A Multidisciplinary Journal, 21(1), 102 116. https://doi.org/10.1080/10705511.2014.859510
• Bandalos, D. L., & Leite, W. (2013). Use of Monte Carlo studies in structural equation modeling research. In G. R. Hancock & R. O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed., pp. 625-666). Information Age.
• Beauducel, A., & Herzberg, P. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling: A Multidisciplinary Journal, 13(2), 186 203. https://doi.org/10.1207/s15328007sem1302_2
• Bollen, K. A. (1989). Structural equations with latent variables. John Wiley & Sons, Inc. https://doi.org/10.1002/9781118619179
• Byrne, B. M. (2016). Structural equation modeling with AMOS: Basic concepts, applications, and programming (3rd ed.). Routledge.
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
• Depaoli, S., & Scott, S. (2015). Frequentist and bayesian estimation of CFA measurement models with mixed item response types: A monte carlo investigation. Structural Equation Modeling: A Multidisciplinary Journal, (September), 1 16. https://doi.org/10.1080/10705511.2015.1044653 (Retraction published 2015, Structural Equation Modeling: A Multidisciplinary Journal, 318)
• DiStefano, C. (2002). The impact of categorization with confirmatory cactor analysis. Structural Equation Modeling: A Multidisciplinary Journal, 9(3), 327 346. https://doi.org/10.1207/S15328007SEM0903_2
• DiStefano, C., & Hess, B. (2005). Using confirmatory factor analysis for construct validation: An empirical review. Journal of Psychoeducational Assessment, 23(3), 225–241. https://doi.org/10.1177/073428290502300303
• Dolan, C. V. (1994). Factor analysis of variables with 2, 3, 5 and 7 response categories: A comparison of categorical variable estimators using simulated data. British Journal of Mathematical and Statistical Psychology, 47(2), 309–326. https://doi.org/10.1111/j.2044-8317.1994.tb01039.x
• Ferguson, E., & Rigdon, E. (1991). The performance of the polychoric correlation coefficient and selected fitting functions in confirmatory factor analysis with ordinal data. Journal of Marketing Research, 28(4), 491–497. https://doi.org/10.2307/3172790
• Finney, S. J., & DiStefano, C. (2013). Nonnormal and categorical data in structural equation modeling. In G. R. Hancock & R. O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed., pp. 439–492). Information Age.
• Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466–491. https://doi.org/10.1037/1082-989X.9.4.466
• Forero, C. G., Maydeu-Olivares, A., & Gallardo-Pujol, D. (2009). Factor analysis with ordinal indicators: A monte carlo study comparing DWLS and ULS estimation. Structural Equation Modeling: A Multidisciplinary Journal, 16(4), 625 641. https://doi.org/10.1080/10705510903203573
• Green, S. B., Akey, T. M., Fleming, K. K., Hershberger, S. L., & Marquis, J. G. (1997). Effect of the number of scale points on chi‐square fit indices in confirmatory factor analysis. Structural Equation Modeling: A Multidisciplinary Journal, 4(2), 108–120. https://doi.org/10.1080/10705519709540064
• Guilford, J. P. (1946). New standards for test evaluation. Educational and Psychological Measurement, 6(4), 427–438. https://doi.org/10.1177/001316444600600401
• Hallquist, M., & Wiley, J. (2017). MplusAutomation: Automating Mplus model estimation and interpretation. Retrieved from https://cran.r-project.org/package=MplusAutomation
• Holtmann, J., Koch, T., Lochner, K., & Eid, M. (2016). A comparison of ML, WLSMV, and bayesian methods for multilevel structural equation models in small samples: A simulation study. Multivariate Behavioral Research, 51(5), 661 680. https://doi.org/10.1080/00273171.2016.1208074
• Jin, S., Luo, H., & Yang-Wallentin, F. (2016). A simulation study of polychoric instrumental variable estimation in structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 23(5), 680 694. https://doi.org/10.1080/10705511.2016.1189334
• Lee, T. K., Wickrama, K., & O’Neal, C. W. (2018). Application of latent growth curve analysis with categorical responses in social behavioral research. Structural Equation Modeling: A Multidisciplinary Journal, 25(2), 294 306. https://doi.org/10.1080/10705511.2017.1375858
• Lee, S.-Y., & Song, X.-Y. (2004). Evaluation of the bayesian and maximum likelihood approaches in analyzing structural equation models with small sample sizes. Multivariate Behavioral Research, 39(4), 653–686. https://doi.org/10.1207/s15327906mbr3904_4
• Lei, P. (2009). Evaluating estimation methods for ordinal data in structural equation modeling. Quality and Quantity, 43(3), 495–507. https://doi.org/10.1007/s11135-007-9133-z
• Li, C.-H. (2014). The performance of MLR, USLMV, and WLSMV estimation in structural regression models with ordinal variables [Unpublished Doctoral dissertation]. Michigan State University.
• Li, C.-H. (2016). Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behavior Research Methods, 48(3), 936–949. https://doi.org/10.3758/s13428-015-0619-7
• Liang, X., & Yang, Y. (2014). An evaluation of WLSMV and Bayesian methods for confirmatory factor analysis with categorical indicators. International Journal of Quantitative Research in Education, 2(1), 17 38. https://doi.org/10.1504/IJQRE.2014.060972
• Liang, X., & Yang, Y. (2016). Confirmatory factor analysis under violations of distributional and structural assumptions: A comparison of robust maximum likelihood and bayesian estimation methods. Journal of Psychological Science, 39(5), 1256–1267. https://doi.org/10.1504/IJQRE.2013.055642
• Lorenzo-Seva, U., & Ferrando, P. J. (2020). Factor (Version 10.10.03) [Computer software]. Universitat Rovira i Virgili.
• MoNE. (2017). Monitoring and evaluation of academic skills report for eight graders. MONE. https://odsgm.meb.gov.tr/meb_iys_dosyalar/2017_11/30114819_iY-web-v6.pdf
• Morata-Ramirez, M. de los A., & Holgado-Tello, F. P. (2013). Construct validity of likert scales through confirmatory factor analysis: A simulation study comparing different methods of estimation based on Pearson and polychoric correlations. International Journal of Social Science Studies, 1(1), 54-61. https://doi.org/10.11114/ijsss.v1i1.27
• Moshagen, M., & Musch, J. (2014). Sample size requirements of the robust weighted least squares estimator. Methodology, 10(2), 60 70. https://doi.org/10.1027/1614-2241/a000068
• Muthén, B. O., & Kaplan, D. (1985). A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology, 38(2), 171–189. https://doi.org/10.1111/j.2044-8317.1985.tb00832.x
• Muthén, B. O., & Kaplan, D. (1992). A comparison of some methodologies for the factor analysis of non-normal Likert variables: A note on the size of the model. British Journal of Mathematical and Statistical Psychology, 45(1), 19–30. https://doi.org/10.1111/j.2044-8317.1992.tb00975.x
• Muthén, L. K., & Muthén, B. O. (2012). Mplus statistical modeling software: Release 7.0 [Computer software]. Muthén & Muthén.
• Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd. ed.). McGraw-Hill.
• Olsson, U. H., Foss, T., Troye, S. V., & Howell, R. D. (2000). The performance of ML, GLS, and WLS estimation in structural equation modeling under conditions of misspecification and nonnormality. Structural Equation Modeling: A Multidisciplinary Journal, 7(4), 557–595. https://doi.org/10.1207/S15328007SEM0704_3
• Oranje, A. (2003, April 21-25). Comparison of estimation methods in factor analysis with categorized variables: Applications to NEAP data [Paper presentation]. Annual Meeting of the National Council on Measurement in Education. Chicago, IL, USA.
• Osborne, J. W., & Banjanovic, E. S. (2016). Exploratory factor analysis with SAS®. SAS Intitute Inc.
• Potthast, M. J. (1993). Confirmatory factor analysis of ordered categorical variables with large models. British Journal of Mathematical and Statistical Psychology, 46(2), 273–286. https://doi.org/10.1111/j.2044-8317.1993.tb01016.x
• R Core Team. (2018). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing. https://www.r-project.org/.
• Revelle, W. (2019). psych: Procedures for psychological, psychometric, and personality research. https://cran.r-project.org/package=psych
• Rhemtulla, M., Brosseau-Liard, P. É., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354–373. https://doi.org/10.1037/a0029315
• Savalei, V., & Rhemtulla, M. (2013). The performance of robust test statistics with categorical data. British Journal of Mathematical and Statistical Psychology, 66(2), 201–223. https://doi.org/10.1111/j.2044-8317.2012.02049.x
• Shi, D., DiStefano, C., McDaniel, H. L., & Jiang, Z. (2018). Examining chi-square test statistics under conditions of large model size and ordinal data. Structural Equation Modeling: A Multidisciplinary Journal, 25(6), 924 945. https://doi.org/10.1080/10705511.2018.1449653
• Thompson, B., & Daniel, L. G. (1996). Factor analytic evidence for the construct validity of scores: A historical overview and some guidelines. Educational and Psychological Measurement, 56(2), 197–208. https://doi.org/10.1177/0013164496056002001
• Wolf, E. J., Harrington, K. M., Clark, S. L., & Miller, M. W. (2013). Sample size requirements for structural equation models: An evaluation of power, bias, and solution propriety. National Institutes of Health, 76(6), 913–934. https://doi.org/10.1177/0013164413495237
• Yang-Wallentin, F., Jöreskog, K. G., & Luo, H. (2010). Confirmatory factor analysis of ordinal variables with misspecified models. Structural Equation Modeling: A Multidisciplinary Journal, 17(3), 392–423. https://doi.org/10.1080/10705511.2010.489003
• Zhao, Y. (2015). The performance of model fit measures by robust weighted least squares estimators in confirmatory factor analysis [Doctoral dissertation, The Pennsylvania State University]. https://etda.libraries.psu.edu/catalog/24901