Which scale short form development method is better? A Comparison of ACO, TS, and SCOFA

Year 2022, Volume: 9 Issue: 3, 583 - 592, 30.09.2022

Hakan Koğar

https://doi.org/10.21449/ijate.946231

Cited By: 1

Abstract

The purpose of this study is to identify which scale short-form development method produces better findings in different factor structures. A simulation study was designed based on this purpose. Three different factor structures and three simulation conditions were selected. As the findings of this simulation study, the model-data fit and reliability coefficients were reported for each factor structure in each simulation condition. All analyses were conducted under the R environment. According to the findings of this study, the increase in the level of misspecification and the decrease in the sample size can significantly affect the model-data fit. In a situation where the factor structure of the scale is getting more and more complex, model-data fit and Omega coefficients decrease. For scales with a unidimensional factor structure, all of the scale short-form development methods are recommended. For scales with multidimensional factor structure, Ant Colony Optimization, and Stepwise Confirmatory Factor Analysis algorithms and for scales with bifactor factor structure, the ACO algorithm is recommended. When viewed from the framework of metaheuristic algorithms, it has been identified that ACO produces better findings than Tabu Search.

Keywords

short form, scale development, ant colony optimization, tabu search, stepwise confirmatory factor analysis

Thanks

Thank you to Dr. Holmes Finch for his support in writing R code.

References

• Anastasi, A. (1982). Psychological Testing (5th ed.). Macmillan.
• Batley, R.M., & Boss, M.W. (1993). The effects on parameter estimation of correlated dimensions and a distribution-restricted trait in a multidimensional item response model. Applied Psychological Measurement, 17(2), 131 141. https://doi.org/10.1177/014662169301700203
• Cayanus, J.L., & Martin, M.M. (2004). An instructor self‐disclosure scale. Communication Research Reports, 21(3), 252-263. https://doi.org/10.1080/08824090409359987
• Colorni, A., Dorigo, M., & Maniezzo, V. (1991). Distributed optimization by ant colonies. In: Varela, F. and Bourgine, P., Eds., Proceedings of the European Conference on Artificial Life, ECAL’91, Paris, Elsevier Publishing, Amsterdam, 134-142.
• Ebesutani, C., McLeish, A.C., Luberto, C.M., Young, J., & Maack, D.J. (2014). A bifactor model of anxiety sensitivity: Analysis of the Anxiety Sensitivity Index-3. Journal of Psychopathology and Behavioral Assessment, 36(3), 452 464. https://doi.org/10.1007/s10862-013-9400-3
• French, B.F., & Finch, W.H. (2011). Model misspecification and invariance testing using confirmatory factor analytic procedures. The Journal of Experimental Education, 79(4), 404-428. https://doi.org/10.1080/00220973.2010.517811
• Gatignon, H. (2010). Confirmatory Factor Analysis. In Statistical Analysis of Management Data (pp. 59-122). Springer. https://doi.org/10.1007/978-1-4419-1270-1_4
• Hu, L.T., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118
• Janssen, A.B., Schultze, M., & Grötsch, A. (2017). Following the ants: Development of short scales for proactive personality and supervisor support by Ant Colony Optimization. European Journal of Psychological Assessment, 33(6), 409. https://doi.org/10.1027/1015-5759/a000299
• Jiang, S., Wang, C., & Weiss, D.J. (2016). Sample size requirements for estimation of item parameters in the multidimensional graded response model. Frontiers in Psychology, 7(Article:109), 1-10. https://doi.org/10.3389/fpsyg.2016.00109
• Jorgensen, T.D., Pornprasertmanit, S., Schoemann, A. M., Rosseel, Y., Miller, P., Quick, C., ..., & Enders, C. (2016). semTools: Useful Tools for Structural Equation Modeling. R package version 0.5 4. Retrieved from https://cran.r project.org/web/packages/semTools/index.html
• Kleka, P., & Soroko, E. (2018). How to avoid the sins of questionnaires abridgement?. Survey Research Methods, 12(2), 147-160. https://doi.org/10.31234/osf.io/8jg9u
• Kruyen, P.M., Emons, W.H., & Sijtsma, K. (2013). On the shortcomings of shortened tests: A literature review. International Journal of Testing, 13(3), 223 248. https://doi.org/10.1080/15305058.2012.703734
• LaNoue, M., Harvey, A., Mautner, D., Ku, B., & Scott, K. (2015). Confirmatory factor analysis and invariance testing between Blacks and Whites of the Multidimensional Health Locus of Control scale. Health Psychology Open, 2(2), 1 16. https://doi.org/10.1177/2055102915615045
• Leite, W.L., Huang, I.-C., & Marcoulides, G. A. (2008). Item selection for the development of short forms of scales using an Ant Colony Optimization Algorithm. Multivariate Behavioral Research, 43, 411–431. https://doi.org/10.1080/00273170802285743
• Marcoulides, K.M., & Falk, C. (2018). Model specification searches in structural equation modeling with R. Structural Equation Modeling, 25(3), 484 491. https://doi.org/10.1080/10705511.2017.1409074
• Nunnally, J.C. (1978). Psychometric Theory (2nd ed.). McGraw-Hill.
• Olaru, G., Witthöft, M., & Wilhelm, O. (2015). Methods matter: Testing competing models for designing short-scale big-five assessments. Journal of Research in Personality, 59, 56-68. https://doi.org/10.1016/j.jrp.2015.09.001
• Raborn, A.W., & Leite, W.L. (2018). ShortForm: An R package to select scale short forms with the ant colony optimization algorithm. Applied psychological measurement, 42(6), 516. https://doi.org/10.1177/0146621617752993
• Raborn, A.W., Leite, W.L., & Marcoulides, K.M. (2020). A comparison of metaheuristic optimization algorithms for scale short-form development. Educational and Psychological Measurement, 80(5), 910 931. https://doi.org/10.1177/0013164420906600
• Reise, S.P. (2012). The rediscovery of bifactor measurement models. Multivariate Behav. Res. 47, 667–696. https://doi.org/10.1080/00273171.2012.715555
• Rosseel, Y. (2012). Lavaan: An R package for structural equation modeling and more. Version 0.5 12 (BETA). Journal of Statistical Software, 48(2), 1 36. https://doi.org/10.18637/jss.v048.i02
• Schroeders, U., Wilhelm, O., & Olaru, G. (2016). Meta-heuristics in short scale construction: Ant colony optimization and genetic algorithm. PLoS One, 11(11), 1-19. https://doi.org/10.1371/journal.pone.0167110
• Singh, K., Junnarkar, M., & Kaur, J. (2016). Measures of Positive Psychology: Development and Validation. Springer.
• Van Abswoude, A.A., van der Ark, L.A., & Sijtsma, K. (2004b). A comparative study of test data dimensionality assessment procedures under nonparametric IRT models. Applied Psychological Measurement, 28(1), 3-24. https://doi.org/10.1177/0146621603259277
• Van Abswoude, A.A., Vermunt, J.K., Hemker, B.T., & van der Ark, L.A. (2004a). Mokken scale analysis using hierarchical clustering procedures. Applied Psychological Measurement, 28(5), 332-354. https://doi.org/10.1177/0146621604265510
• Yang, Y., & Liang, X. (2013). Confirmatory factor analysis under violations of distributional and structural assumptions. International Journal of Quantitative Research in Education, 1(1), 61-84. https://doi.org/10.1504/ijqre.2013.055642