Application of the professional maturity scale as a computerized adaptive testing

Year 2023, Volume: 10 Issue: 3, 580 - 593, 22.09.2023

Süleyman Demir , Derya Çobanoğlu Aktan , Neşe Güler

https://doi.org/10.21449/ijate.1262199

Abstract

This study has two main purposes. Firstly, to compare the different item selection methods and stopping rules used in Computerized Adaptive Testing (CAT) applications with simulative data generated based on the item parameters of the Vocational Maturity Scale. Secondly, to test the validity of CAT application scores. For the first purpose, simulative data produced based on Vocational Maturity Scale item parameters were analyzed under different item selection methods (Maximum Fisher Information [MFI],Maximum Likelihood Weighted Information [MLWI] Maximum Posterior Weighted Information [MPWI] Maximum Expected Information [MEI] Minimum Expected Posterior Variance [MEPV] Maximum Expected Posterior Weighted Information [MEPWI]) and stopping rules (Standard Error [SE]<0.30, SE<0.50, SE <0.70, Number of Item [NI]=10, NI=20) by calculating the average number of items, standard error averages, correlation coefficients, bias, and RMSE statistics. For all the conditions of the item selection methods, standard error averages, correlation coefficients, bias, and RMSE statistics showed similar results. When the average number of items is considered, MFI and SE<0.30 were found as most appropriate methods to be used in CAT application. For the second purpose of the study, the paper-pencil form of the Vocational Maturity scale and CAT version were administered to 33 students. A moderate, positive, and statistically significant relationship was found between the CAT application scores and the paper-pencil form scores on the vocational maturity scale. As a result, it can be said that the vocational maturity scale can be applied as a computerized adaptive test and can be used in career guidance processes.

Keywords

Computerized Adaptive Test (CAT), CAT application, Item selection method, Stopping rule, Vocational maturity

Ethical Statement

Hacettepe University, 24.10.2017, 433-3695.

References

• AERA, APA, & NCME (2014). Standards for educational and psychological testing. American Educational Research Association.
• Akdaş, G., & Ekinci, M. (2016). Sağlık meslek lisesi öğrencilerinin mesleki olgunluk düzeylerinin ve algıladıkları aile desteğinin incelenmesi [Analysis of vocational school of health students’ professional maturity and family support perception levels]. Uluslararası Hakemli Psikiyatri ve Psikoloji Araştırmaları Dergisi 7, 83 100. https://doi.org/10.17360/UHPPD.2016723147
• Akıntuğ, Y., & Birol, C. (2011). Lise öğrencilerinin mesleki olgunluk ve karar verme stratejilerine yönelik karşılaştırmalı analiz [Comparative analysis of vocational maturity and decision making strategies of high school students]. Hacettepe Üniversitesi Eğitim Fakültesi Dergisi, 41, 1 12. http://efdergi.hacettepe.edu.tr/yonetim/icerik/makaleler/675 published.pdf
• Aybek, E.C., & Demirtaşlı, R.N. (2017). Computerized adaptive test (CAT) applications and item response theory models for polytomous items. International Journal of Research in Education and Science, 3(2), 475-487. https://doi.org/10.21890/ijres.327907
• Aybek, E.C., & Çıkrıkçı, R.N. (2018). Kendini Değerlendirme Envanteri’nin bilgisayar ortamında bireye uyarlanmış test olarak uygulanabilirliği [Applicability of the Self-Assessment Inventory as a computerized adaptive test]. Turkish Psychological Counseling and Guidance Journal, 8(50), 117 141. https://dergipark.org.tr/tr/pub/tpdrd/issue/40299/481364
• Bardhoshi G., & Erford B.T. (2017). Processes and procedures for estimating score reliability and precision. Measurement and Evaluation in Counseling and Development, 50(4), 256-263. https://doi.org/10.1080/07481756.2017.1388680
• Cattell R.B. (1986). The psychometric properties of tests: Consistency, validity, and efficiency. In Cattell R.B., Johnson R.C. (Eds.), Functional psychological testing (pp. 54-78). Brunner/Mazel.
• Cattell R.B., Eber H.W., & Tatsuoka M.M. (1970). Handbook for the Sixteen Personality Factor Questionnaire (16PF). Institute for Personality and Ability Testing.
• Choi, S.W. (2009). Firestar: Computerized adaptive testing simulation program for polytomous item response theory models. Applied Psychological Measurement, 33(1), 644-645. https://doi.org/10.1177/0146621608329892
• Choi, S.W., & Swartz, R.J. (2009). Comparison of CAT item selection criteria for polytomous items. Applied Psychological Measurement, 33(1), 419 440. https://doi.org/10.1177/0146621608327801
• Choi, Y., & McClenen, C. (2020). Development of adaptive formative assessment system using computerized adaptive testing and dynamic bayesian networks. Applied Sciences, 10(22), 81-96. https://doi.org/10.3390/app10228196
• Davis, L.L. (2002). Strategies for controlling item exposure in computerized adaptive testing with polytomously scored items [Unpublished doctoral dissertation]. The University of Texas.
• Davis, L.L., & Dodd, B.G. (2008). Strategies for controlling item exposure in computerized adaptive testing with the partial credit model. Journal of Applied Measurement, 9(1), 1-17. https://pubmed.ncbi.nlm.nih.gov/18180546/
• Deyo, R.A., Diehr, P., & Patrick, D.L. (1991). Reproducibility and responsiveness of health status measures: Statistics and strategies for evaluation. Controlled Clinical Trials, 12(4), 142-158. https://doi.org/10.1016/S0197-2456(05)80019-4
• Dodd, B.G., De Ayala, R.J., & Koch, W.R. (1995). Computerized adaptive testing with polytomous items. Applied Psychological Measurement, 19(1), 5 22. https://doi.org/10.1177/014662169501900103
• Gardner, W., Shear, K., Kelleher, K.J., Pajer, K.A., Mammen, O., Buysse, D., & Frank, E., (2004). Computerized adaptive measurement of depression: A simulation study. BMC Psychiatry, 4(13).
• Gibbons C., Bower P., Lovell K., Valderas J., & Skevington S. (2016). Electronic quality of life assessment using computer-adaptive testing. Journal of Medical Internet Research, 18. https://doi.org/10.2196/jmir.6053
• Giordano, A., Testa, S., Bassi, M. et al. (2023). Applying multidimensional computerized adaptive testing to the MSQOL-54: a simulation study. Health Qual Life Outcomes 21, 61 https://doi.org/10.1186/s12955-023-02152-8
• Green, S.B. (1991). How many subjects does it take to do a regression analysis?. Multivariate Behavioral Research, 26, 499‐510.
• Hambleton, R.K., Swaminathan, H., & Rogers, H.J. (1991). Fundamentals of item response theory. Sage.
• Harris, R.J. (1985). A primer of multivariate statistics. Academic Press.
• Harrison, C., Loe, B.S., Lis, P., & Sidey-Gibbons, C. (2020). Maximizing the potential of patient-reported assessments by using the open-source concerto platform with computerized adaptive testing and machine learning. Journal of Medical Internet Research, 22(10), 1–8. https://doi.org/10.2196/20950
• Ho, T. (2010). A Comparison of item selection procedures using different ability estimation methods in computerized adaptive testing based on the Generalized Partial Credit Model, [Unpublished doctoral dissertation]. University of Texas.
• Kutlu, M. (2012). Anadolu ve genel lise öğrencilerinin çeşitli değişkenlere göre mesleki olgunluk düzeylerinin incelenmesi [An analysis of vocational maturity levels of anatolian and general high school students in terms of some variables]. İnönü Üniversitesi Eğitim Fakültesi Dergisi, 13(1), 23-41. https://dergipark.org.tr/tr/download/article-file/92233
• Kuzgun, Y., & Bacanlı, F. (2005). Mesleki Olgunluk Ölçeği el kitabı [Professional Maturity Scale handbook]. MEB Basımevi.
• Linacre, J.M. (2000). Computer-adaptive testing: A methodology whose time has come. In Chae, S., Kang, U., Jeon E., & Linacre J.M. (Eds.), Development of computerized middle school achievement test (in Korean). Komesa Press.
• Liu, K., Zhang, L., Tu, D., & Cai, Y. (2022). Developing an Item bank of computerized adaptive testing for eating disorders in Chinese University students. SAGE Open, 12(4). https://doi.org/10.1177/21582440221141273
• Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Addison-Wesley.
• Nunnally J., & Bernstein I.H. (1994). Psychometric theory. McGraw-Hill.
• Orhan, A.A., & Ültanır, E. (2014). Lise öğrencilerinin mesleki olgunluk düzeyleri ile karar verme düzeyleri [Vocational maturity level and decision making strategies of high school students]. Ufuk Üniversitesi Sosyal Bilimler Enstitüsü Dergisi, 3(5), 43 55. https://dergipark.org.tr/tr/download/article-file/1358749
• Passos, V., Berger, M.P.F., & Tan, F.E. (2007). Test design optimization in CAT early stage with the Nominal Response Model. Applied Psychological Measurement, 31(3), 213–232. https://doi.org/10.1177/01466216062915
• Penfield, R.D. (2006). Applied Bayesian item selection approaches to adaptive tests using polytomous items. Applied Measurement in Education, 19, 1 20. https://doi.org/10.1207/s15324818ame1901_1
• Petersen, M.A., Gamper, E.M., Costantini, A., Giesinger, J.M., Holzner, B., Johnson, C., Sztankay, M., Young, T., Groenvold, M. (2016). An emotional functioning item bank of 24 items for computerized adaptive testing (CAT) was established. Journal of Clinical Epidemiology, 70, 90–100. https://doi.org/10.1016/j.jclinepi.2015.09.002
• Reckase, M.D. (1989). Adaptive testing: The evolution of a good idea. Educational Measurement Issues and Practice, 8, 11 15. https://doi.org/10.1111/j.1745 3992.1989.tb00326.x
• Sahranç, Ü. (2000). Lise öğrencilerinin mesleki olgunluk düzeylerinin denetim odaklarına göre bazı değişkenler açısından incelenmesi [A Study on some variables affecting career maturity levels of high school students depending on their locus of control], [Unpublished master dissertation]. Hacettepe University.
• Smits, N., Cuijpers, P., & van Straten, A. (2011). Applying computerized adaptive testing to the CES D scale: A simulation study. Psychiatry Research, 188(1), 147 155. https://doi.org/10.1016/j.psychres.2010.12.001
• Stochl, J., Böhnke, J.R., Pickett, K.E., & Croudace, T.J. (2016). An evaluation of computerized adaptive testing for general psychological distress: Combining GHQ-12 and Affectometer-2 in an item bank for public mental health research. BMC Medical Research Methodology, 16(1), 1-15. https://doi.org/10.1186/s12874-016-0158-7
• Super, D.E. (1957). The psychology of careers. Harper.
• Sürücü, M. (2005). Lise öğrencilerinin mesleki olgunluk ve algıladıkları sosyal destek düzeylerinin incelenmesi [High school students' vocational maturity and perceived social support level], [Unpublished master dissertation]. Gazi University.
• Şahin, M.D., & Gelbal, S. (2020). Development of a multidimensional computerized adaptive test based on the bifactor model. International Journal of Assessment Tools in Education, 7(3), 323-342. https://doi.org/10.21449/ijate.707199
• Tabachnick, B.G., & Fidell, L.S. (1996). Using multivariate statistics. HarperCollins.
• Thissen, D., & Mislevy, R.J. (2000). Testing algorithms. In H. Wainer (Ed.). Computerized adaptive testing, (101-135). Lawrence Erlbaum Assc.
• Thompson, N.A., & Weiss, D.J. (2011). A framework for the development of computerized adaptive tests. Practical Assessment, Research and Evaluation, 16(1), 1 9. https://doi.org/10.7275/wqzt-9427
• Thorndike R.L., & Hagen, E. (1961). Measurement and evaluation in psychology and education. John Wiley and sons.
• Turgut, M.F., & Baykul, Y. (2013). Eğitimde ölçme ve değerlendirme [Measurement and evaluation in education]. PegemA Yayıncılık.
• Ulaş, Ö., & Yıldırım, İ. (2015). Lise öğrencilerinde mesleki olgunluğun yordayıcıları [Predictors of career maturity among high school students]. Hacettepe Üniversitesi Eğitim Fakültesi Dergisi, 30(2), 151 165. http://www.efdergi.hacettepe.edu.tr/yonetim/icerik/makaleler/14-published.pdf
• Ürün, A.E. (2010). Lise öğrencilerinin kendine saygı düzeyleri ile mesleki olgunlukları arasındaki ilişki [The relationship between the self-esteem level and the vocational maturity of high school students] [Unpublished master dissertation]. Balıkesir University.
• van der Linden, W.J. (1998). Bayesian item-selection criteria for adaptive testing. Psychometrika, 62, 201–216. https://link.springer.com/article/10.1007/BF02294775
• van der Linden, W.J., & Pashley, P.J. (2000). Item selection and ability estimation in adaptive testing. In W.J. van der Linden & C.A.W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 1–25). Kluwer.
• Van Rijn, P., Eggen, T.J., Hemker, B.T., & Sanders, P.F. (2002). Evaluation of selection procedures for computerized adaptive testing with polytomous items. Applied Psychological Measurement 26, 393- 411. https://doi.org/10.1177/014662102237796
• Veldkamp, B.P. (2003). Item selection in Polytomous CAT. In H., Yanai, A., Okada, K., Shigemasu, Y., Kano & J.J. Meulman (Eds.), New Developments in Psychometrics (pp. 207-214). Springer Verlag.
• Wilson, C.R., & Morgan, B.L. (2007). Understanding power and rules of thumb for determining sample sizes. Tutorials in Quantitative Methods for Psychology, 3(2), 43 50. https://doi.org/10.20982/tqmp.03.2.p043
• Wise S.L., & Kingsbury G.G. (2000). Practical issues in developing and maintaining a computerized adaptive testing program. Psicológica, 21, 135 155. https://www.uv.es/revispsi/articulos1y2.00/wise.pdf
• Yasuda, J., Mae, N., Hull, M.M., & Taniguchi, M., (2021). Optimizing the length of computerized adaptive testing for the force concept inventory. Physical Review Physics Education Research, 17(1), 1 15. https://doi.org/10.1103/PhysRevPhysEducRes.17.010115