Examining Cross-Cultural Applicability via Generalizability Theory

Year 2023, Volume: 10 Issue: 1, 178 - 189, 30.01.2023

Sümeyra Soysal

https://doi.org/10.17275/per.23.10.10.1

Abstract

Applying a measurement instrument developed in a specific country to other countries raise a critical and important question of interest in especially cross-cultural studies. Confirmatory factor analysis (CFA) is the most preferred and used method to examine the cross-cultural applicability of measurement tools. Although CFA is a sophisticated technique to investigate various equivalence types (structural, metric, scalar and alike.), it has some limitations. In light of the classical test theory, when a measurement tool is not invariant between countries, what factors contribute to the error variance become unclear. Also, CFA reveals little as to how dimensionality of the relevant measurement tool affects measurement invariance. Hence, a fundamental focus of this study is to examine the measurement comparability or cross-cultural applicability for different countries on an international assessment using generalizability theory (G-theory) in educational science studies. With multi-faceted design, the contribution of dimensionality to error variance is examined, as well. For illustration purposes, eight scales from PISA 2012 student questionnaire dataset related to attitudes towards mathematics are used. The study is based on data from Türkiye, Finland and USA. The unbalanced multi-faceted designs are performed using G String IV. In conclusion, almost all results supported all research expectations. From the estimations of the G-theory, it can be rightly deduced cross-nationally applicability of the attitudes towards mathematics scales from these research findings.

Keywords

cross-cultural applicability, generalizability;, multi-facet analysis, G-theory

References

• Block, R., & Norman, G. (2012). G String IV (Version 6.2.1.2). [Software]. Available from http://www.papaworx.com/
• Brennan, R. L. (2001a). Generalizability theory. Springer
• Brennan, R. L. (2001b). Manual for urGENOVA. Iowa City, IA: Iowa Testing Programs, University of Iowa.
• Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Wadsworth Thomson Learning.
• Cronbach, L.J., Rajarathnam, N. & Gleser, G.C. (1963). Theory of generalizability: a liberalization of reliability theory. British Journal of Statistical Psychology 16(2), 137–163.
• Çokluk, Ö., Şekercioğlu, G., & Büyüköztürk, Ş. (2010). Sosyal bilimler için çok değişkenli istatistik: SPSS ve LISREL uygulamaları [Multivariate statistics for social sciences: SPSS and LISREL applications]. Pegem Akademi
• Durvasula, S., Andrews, J. C., Lysonski, S., & Netemeyer, R. G. (1993). Assessing the cross-national applicability of consumer behavior models: A model of attitude toward advertising in general. Journal of Consumer Research, 19, 626–636.
• Durvasula, S., Netemeyer, R. G., Andrews, J. C., & Lysonski, S. (2006). Examining the cross-national applicability of multi-item, multi-dimensional measures using generalizability theory. Journal of International Business Studies, 37(4), 469–483.
• Durvasula, S. & Lysonski, S. (2016) Finding cross-national consistency: Use of G-theory to validate acculturation to global consumer culture measure. Journal of Global Marketing, 29(2), 57-70.
• Eisend, M. (2009). A cross-cultural generalizability study of consumers' acceptance of product placements in movies. Journal of Current Issues & Research in Advertising, 31(1), 15-25.
• Horn, J. L., & McArdle, J. J. (1992). A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research, 18, 117 – 144.
• Johansson, S. (2016). International large-scale assessments: what uses, what consequences?, Educational Research, 58(2), 139-148
• Malhotra, M. K., & Sharma, S. (2008). Measurement equivalence using generalizability theory: An examination of manufacturing flexibility dimensions. Decision Sciences, 39(4), 643–669.
• Matsumoto, D., & Van de Vijver, F. J. R. (Eds.). (2010). Cross-cultural research methods in psychology. Cambridge University Press.
• Muthen, L., & Muthen, M. (2012). Mplus Software (Version 7). [Software]. Available from https://www.statmodel.com/
• Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. McGraw-Hill.
• OECD (2014). PISA 2012 technical report. OECD Publishing.
• Rentz, J. O. (1987). Generalizability theory: A comprehensive method for assessing and improving the dependability of marketing measures. Journal of Marketing Research, 24(1), 19–28.
• Sharma, S., & Weathers, D. (2003). Assessing generalizability of scales used in cross-national research. International Journal of Research in Marketing, 20(3), 287–95.
• Shavelson, R.J. & Webb, N.M. (1991). Generalizability theory: A primer. Sage Publications Steenkamp, J. E. M., & Baumgartner, H. (1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25(1), 78–90.
• Van de Vijver, F.J.R. & Leung, K. (1997). Methods and data analysis of comparative research. In J.W. Berry, Y.P. Poortinga and J. Pandey (eds.), Handbook of Cross-Cultural Psychology, Volume One: Theory and Method (pp. 247–300). Allyn & Bacon.
• Van de Vijver, F.J.R. & Leung, K. (2011). Equivalence and bias: A review of concepts, models, and data analytic procedures. In D. Matsumo and F. J. R. Van de Vijver (eds.), Cross-cultural research methods in psychology (pp. 17-45), Cambridge University Press.
• Van de Vijver, F. J. R., Leung, K., Fetvadjiev, V. H., He, J. & Fontaine, J. R. (2021). Methods and data analysis for cross-cultural research (Second edition). Cambridge University Press.