Investigating the sources of differential item functioning: A sample critical thinking motivation scale

Year 2023, Volume: 10 Issue: 3, 434 - 453, 22.09.2023

Fatma Betül Kurnaz , Hüseyin Yıldız

https://doi.org/10.21449/ijate.1279152

Abstract

Investigating the existence of items with differential item functioning (DIF) may provide more accurate comparisons of group differences in studies that aim to compare scores obtained in a test by groups with different characteristics. In the present study, a scale measuring critical thinking motivation that was adapted to the Turkish culture was applied to 817 participants, who were high school graduates, university students, and university graduates. The aim of the study was to examine whether the data collected from these participants had DIF or not. Hence, DIF analysis of the collected data was performed via the "lordif" function in the R "lordif" package. DIF was found to occur in twelve items, three of which were related to gender and nine to level of education. While it was revealed that the content of the items was the source of gender related DIF, the source of DIF related to level of education was found to be the language and expression of the items.

Keywords

Differential item functioning, Critical thinking, Critical thinking motivation

References

• Altıntaş, Ö., & Kutlu, Ö. (2019). Investigating differential item functioning of Ankara University examination for foreign students by recursive partitioning analysis in the Rasch model. International Journal of Assessment Tools in Education, 6(4), 602–616. https://doi.org/10.21449/ijate.554212
• Ateşok Deveci, N. (2008). Examination of Inter-university Board foreign language test in the frame of item bias. [Doctoral dissertation, Ankara University]. https://tez.yok.gov.tr
• Athman Ernst, J., & Monroe, M. (2004). The effects of environment‐based education on students' critical thinking skills and disposition toward critical thinking. Environmental Education Research, 10(4), 507-522. https://doi.org/10.1080/1350462042000291038
• Bar-Tal, D. (1978). Attributional analysis of achievement-related behavior. Review of Educational Research, 48(2), 259–271. https://doi.org/10.3102/00346543048002259
• Baron, J. (1985). Rationality and intelligence. Cambridge University Press.
• Büyüköztürk, Ş. (2021). Sosyal bilimler için veri analizi el kitabı [Data analysis handbook for social sciences]. PegemA.
• Byrne, B.M., Oakland, T., Leong, F.T.L., Van De Vijver, F.J.R., Hambleton, R.K., Cheung, F. M., & Bartram, D. (2009). A critical analysis of cross-cultural research and testing practices: Implications for improved education and training in psychology. Training and Education in Professional Psychology, 3, 94-105. https://doi.org/10.1037/a0014516
• Choi, S.W., Gibbons, L.E., & Crane, P.K. (2011). Lordif: An R package for detecting differential item functioning using iterative hybrid ordinal logistic regression/item response theory and Monte Carlo simulations. Journal of Statistical Software, 39(8), 1. https://doi.org/10.18637/jss.v039.i08
• Cole, D.A., Maxwell, S.E., Avery, R., & Salas, E. (1993). Multivariate group comparisons of variable systems: MANOVA and structural equation modeling. Psychological Bulletin, 114, 174-184. https://doi.org/10.1037/0033-2909.114.1.174
• Crandall, V.C., Katkovsky, W., & Crandall, V.J. (1965). Children’s belief in their own control of reinforcement in intellectual-academic achievement situations. Child Development, 36, 91–109. https://doi.org/10.2307/1126783
• Crane, P.K., Gibbons, L.E., Jolley, L., & Van Belle, G. (2006). Differential item functioning analysis with ordinal logistic regression techniques: DIF detect and difwithpar. Medical Care, 44(11), 115-123. https://www.jstor.org/stable/41219511
• Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Thomson Learning.
• Davidov, E., Meuleman, B., Cieciuch, J., Schmidt, P., & Billiet, J. (2014). Measurement equivalence in cross-national research. Annual Review of Sociology, 40, 55-75. https://doi.org/10.1146/annurev-soc-071913-043137
• Dewey, J. (1930). Human nature and conduct. The Modern Library.
• do Nascimento, C.D., Peter, W.F., Ribeiro, I.M., Moreira, B.S., Lima, V.P., Kirkwood, R.N., & Bastone, A.C. (2021). Cross-cultural validity of the animated activity questionnaire for patients with hip and knee osteoarthritis: A comparison between the Netherlands and Brazil, Brazilian Journal of Physical Therapy, 000(xxxx), 1-8. http://doi.org/10.1016/j.bjpt.2021.06.002
• Dönmez, B., & Kaya, F. (2016). Eleştirel Düşünme Motivasyonu Ölçeği’nin Türkçe’ye uyarlanması [Turkish adaptation study of Critical Thinking Motivational Scale]. HAYEF Journal of Education, 13 2(25), 159 173. https://dergipark.org.tr/tr/pub/iuhayefd/issue/24491/259590
• Eccles, J.S., Adler, T.F., Futterman, R., Goff, S.B., Kaczala, C.M., & Meece, J.L. (1983). Expectancies, values and academic behaviors. In J. T. Spence (Ed.), Achievement and Achievement Motives (pp. 75–146). San Francisco Freeman.
• Ennis, R.H. (1991). Critical thinking: A streamlined conception. Teaching Philosophy, 14, 5- 25. http://doi.org/10.1057/9781137378057_2
• Ennis, R.H. (1993). Critical thinking assessment. Theory into Practice, 32, 179-186. https://doi.org/10.1080/00405849309543594
• Ennis, R.H. (1996). Critical thinking dispositions: Their nature and assessability. Informal Logic, 18(2), 165-182. https://doi.org/10.22329/il.v18i2.2378
• Ersözlü, A. N., & Arslan, M. (2009). The effect of developing reflective thinking on metacognitional awareness at primary education level in Turkey. Reflective Practice, 10, 683–695. https://doi.org/10.1080/14623940903290752
• Facione, P.A., & Facione, N.C. (1992). The California critical thinking dispositions inventory. California Academic Press.
• Farmer, H.S., & Vispoel, W.P. (1990). Attributions of female and male adolescents for real-life failure experiences. Journal of Experimental Education, 58(2), 127–140. https://doi.org/10.1080/00220973.1990.10806529
• Feingold, A. (1994). Gender differences in personality: A meta analysis. Psychological Bulletin, 116, 429-456. https://doi.org/10.1037/0033-2909.116.3.429
• Ferne, T., & Rupp, A. (2007). A synthesis of 15 years of research on DIF in language testing: Methodological advances, challenges, and recommendations. Language Assessment Quarterly, 4, 113-148. https://doi.org/10.1080/15434300701375923
• French, B.F., Hand, B., Nam, J., Yen, H.J., & Vazquez, J.A.V. (2014). Detection of differential item functioning in the Cornell Critical Thinking Test across Korean and North American students. Psychological Test and Assessment Modeling, 56(3), 275.
• French, B.F., Hand, B., Therrien, W.J., & Vazquez, J.A.V. (2012). Detection of sex differential item functioning in the Cornell Critical Thinking Test. Europen Journal of Psychological Assessment, 28(3), 201-207. http://doi.org/10.1027/1015-5759/a000127
• Frieze, I.H. (1975). Women’s expectations for and causal attributions of success and failure. In T. Mednick, S. Tangi, & L.W. Hoffman (Eds.), Women and achievement. Social and motivational analysis. (pp. 158–171). John Wiley and Sons.
• Galic, Z., Scherer, K.T., & Leberton, J.M. (2014). Examining the measurement equivalence of the conditional reasoning test for aggression across U.S. and Croatian samples. Psychological Test and Assessment Modeling, 56, 195 216. http://darhiv.ffzg.unizg.hr/id/eprint/5547
• Gallos, J.V. (1995). Gender and silence. Collage Teaching, 43(3), 101-105. http://doi.org/10.1080/87567555.1995.9925525
• Garcia, J.M., Gallagher, M.W., O’Bryant, S.E., & Medina, L.D. (2021). Differential item functioning of the Beck Anxiety Inventory in a rural, multi-ethnic cohort. Journal of Affective Disorders, 293, 36-42. http://doi.org/10.1016/j.jad.2021.06.005
• Garcia, T., & Pintrich, P.R. (1992). The effect of PBL curriculum on students’ motivation and self-regulation. Paper presented at The Biennial Conference of The European Association for Research on Learning and Instruction, Italy.
• Gierl, M., Khaliq, S.N., & Boughton, K. (1999). Gender differential item functioning in mathematics and science: Prevalence and policy implications. In Improving Large-Scale Assessment in Education Symposium at the Annual Meeting of the Canadian Society for the Study of Education, Canada.
• Gilligan, C. (1979). Woman’s place in man’s life cycle. Harvard Educational Review, 49, 431-446. https://doi.org/10.17763/haer.49.4.h136573541l3g463
• Glevey, K.E. (2006). Promoting thinking skills in education. London Review of Education, 4(3), 291-302. http://doi.org/10.1080/14748460601044005
• Glover, T.A., & Albers, C.A. (2007). Consideraitons for evaluating universal screening assessments. Journal of School Psychology, 45(2), 117 135. http://doi.org/10.1016/j.jsp.2006.05.005
• Gök, B., Kabasakal, K.A., & Kelecioğlu, H. (2014). PISA 2009 öğrenci anketi tutum maddelerinin kültüre göre değişen madde fonksiyonu açısından incelenmesi [Analysis of attitude items in PISA 2009 student questionnaire in terms of differential item functioning based on culture]. Journal of Measurement and Evaluation in Education and Psychology, 5(1), 72-87. https://doi.org/10.21031/epod.64124
• Halpern, D.F. (1998). Teaching critical thinking for transfer across domains. American Psychologist, 53, 449-455. https://doi.org/10.1037/0003-066X.53.4.449
• Hambelton, R., & Rogers, J. (1995). Item bias review. Practical Assessment, Research, and Evaluation, 4(6), https://doi.org/10.7275/jymp-md73
• Hambleton, R.K., & Rovinelli, R.J. (1986). Assessing the dimensionality of a set of test items. Applied Psychological Measurement, 10(3), 287 302. https://doi.org/10.1177/014662168601000307
• Hambleton, R.K., & Swaminathan, H. (1989). Item response theory: Principles and applications. Kluwer Nijhoff.
• Hambleton, R.K., Swaminathan, H., & Rogers, H.J. (1991). Fundamentals of item response theory. Sage.
• Ingle, C.O. (2007). Predictors of critical thinking ability among college students. [Doctoral dissertation]. Available from ProQuest Dissertations and Theses Database. (UMI No. 3263681).
• Jensen, A.R. (1980). Bias in mental testing. Free Press.
• Kalaycioğlu, D.B., & Kelecioğlu, H. (2011). Öğrenci Seçme Sınavı’nın madde yanlılığı açısından incelenmesi [Item Bias Analysis of the University Entrance Examination]. Education and Science, 36(161), 3-11.
• Karakaya, İ., & Kutlu, Ö. (2012). Seviye Belirleme Sınavındaki Türkçe alt testlerinin madde yanlılığının incelenmesi [An Investigation of Item Bias in Turkish Sub Tests in Level Determination Exam]. Education and Science, 37(165). http://egitimvebilim.ted.org.tr/index.php/EB/article/view/1342
• King, P.M., Wood, P.K., & Mines, R.A. (1990). Critical thinking among college and graduate students. The Review of Higher Education, 13, 167 186. http://doi.org/10.1353/rhe.1990.0026
• Kholberg, L. (1973). Continuities and discontinuities in childhood and adult moral development revisited. In P. B. Baltes & L. R. Goutlet (Eds.), Lifespan developmental psychology: Research and theory (pp. 179-204). Academic Press.
• Kloosterman, P. (2001). Attributions, performance following failure, and motivation in mathematics. In E. Fennema & G.C. Leder (Eds.), Mathematics and gender. Teachers College Press.
• Köse, İ.A. (2015). PISA 2009 öğrenci anketi alt ölçeklerinde (Q32-Q33) bulunan maddelerin değişen madde fonksiyonu açısından incelenmesi [Investigation of items in PISA 2009 student questionnaire subscales (Q32-Q33) in terms of differential item functioning]. Kastamonu Education Journal, 23(1), 227 240. https://dergipark.org.tr/en/pub/kefdergi/issue/22600/241461
• Kurnaz, F.B., & Kelecioğlu, H. (2008). Investigation of Peabody Picture Vocabulary Test from the point of item bias. World Applied Sciences Journal, 3(2), 231-239. https://www.academia.edu/7282678/Investigation_of_Peabody_Picture_Vocabulary_Test_from_the_point_of_item_bias_peabody_picture_vocabulary_test
• Kurnaz Adıbatmaz, F.B., & Yıldız, H. (2020). The Effects of distractors to differential item functioning in Peabody Picture Vocabulary Test. Journal of Theoretical Educational Science, 13(3), 530-547. https://dergipark.org.tr/tr/pub/akukeg/issue/54987/621581
• Lau, C., Chiesi, F., Saklofske, D.H., Yan, G., & Li, C. (2020). How essential is the essential resilience scale? Differential item functioning of Chinese and English versions and criterion validity. Personality and Individual Differences, 155, 109666. http://doi.org/10.1016/j.paid.2019.109666
• Lau, C., Swindall, T., Chiesi, F., Quilty, L.C., Chen, H.C., Chan, Y.C., ... & Torres-Marín, J. (2023). Cultural differences in how people deal with ridicule and laughter: Differential item functioning between the Taiwanese Chinese and Canadian English versions of the PhoPhiKat-45. European Journal of Investigation in Health, Psychology and Education, 13(2), 238-258. https://doi.org/10.3390/ejihpe13020019
• Maller, S.J. (2001). Differential item functioning in the WISC-III: Item parameters for boys and girls in the national standardization sample. Educational and Psychological Measurement, 61(5), 793-817. https://doi.org/10.1177/00131640121971527
• McLean, C.P., & Miller, N.A. (2010). Changes in critical thinking skills following a course on science and pseudoscience: A quasi-experimental study. Teaching of Psychology, 37, 85-90. https://doi.org/10.1080/00986281003626714
• Mcpeck, J.E. (1990). Teaching critical thinking. Routledge.
• Meece, J.L., Glienke, B. B., & Burg, S. (2006). Gender and motivation. Journal of School Psychology, 44(5), 351-373. https://doi.org/10.1016/j.jsp.2006.04.004
• Mertler, C.A., Vannatta, R.A., & LaVenia, K.N. (2005). Advanced and multivariate statistical methods: Practical application and interpretation. Pyrczak. https://doi.org/10.4324/9781003047223
• Millsap, R.E. (2012). Statistical approaches to measurement invariance. Routledge.
• Muraki, E. (1992). A generalized partial credıt model: Applicatıon of an em algorithm. ETS Research Report Series. https://doi.org/10.1002/j.2333-8504.1992.tb01436.x
• Nielsen, T., & Dammeyer, J. (2019). Measuring higher education students’ perceived stress: An IRT-based construct validity study of the PSS-10. Studies in Educational Evaluation, 63, 17-25. http://doi.org/10.1016/j.stueduc.2019.06.007
• Osterlind, S.J. (1983). Test item bias. Sage.
• Parsons, J., Adler, T.F., & Kaczala, C.M. (1984). Socialization of achievement attitudes and beliefs: Parental influences. Child Development, 53, 322 339. https://doi.org/10.2307/1128973
• Paul, R.W. (1990). Critical thinking: What every person needs to survive in a rapidly changing world. Center for Critical Thinking and Moral Critique, Sonoma State University.
• Perkins, D.N., Jay, E., & Tishman, S. (1993). Beyond abilities: A dispositional theory of thinking. Merrill-Palmer Quarterly, 39(1), 1-21. https://www.jstor.org/stable/23087298
• Poortinga, Y.H. (1989). Equivalence of cross-cultural data: An overview of basic issues. International Journal of Psychology, 24(6), 737 756. https://doi.org/10.1080/00207598908247842
• Ruble, D., Greulich, F., Pomerantz, E.M., & Gochberg, B. (1993). The role of gender-related processes in the development of sex differences in self-evaluation and depression. Journal of Affective Disorders, 29, 97–128. https://doi.org/10.1016/0165-0327(93)90027-H
• Serin, Q., Serin, N.B., Saracaloğlu, A.S., & Ceylan, A. (2010). The examination of critical thinking styles of university students (TRNC Sample). Procedia Social and Behavioral Sciences, 9, 864–868. https://doi.org/10.1016/j.sbspro.2010.12.250
• Steinberg, L., & Thissen, D. (2006). Using effect sizes for research reporting: examples using item response theory to analyze differential item functioning. Psychological Methods, 11, 402-415. https://doi.org/10.1037/1082-989X.11.4.402
• Stump, T.E., Monahan, P., & Mchorney, C.A. (2005). Differential item functioning in the short portable mental status questionnaire. Research on Aging, 27(3), 355-384. https://doi.org/10.1177/0164027504273784
• Schwartz, S., & Meyer, I.H. (2010). Mental health disparities research: The impact of within and between group analyses on tests of social stress hypotheses. Social Science & Medicine, 70(8), 1111-1118. https://doi.org/10.1016/j.socscimed.2009.11.032
• Şengül Avşar, A., & Emons, W.H.M. (2021). A cross-cultural comparison of non-cognitive outputs towards science between Turkish and Dutch students taking into account detected person misfit. Studies in Educational Evaluation, 70(101053). http://doi.org/10.1016/j.stueduc.2021.101053
• Tabachnick, B.G., & Fidell, L.S. (2007). Using multivariate statistics. Pearson.
• Tedesco, J. (1991). Women’s ways of knowing/women’s ways of composing. Rhetoric Review, 9(2), 246-256. http://doi.org/10.1080/07350199109388931
• Tsui, L. (2000). Effects of campus culture on students’ critical thinking. The Review of Higher Education, 23(4), 421-441. http://doi.org/10.1353/rhe.2000.0020
• Turkish Language Assosiation (n.d.). Ability. In Updated Turkish Dictionary. Retrieved February 28, 2021. https://www.tdk.gov.tr/
• Usta, H.G. (2020). Sınav kaygı ölçeği maddelerinin çeşitli yöntemlere göre değişen madde fonksiyonlarının incelenmesi [Analysis of Test Anxiety Scale items in terms of differential item functioning by different methods]. Cumhuriyet International Journal of Education, 9(4), 1225-1242. https://doi.org/10.30703/cije.703337
• Valenzuela, J., Nieto, A.M., & Saiz, C. (2011). Critical thinking motivational scale: A contribution to the study of relationship between critical thinking and motivation. Journal of Research in Educational Psychology, 9(2), 823 848. http://repositorio.ual.es/bitstream/handle/10835/819/Art_24_588.pdf?sequence=1
• Yıldırım, H., & Büyüköztürk, Ş. (2018). Using the Delphi Technique and focus-group interviews to determine item bias on the Mathematics Section of the Level Determination Exam for 2012. Educational Sciences: Theory & Practice, 18(2), 447-470. http://doi.org/10.12738/estp.2018.2.0317