Comparison of Item Difficulty Analyses of Exams Used in Teaching Turkish as a Foreign Language with Instructors’ Perceptions of Item Difficulty

Year 2024, Volume: 14 Issue: 3, 473 - 495

Funda Keskin , Seçil Alaca

Abstract

In numerous studies focusing on assessment and evaluation of teaching Turkish as a foreign language, researchers have frequently identified issues related to the standardization and low validity and reliability of exams. Addressing these issues and investigating the underlying causes is paramount. Given the development of assessment tools by Turkish language teaching centers are typically the responsibility of instructors, it is essential to understand their perspectives regarding these tools. This study aimed to evaluate the perceptions of instructors concerning item difficulty in the context of teaching Turkish as a foreign language. Initially, item analyses were conducted on reading tests included in assessment tools designed by a Turkish language teaching center for B1, B2, and C1 proficiency levels. Instructors from various Turkish language teaching centers were asked to evaluate item difficulty through a prepared questionnaire. Data regarding instructors educational backgrounds, experiences, and involvement in exam creation were collected. Various analytical methods were employed to examine and interpret the obtained data. Item analysis results of examined tests were compared with instructors' perceptions of difficulty using fit analysis. Accuracy of instructors' item difficulty estimates was calculated for each instructor using Error Matrix, and success rates determined. To identify the effects of instructors' characteristics on item difficulty estimation, t-test and ANOVA analyses were performed. These analyses results were interpreted alongside item analyses, and recommendations provided to enhance the assessment and evaluation literacy of instructors teaching Turkish as a foreign language.

Keywords

Turkish as a foreign language, item analysis, item diffuculty perception, assesment in teaching Turkish as a foreign language

References

• Ahmadi, M. R. S., & Ketabi, S. (2020). Features of language assessment literacy in Iranian English language teachers’ perceptions and practices. Journal of Teaching Language Skills, 38(1), 191-223. https://doi.org/10.22099/jtls.2020.34843.2739
• Alderson, C. (1993). Judgments in language testing. In D. Douglas & C. Chapelle (Eds.), A new decade in language testing (pp. 46-57). TESOL.
• Alderson, J. C., Clapham, C., & Wall, D. (1995). Language test construction and evaluation. Cambridge University Press.
• Altıntaş, N. (2022). Yabancı dil olarak Türkçe öğreten öğretim elemanlarının sınav hazırlama ve ölçme değerlendirme yeterlik algıları [Perceptions of Exam Preparation and Assessment Competencies Among Instructors Teaching Turkish as a Foreign Language] (Thesis Number: 747926) [Master Thesis, Dokuz Eylül University]. Turkish Council of Higher Education Thesis Center.
• Aydemir, P. D. Y., & Çiftçi, Y. Ö. (2008). A study on the questioning skills of literature teacher candidates. Van Yüzüncü Yıl University Journal of Education Faculty, 5(2), 103-115. Retrieved from https://dergipark.org.tr/en/pub/yyuefd/issue/13714/166035
• Bachman, L. F. (2002). Some reflections on task-based language performance assessment. Language Testing, 19(4), 453-476. https://doi.org/10.1191/0265532202lt240oa
• Bahtiar, I., & Purnawarman, P. (2020). Investigating English teachers' comprehension in language assessment literacy (LAL). Advances in Social Science, Education and Humanities Research, 508, 303-310. https://doi.org/10.2991/assehr.k.201214.253
• Bejar, I. I. (1983). Subject matter experts' assessment of item statistics. Applied Psychological Measurement, 7(3), 303-310. https://doi.org/10.1177/014662168300700305
• Bøhn, H., & Tsagari, D. (2021). Teacher educators’ conceptions of language assessment literacy in Norway. Journal of Language Teaching and Research, 12(2), 222-233. https://doi.org/10.17507/jltr.1202.02
• Boylu, E. (2019). Yabancılara Türkçe öğretiminde ölçme değerlendirme uygulamaları ve standart oluşturma. [Measurement and evaluation practices and standardization in teaching Turkish to foreigners] (Thesis Number: 542435) [Doctoral Dissertation, Çanakkale Onsekiz Mart University]. Turkish Council of Higher Education Thesis Center.
• Çavuşoğlu, R., & Işık, A. D. (2021). Assessment and evaluation process of Turkish language teaching centers (TÖMER). The Journal of Limitless Education and Research, 6(2), 291-315. https://doi.org/10.29250/sead.958711
• Cohen, L., Manion, L., & Morrison, K. (2000). Research methods in education (5th ed.). Routledge.
• Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Cengage Learning.
• Erdoğdu, M. Y., & Kurt, F. (2012). Investigation of teachers' perception of their competencies in assessment and evaluation in terms of certain variables. Electronic Journal of Education Sciences, 1(2), 23-36. Retrieved from https://dergipark.org.tr/en/pub/ejedus/issue/15938/167586
• Firoozi, T., Razavipour, K., & Ahmadi, A. (2019). The language assessment literacy needs of Iranian EFL teachers with a focus on reformed assessment policies. Language Testing in Asia, 9(1), 1-12. https://doi.org/10.1186/s40468-019-0078-7
• Fergadiotis, G., Swiderski, A., & Hula, W. D. (2019). Predicting confrontation naming item difficulty. Aphasiology, 33(6), 689–709. https://doi.org/10.1080/02687038.2018.1495310
• Fitriyah, I., Massitoh, F., & Widiati, U. (2022). Classroom-based language assessment literacy and professional development need between novice and experienced EFL teachers. Indonesian Journal of Applied Linguistics, 12(1), 124-134. https://doi.org/10.17509/ijal.v12i1.46539
• Freedle, R., & Kostin, I. (1999). Does the text matter in a multiple-choice test of comprehension? The case for construct validity of the TOEFL's minitalks. Language Testing, 16(1), 2-32. https://doi.org/10.1177/026553229901600101
• Gedik, E. (2017). Yabancı dil olarak Türkçe öğretiminde ölçme ve değerlendirme [Assessment and Evaluation in Teaching Turkish as a Foreign Language] (Thesis Number: 458489) [Master Thesis, Istanbul Arel University]. Turkish Council of Higher Education Thesis Center.
• Hambleton, R. K., & Jirka, S. J. (2006). Anchor-based methods for judgmentally estimating item statistics. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 399-420). Lawrence Erlbaum Associates.
• Hatipoğlu, Ç. (2015). Language assessment in English teacher education programs in Turkey. Journal of Language Testing, 32(2), 243-265. https://doi.org/10.1177/0265532214565635
• Işıkoğlu, M. (2015). Yabancı dil olarak Türkçe öğretiminde kullanılan yeterlik sınavlarının madde yazımı bakımından incelenmesi (Mersin ve Sakarya üniversiteleri örneği) [Analysis of proficiency exams developed for teaching Turkish as a foreign language in terms of item writing: Samples of Mersin and Sakarya Universities] (Thesis Number: 394791) [Doctoral Dissertation, Gazi University]. Turkish Council of Higher Education Thesis Center.
• Karagöl, E. (2020). Proficiency exams in teaching Turkish as a foreign language in TÖMER (Turkish and foreign languages research and application centers). Journal of Language and Linguistic Studies, 16(2), 930-947. https://doi.org/10.17263/jlls.759347
• Kostin, I. (2004). Exploring item characteristics that are related to the difficulty of TOEFL dialogue items. ETS Research Report Series, 2004(1), i-71. https://doi.org/10.1002/j.2333-8504.2004.tb01938.x
• Kutlu, Ö., Doğan, C. D., & Karakaya, İ. (2010). Öğrenci başarısının belirlenmesi: Performansa ve portfolyoya dayalı durum belirleme [Assessing Student Achievement: Performance and Portfolio-Based Evaluation] (3rd ed.). Pegem Akademi.
• Latif, M. W. (2021). Exploring tertiary the EFL practitioners' knowledge base component of assessment literacy: Implications for teacher professional development. Language Testing in Asia, 11(1), 1-22. https://doi.org/10.1186/s40468-021-00130-9
• Levi, T., & Inbar, O. (2019). Assessment literacy or language assessment literacy: Learning from the teachers. Language Assessment Quarterly, 17(3), 1-15. https://doi.org/10.1080/15434303.2019.1692347
• Liu, J., & Li, X. (2020). Assessing young English learners: Language assessment literacy of Chinese primary school English teachers. International Journal of TESOL Studies, 2, 36-49. https://doi.org/10.46451/ijts.2020.12.05
• Mercan, Ö., & Göktaş, B. (2023). Türkçenin yabancı dil olarak öğretimi sertifika programları ile CELTA’nın karşılaştırılması: Bir program önerisi [A Comparison of Turkish as a Foreign Language Certification Programs and CELTA: A Program Proposal]. Cumhuriyet International Journal of Education, 12(3), 715-731. https://doi.org/10.30703/cije.1258699
• Mede, E., & Atay, D. (2017). English language teachers’ assessment literacy: The Turkish context. Dil Dergisi, 168(1), 1-5. Retrieved from https://www.ceeol.com/search/article-detail?id=548906
• Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded source book (2nd ed.). Sage Publications.
• Nichols, T. R., Wisner, P. M., Cripe, G., & Gulabchand, L. (2010). Putting the kappa statistic to use. The Quality Assurance Journal, 13(3-4), 57-61. https://doi.org/10.1002/qaj.481
• Oktay, M. R. (2015). An analysis of sub-questions in Turkish textbooks used in teaching Turkish as a foreign language in terms of cognitive levels in Bloom's Taxonomy (Thesis Number: 463400) [Master Thesis, Başkent University]. Turkish Council of Higher Education Thesis Center.
• Ölmezer Öztürk, E., & Aydın, B. (2018). Developing and validating language assessment knowledge scale (LAKS) and exploring the assessment knowledge of EFL teachers. Hacettepe University Journal of Education, 34(3), 602-620. https://doi.org/10.16986/HUJE.2018043465
• Özcan, S., & Akçan, K. (2010). Investigation of questions prepared by science teacher candidates in terms of content and Bloom's Taxonomy. Kastamonu Journal of Education, 18(1), 323-330. Retrieved from https://dergipark.org.tr/en/pub/kefdergi/issue/49066/626064
• Özdemir, S. (2023). Yabancı dil olarak Türkçe öğretiminde ölçme ve değerlendirmeye yönelik araştırmaların eğilimleri [Trends in Research on Assessment and Evaluation in Teaching Turkish as a Foreign Language]. İnönü Üniversitesi Eğitim Fakültesi Dergisi, 24(1), 537-559. https://doi.org/10.17679/inuefd.1071671
• Özdemir, S., & Eke, H. (2023). Yabancılara Türkçe öğretiminde uygulanan A1 ve A2 kur sınavlarının madde yazma ilkeleri açısından incelenmesi [An Examination of A1 and A2 Level Exams in Teaching Turkish to Foreigners in Terms of Item Writing Principles]. Trakya Eğitim Dergisi, 13(1), 365-380. https://doi.org/10.24315/tred.700445
• Razavipour, K., & Rezagah, K. (2018). Language assessment in the new English curriculum in Iran: Managerial, institutional, and professional barriers. Language Testing in Asia, 8(1), 1-12. https://doi.org/10.1186/s40468-018-0061-8
• Rupp, A. A., Garcia, P., & Jamieson, J. (2001). Combining multiple regression and CART to understand item difficulty in second language reading and listening comprehension tests. International Journal of Testing, 1(3-4), 185-216. https://doi.org/10.1080/15305058.2001.9669479
• Sertdemir, E. (2021). An analysis of Turkish proficiency exams used in teaching Turkish as a foreign language according to the Revised Bloom's Taxonomy (Thesis Number: 652111) [Master Thesis, Çanakkale Onsekiz Mart University]. Turkish Council of Higher Education Thesis Center.
• Shohamy, E. (1984). Does the testing method make a difference? The case of reading comprehension. Language Testing, 1(2), 147-170. https://doi.org/10.1177/026553228400100203
• Şimşek, A. (2016). A comparative analysis of common mistakes in achievement tests prepared by school teachers and corporate trainers. European Journal of Science and Mathematics Education, 4(4), 477-489. https://doi.org/10.30964/ejsme.v4i4.467
• Tao, N. (2014). Development and validation of classroom assessment literacy scales: English as a foreign language (EFL) teachers in a Cambodian higher education setting (Unpublished doctoral dissertation). Victoria University, Australia. Retrieved from https://vuir.vu.edu.au/25850/
• Ustabulut, M. Y. (2021). Examination of education program literacy levels of instructors teaching Turkish as a foreign language. Fırat University Journal of Social Sciences, 31(3), 1235-1243. https://doi.org/10.18069/firatsbed.941957
• Witten, I. H., Frank, E., Hall, M. A., & Pal, C. J. (2005). Practical machine learning tools and techniques. In Data mining (2nd ed., Vol. 2, pp. 403-413). Elsevier.
• Yıldız, Ü., & Tepeli, Y. (2014). A study on teacher qualifications in teaching Turkish as a foreign language. International Journal of Language Academy, 2(4), 564-578. https://doi.org/10.18033/ijla.182