Examining The Rater Drift in The Assessment of Presentation Skills in Secondary School Context

Year 2023, Volume: 14 Issue: 1, 62 - 75, 25.03.2023

Aslıhan Erman Aslanoğlu , Mehmet Şata

https://doi.org/10.21031/epod.1213969

Cited By: 1

Abstract

The alternative assessment, including peer assessment, helps students develop metacognition among the sub-categories of assessment types. Despite the advantage of alternative assessment, reliability and validity issues are the most significant problems in alternative assessment. This study investigated the rater drift, one of the rater effects, in peer assessment. The performance of 8 oral presentations based on group work in the Science and Technology course was scored by 7th-grade students (N=28) using the rubric researchers developed. The presentations lasted for four days, with two presentations each day. While examining the time-dependent drift in rater severity in peer assessment, the many-Facet Rasch Measurement model was used. Two indexes (interaction term and standardized differences) were calculated with many-facet Rasch measurement to determine the raters who made rater drift either individually or as a group. The analysis examined the variance of scores in the following days compared to the first day’s scores. Accordingly, the two methods used to determine rater drift gave similar results, and some raters at the individual level tended to be more severe or lenient over time. However, no significant rater drift at the group level showed that drifts had no specific models.

Keywords

Alternative assessment, many-facet Rasch measurement, peer assessment, validity, rater drift

References

• Andrade, H. G. (2005). Teaching with rubrics: The good, the bad, and the ugly. College Teaching, 53(1), 27-31. https://doi.org/10.3200/CTCH.53.1.27-31
• Azarnoosh, M. (2013). Peer assessment in an EFL context: attitudes and friendship bias. Language Testing in Asia, 3(1), 1-10. https://doi.org/10.1186/2229-0443-3-11
• Baird, J. A., Hayes, M., Johnson, R., Johnson, S., & Lamprianou, I. (2013). Marker effects and examination reliability. A Comparative exploration from the perspectives of generalisability theory, Rash model and multilevel modelling. Oxford University Centre for Educational Assessment. https://core.ac.uk/download/pdf/15171449.pdf
• Batmaz, H., Türk, N., Kaya, A., & Yıldırım, M. (2022). Cyberbullying and cyber victimization: examining mediating roles of empathy and resilience. Current Psychology, 1-11. https://doi.org/10.1007/s12144-022-04134-3
• Black, P., Harrison, C., & Lee, C. (2003). Assessment for learning: Putting it into practice. McGraw-Hill. http://www.mcgraw-hill.co.uk/html/0335212972.html
• Börkan, B. (2017). Rater severity drift in peer assessment. Journal of Measurement and Evaluation in Education and Psychology, 8(4), 469-489. https://doi.org/10.21031/epod.328119
• Bushell, G. (2006). Moderation of peer assessment in group projects. Assessment and Evaluation in Higher Education, 31(1), 91–108. https://doi.org/10.1080/02602930500262395
• Congdon, P. J., & McQueen, J. (2000). The stability of rater severity in large-scale assessment programs. Journal of Educational Measurement, 37(2), 163-178. https://psycnet.apa.org/doi/10.1111/j.1745-3984.2000.tb01081.x
• Dochy, F. (2001). A new assessment era: different needs, new challenges. Learning and Instruction, 10(1), 11-20. https://doi.org/10.1016/S0959-4752(00)00022-0
• Donnon, T., McIlwrick, J., & Woloschuk, W. (2013). Investigating the reliability and validity of self and peer assessment to measure medical students’ professional competencies. Creative Education, 4(6A), 23-28. http://dx.doi.org/10.4236/ce.2013.46A005
• Erman Aslanoğlu, A., Karakaya, İ., & Şata, M. (2020). Evaluation of university students’ rating behaviors in self and peer rating process via many facet rasch model. Eurasian Journal of Educational Research, 20(89), 25-46. https://dergipark.org.tr/en/pub/ejer/issue/57497/815802
• Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of Educational Research, 70(3), 287–322. https://doi.org/10.2307/1170785
• Farrokhi, F., Esfandiari, R. ve Dalili, M.V. (2011). Applying the Many-Facet Rasch Model to detect centrality in self-Assessment, peer-assessment and teacher assessment. World Applied Sciences Journal 15 (Innovation and Pedagogy for Lifelong Learning), 70-77. http://www.idosi.org/wasj/wasj15(IPLL)11/12.pdf
• Farrokhi, F., Esfandiari, R., & Schaefer, E. (2012). A many-facet Rasch measurement of differential rater severity/leniency in three types of assessment. JALT Journal, 34(1), 79-101.
• Hafner, J. C., & Hafner, P. M. (2003). Quantitative analysis of the rubric as an assessment tool: An empirical study of student peer-group rating. International Journal of Science Education, 25(12), 1509–1528. https://doi.org/10.1080/0950069022000038268
• Harik, P., Clauser, B. E., Grabovsky, I., Nungester, R. J., Swanson, D., & Nandakumar, R. 2009). An examination of rater drift within a generalizability theory framework. Journal of Educational Measurement, 46(1), 43-58. https://doi.org/10.1111/j.1745-3984.2009.01068.x
• Hauenstein, N. M. A. & McCusker, M. E. (2017). Rater training: Understanding effects of training content, practice ratings, and feedback. International Journal of Selection and Assessment, 25, 253-266. https://psycnet.apa.org/doi/10.1111/j.1745-3984.2009.01068.x
• Humphris GM, & Kaney S. (2001). Examiner fatigue in communication skills objective structured clinical examinations. Medical Education, 35(5), 444-449. https://doi.org/10.1046/j.1365-2923.2001.00893.x
• Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130-144. https://doi.org/10.1016/j.edurev.2007.05.002
• Kaya, A., Türk, N., Batmaz, H., & Griffiths, M. D. (2023). Online gaming addiction and basic psychological needs among adolescents: the mediating roles of meaning in life and responsibility. International Journal of Mental Health and Addiction, 1-25. https://doi.org/10.1007/s11469-022-00994-9
• Kassim, A.N.L. (2007, June 14-16). Exploring rater judging behaviour using the many-facet Rasch model [Conference Session]. The Second Biennial International Conference on Teaching and Learning of English in Asia: Exploring New Frontiers (TELiA2), Universiti Utara, Malaysia. http://repo.uum.edu.my/id/eprint/3212/
• Kim, Y., Park, I., & Kang, M. (2012). Examining rater effects of the TGMD-2 on children with intellectual disability. Adapted Physical Activity Quarterly, 29(4), 346-365. https://doi.org/10.1123/apaq.29.4.346
• Knoch, U., Read, J., & von Randow, J. (2007). Re-training writing raters online: How does it compare with face-to-face training?. Assessing Writing, 12(1), 26–43. https://doi.org/10.1016/j.asw.2007.04.001 Kubiszyn, T., & Borich, G. (2013). Educational testing and measurement: Classroom application and practice (10th ed.). John Wiley & Sons. https://l24.im/jV6yYCJ
• Kutlu, Ö., Doğan, C. D., & Karaya, İ. (2014). Öğrenci başarısının belirlenmesi: Performansa ve portfolyoya dayalı durum belirleme [Determining student success: Assessment based on performance and portfolio]. Pegem. https://l24.im/k5cn
• Leckie, G., & Baird, J. A. (2011). Rater effects on essay scoring: A multilevel analysis of severity drift, central tendency, and rater experience. Journal of Educational Measurement, 48(4), 399-418. https://doi.org/10.1111/j.1745-3984.2011.00152.x
• Linacre, J.M. (2017). A user's guide to FACETS: Rasch-model computer programs. MESA Press.
• Liu, H., & Brantmeier, C. (2019). “I know English”: Self-assessment of foreign language reading and writing abilities among young Chinese learners of English. System, 80, 60-72. https://doi.org/10.1016/j.system.2018.10.013
• Lunz, M. E., & Stahl, J. A. (1990). Judge consistency and severity across grading periods. Evaluation & the Health Professions, 13(4), 425-444. https://psycnet.apa.org/doi/10.1177/016327879001300405
• McLaughlin K, Ainslie M, Coderre S, Wright B & Violato C. (2009). The effect of differential rater function over time (DRIFT) on objective structured clinical examination ratings. Medical Education, 43(10), 989–992. https://doi.org/10.1111/j.1365-2923.2009.03438.x
• Messick, S. (1994). Alternative modes of assessment, uniform standards of validity. ETS Research Report Series, 2, 1-22. https://doi.org/10.1002/j.2333-8504.1994.tb01634.x
• Messick, S. (1996). Validity of performance assessments. In G. Phillips (Ed.), Technical issues in large-scale performance assessment (pp. 1–18). National Center for Education Statistics.
• Moore, B. B. (2009). A consideration of rater effects and rater design via signal detection theory (Publication No. 3373803). [Doctoral dissertation, Columbia University]. ProQuest Dissertations & Theses Global.
• Mulqueen C., Baker D. & Dismukes R.K. (2000, April). Using multifacet rasch analysis to examine the effectiveness of rater training [Conference Session]. 15th Annual Meeting of the Society for Industrial/Organizational Psychology, New Orleans, LA. https://www.air.org/sites/default/files/2021-06/multifacet_rasch_0.pdf
• Myford, C. M. (1991, April 3-7). Judging acting ability: The transition from notice to expert [Conference Session]. Annual Meetin of the American Educational Research Association, Chicago IL. https://files.eric.ed.gov/fulltext/ED333032.pdf
• Myford, C. M., & Wolfe, E. M. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Aplied Measurement, 4(4), 386-422. https://psycnet.apa.org/record/2003-09517-007
• Myford, C. M., & Wolfe, E. M. (2009). Monitoring rater performance over time: A framework for detecting differential accuracy and differential scale use. Journal of Educational Measurement, 46(4), 371-389. https://psycnet.apa.org/doi/10.1111/j.1745-3984.2009.00088.x
• Oosterhof, A. (2003). Developing and using classroom assessment (3th ed.). Merrill/Prentice Hall. https://l24.im/OCKvkg2
• Orsmond P, Merry S, Reiling K (2000) The use of student-derived marking criteria in peer and self-assessment. Assessment &Evaluation Higher Education, 25(1), 21–38. https://doi.org/10.1080/02602930050025006
• Patri, M. (2002). The influence of peer feedback on self- and peer-assessment of oral skills. Language Testing, 19(2), 109–131. https://doi.org/10.1191/0265532202lt224oa
• Pinot de Moira, A., Massey, C., Baird, J., & Morrissey, M. (2002). Marking consistency over time. Research in Education, 67(1), 79–87. https://doi.org/10.7227/RIE.67.8
• Raju, N. S. (1990). Determining the significance of estimated signed and unsigned areas between two item response functions. Applied Psychological Measurement, 14(2), 197-207. https://psycnet.apa.org/doi/10.1177/014662169001400208
• Swaminathan, H., & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361-370. https://psycnet.apa.org/doi/10.1111/j.1745-3984.1990.tb00754.x
• Şata. M. (2020a). Quantitative research approaches [Nicel araştırma yaklaşımları]. In Oğuz. E. (Ed.). Research methods in education [Eğitimde araştırma yöntemleri] (p. 77-98). Eğiten Kitap publishing.
• Topping, K. (2003). Self and peer assessment in school and university: Reliability, validity and utility. In M. S. Segers, Dochy,R., & E. C. Cascallar (Eds.), In optimising new modes of assessment: In search of qualities and standards (pp. 55-87). Springer Dordrecht. https://doi.org/10.1007/0-306-48125-1
• Topping, K. (2009). Peer assessment. Theory Into Practice, 48(1), 20-27. https://doi.org/10.1080/00405840802577569
• Topping, K. (2017). Peer assessment: learning by judging and discussing the work of other learners. Interdisciplinary Education and Psychology, 1(1), 1-17. https://doi.org/10.31532/InterdiscipEducPsychol.1.1.007
• Wolfe, E. W., Myford, C. M., Engelhard Jr. G., & Manalo, J. R. (2007). Monitoring reader performance and DRIFT in the AP® English literature and composition examination using benchmark essays. College Board. https://files.eric.ed.gov/fulltext/ED561038.pdf
• Yurdabakan, İ., & Cihanoğlu, M. O. (2009). The effects of cooperative reading and composition technique with the applications of self and peer assessment on the levels of achivement, attitude and strategy use. Dokuz Eylul University The Journal of Graduate School of Social Sciences, 11(4), 105-123. https://l24.im/VbHg