Analysis of Peer and Self-Assessments Using the Many-facet Rasch Measurement Model and Student Opinions

Year 2023, Volume: 14 Issue: 3, 266 - 286, 30.09.2023

Seda Demir

https://doi.org/10.21031/epod.1344196

Cited By: 1

Abstract

The aim of this study is to analyze the peer and self-assessments of higher education students' oral presentation skills with the many-facet Rasch measurement model and to determine students' opinions on peer and self-assessment. In the study, convergent parallel method, one of the mixed-method research approaches, was used. The study group consisted of 11 university students studying at a state university in the 2022-2023 academic year. The FACETS program was used to analyze the data. The three facets identified in the study were the assessee (11 students), the assessor (11 students) and the items (16 items). Therefore, 11 participants scored (peer and self-assessment) on a 16-item assessment form. In addition, students' opinions on peer and self-assessment were obtained through three open-ended interview questions prepared by the researcher. According to the results of the study, it was determined that there was a statistically significant difference between the students in terms of their oral presentation skills, between the assessors in terms of their strictness/generosity in scoring, and between the criteria (items) in terms of the level of difficulty in realization. In addition, the participant opinions obtained from each interview question were analyzed through themes and sub-themes formed according to the general thoughts on peer and self-assessment, experiences, and whether the participants considered themselves as a reliable rater or not. In terms of practice, it can be suggested to provide detailed and enlightening information to students before peer and/or self-assessment in the classroom environment, and to give quick feedback to those who have not done the assessment appropriately. In addition, the reasons for the biases identified in peer and self-assessments in the current study can be investigated in future studies.

Keywords

Peer assessment, Self-assessment, Many-facet Rasch measurement model, Oral presentation skills

References

• Akın, Ö. ve Baştürk, R. (2012). The evaluation of the basic skills in violin training by many facet Rasch model. Pamukkale University Journal of Education, 31(1), 175-187. https://dergipark.org.tr/en/pub/pauefd/issue/11112/132860
• Atılgan, H. (2005). Analysis of Special Ability sellection examination for music education department using many-facets Racsh measurement (İnönü University case). Eurasian Journal of Educational Measurement, 0(20), 62-73. https://web.s.ebscohost.com/ehost/pdfviewer/pdfviewer?vid=0&sid=1c78c0f3-93e0-43b5-9499-026878d2fcb1%40redis
• Baştürk, R. (2008). Applying the many facet Rasch model to evaluate PowerPoint presentation performance in higher education. Assessment and Evaluation in Higher Education, 33(4), 431-444. https://doi.org/10.1080/02602930701562775
• Baştürk, R. (2010). Bilimsel araştırma ödevlerinin çok yüzeyli Rasch ölçme modeli ile değerlendirilmesi. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 1(1), 51-57. https://dergipark.org.tr/en/pub/epod/issue/5808/77254
• Boud, D., & Falchikov, N. (1989). Quantitative studies of student self-assessment in higher education: a critical analysis of findings. Higher Education, 18(5), 529-549. https://doi.org/10.1007/BF00138746
• Brown, S. (2004). Assessment for learning. Learning and Teaching in Higher Education, 5(1), 81-89. https://eprints.glos.ac.uk/3607/
• Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65(3), 245-281. https://doi.org/10.3102/00346543065003245
• Cheong, C. M., Luo, N., Zhu, X., Lu, Q., & Wei, W. (2023). Self-assessment complements peer assessment for undergraduate students in an academic writing task. Assessment & Evaluation in Higher Education, 48(1), 135-148. https://doi.org/10.1080/02602938.2022.2069225
• Creswell, J. W. (2014). A concise introduction to mixed methods research. SAGE.
• Crisp, V., & Ward, C. (2008). The development of a formative scenario-based computer assisted assessment tool in psychology for teachers: The PePCAA project. Computers & Education, 50(4), 1509-1526. https://doi.org/10.1016/j.compedu.2007.02.004
• Eckes, T. (2005). Examining rater effects in TestDaF writing and speaking performance assessments: A many-facet Rasch analysis. Language Assessment Quarterly: An International Journal, 2(3), 197-221. https://doi.org/10.1207/s15434311laq0203_2
• Elhan, A. H., & Atakurt, Y. (2005). Why is it necessary to use Rasch analysis when evaluating measures? Ankara Üniversitesi Tıp Fakültesi Mecmuası, 58(1), 47-50. https://doi.org/10.1501/Tipfak_0000000134
• Evans, A. T., Mcnutt, R. A., Fletcher, S. W., & Fletcher, R. H. (1993). The characteristics of peer reviewers who produce good-quality reviews. Journal of General Internal Medicine, 8(8), 422-428. https://doi.org/10.1007/bf02599618
• Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of Educational Research, 70(3), 287-322. https://doi.org/10.3102/00346543070003287
• Fete, M. G., Haight, R. C., Clapp, P., & McCollum, M. (2017). Peer evaluation instrument development, administration, and assessment in a team-based learning curriculum. American Journal of Pharmaceutical Education, 81(4), 1-10. https://doi.org/10.5688/ajpe81468
• Gu, C. (2020). Student Peer Assessment. Review of Educational Theory, 3(2), 74-78. https://doi.org/10.30564/ret.v3i2.1762
• Güler, N. (2014). Analysis of open-ended statistics questions with many facet Rasch model. Eurasian Journal of Educational Research, 55, 73-90. http://dx.doi.org/10.14689/ejer.2014.55.5
• Gürlen, E. , Boztunç Öztürk, N. & Eminoğlu, E. (2019). Investigation of the reliability of teachers, self and peer assessments at primary school level with Generalizability Theory. Journal of Measurement and Evaluation in Education and Psychology, 10(4), 406-421. http://dx.doi.org/10.21031/epod.583891
• Hanrahan, S. J., & Isaacs, G. (2001). Assessing self- and peer-assessment: The students' views. Higher Education Research & Development, 20(1), 53-70. https://doi.org/10.1080/07294360123776
• Hoo, H. T., Deneen, C., & Boud, D. (2022). Developing student feedback literacy through self and peer assessment interventions. Assessment & Evaluation in Higher Education, 47(3), 444-457. https://doi.org/10.1080/02602938.2021.1925871
• Köse, İ. A., Usta, H. G., & Yandı A. (2016). Evaluation of presentation skills by using many facets Rasch model. Bolu Abant İzzet Baysal University Journal of Faculty of Education, 16(4), 1853-1864. https://dergipark.org.tr/en/pub/aibuefd/issue/28550/304600
• Linacre, J. M. (1993). Rasch-based generalizability theory. Rasch Measurement Transactions, 7(1), 283-284. https://www.rasch.org/rmt/rmt71h.htm
• Linacre, J. M. (1994). Many-facet rasch model (2nd ed.). Mesa Press.
• Linacre, J. M. (2023). A user’s guide to FACETS: Rasch-model computer programs (Program manual 3.85. 1). Chicago, IL. https://www.winsteps.com/a/Facets-Manual.pdf
• Liu, N. F., and D. Carless. 2006. “Peer feedback: The learning element of peer assessment.” Teaching in Higher Education 11(3), 279-290. https://doi.org/10.1080/13562510600680582
• Liu, E. Z.-F., Lin, S. S. J., Chiu, C.-H., & Yuan, S.-M. (2001). Web-based peer review: the learner as both adapter and reviewer. IEEE Transactions on Education, 44(3), 246-251. https://doi.org/10.1109/13.940995
• Lu, J., & Law, N. (2011). Online peer assessment: effects of cognitive and affective feedback. Instructional Science, 40(2), 257-275. https://doi.org/10.1007/s11251-011-9177-2
• Lundstrom, K., and W. Baker. 2009. “To give is better than to receive: The benefits of peer review to the reviewer’s own writing.” Journal of Second Language Writing 18(1), 30-43. https://doi.org/10.1016/j.jslw.2008.06.002
• Mulqueen C., Baker D. & Dismukes R.K. (2000) Using multifacet Rasch analysis to examine the effectiveness of rater training. SIOP.
• Mumpuni, K. E., Priyayi, D. F., & Widoretno, S. (2022). How do students perform a peer assessment? International Journal of Instruction, 15(3), 751-766. https://doi.org/10.29333/iji.2022.15341a
• Nakamura, Y. (2000). Many-facet Rasch based analysis of communicative language testing results. Journal of Communication Students, 12, 3-13. https://eric.ed.gov/?id=ED449678
• Nakamura, N. (2002). Teacher assessment and peer assessment in practice. Educational Studies, 44, 204-215. https://files.eric.ed.gov/fulltext/ED464483.pdf
• Nicol, D. J., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218. https://doi.org/10.1080/03075070600572090
• Orluwene, G. W., & Ekim, D. K. (2020). Promoting self-regulated learning through self-and peer-assessment techniques among secondary school students. International Journal of Arts and Commerce, 9(4), 1-16. https://ijac.org.uk/articles/9.4.1.1-16.pdf
• Panadero, E., Alqassab, M., Fernández Ruiz, J., & Ocampo, J. C. (2023). A systematic review on peer assessment: intrapersonal and interpersonal factors. Assessment & Evaluation in Higher Education, 1-23. https://doi.org/10.1080/02602938.2023.2164884
• Pantiwati, Y., & Husamah. (2017). Self and peer assessments in active learning model to increase metacognitive awareness and cognitive abilities. International Journal of Instruction, 10(4), 185-202. https://doi.org/10.12973/iji.2017.10411a
• Prieto, G., & Nieto, E. (2014). Analysis of rater severity on written expression exam using many faceted Rasch measurement. Psicologica, 35(2), 385-397. https://www.redalyc.org/pdf/169/16931314011.pdf
• Randall, J. & Engelhard, G. Jr. (2009). Examining teacher grades using Rasch measurement theory. Journal of Educational Measurement, 46(1), 1-18. https://doi.org/10.1111/j.1745-3984.2009.01066.x
• Sande, J. C. G., & Godino-Llorente, J. I. (2014). Peer assessment and self-assessment: Effective learning tools in higher education. International Journal of Engineering Education, 30(3), 711-721. https://oa.upm.es/35804/
• Semerci, Ç. (2011a). Doktora yeterlikler çerçevesinde öğretim üyesi, akran ve öz değerlendirmelerin Rasch ölçme modeliyle analizi. Journal of Measurement and Evaluation in Education and Psychology, 2(2), 164-171. https://dergipark.org.tr/en/pub/epod/issue/5804/77226
• Semerci, Ç. (2011b). Analyzing microteaching applications with many-facet Rasch measurement model. Education and Science, 36 (161), 14-25. http://egitimvebilim.ted.org.tr/index.php/EB/article/view/145
• Semerci, Ç., Semerci, N., & Duman, B. (2013). Analysis of seminar presentation performances of postgraduate students with many-facet Rasch model. The Journal of Sakarya Unıversity Education Faculty, 0(25), 7-22. https://dergipark.org.tr/tr/pub/sakaefd/issue/11221/133978
• Stefani, L. A. J. (1994). Peer, self and tutor assessment: Relative reliabilities. Studies in Higher Education, 19(1), 69-75. https://doi.org/10.1080/03075079412331382153
• Sudweeks, R. R., Reeve, S., & Bradshaw, W. S. (2004). A comparison of generalizability theory and many-facet Rasch measurement in an analysis of college sophomore writing. Assessing Writing, 9(3), 239-261. https://doi.org/10.1016/j.asw.2004.11.001
• Topping, K. J., Smith, E. F., Swanson, I., & Elliot, A. (2000). Formative Peer Assessment of Academic Writing Between Postgraduate Students. Assessment & Evaluation in Higher Education, 25(2), 149-169. https://doi.org/10.1080/713611428
• Tseng, S.-C., & Tsai, C.-C. (2007). On-line peer assessment and the role of the peer feedback: A study of high school computer course. Computers & Education, 49(4), 1161-1174. https://doi.org/10.1016/j.compedu.2006.01.007
• Turner, J. (2003). Examining on art portfolio assessment using a many facet Rasch measurement model [Unpublished dissertation]. Boston College.
• Uyanık, G. K., Güler, N., Teker, G. T., & Demir, S. (2019). The analysis of elementary science education course activities through many-facet Rasch model. Kastamonu Education Journal, 27(1), 139-150. https://doi.org/10.24106/kefdergi.2417
• Wright, B.D., & Linacre, J.M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370-382. https://www.rasch.org/rmt/rmt83b.htm
• Yan, Z., & Brown, G. T. (2017). A cyclical self-assessment process: Towards a model of how students engage in self-assessment. Assessment & Evaluation in Higher Education, 42(8), 1247-1262. https://doi.org/10.1080/02602938.2016.1260091
• Yan, Z., Wang, X., Boud, D., & Lao, H. (2023). The effect of self-assessment on academic performance and the role of explicitness: a meta-analysis. Assessment & Evaluation in Higher Education, 48(1), 1-15. https://doi.org/10.1080/02602938.2021.2012644
• Yüzüak, A. V., Yüzüak, B., & Kaptan, F. (2015). A many-facet Rasch measurement approach to analyze peer and teacher assessment for authentic assessment task. Journal of Measurement and Evaluation in Education and Psychology, 6(1), 1-11. https://doi.org/10.21031/epod.57425
• Zhan, Y., Yan, Z., Wan, Z. H., Wang, X., Zeng, Y., Yang, M., & Yang, L. (2023). Effects of online peer assessment on higher‐order thinking: A meta‐analysis. British Journal of Educational Technology, 54, 817-835. https://doi.org/10.1111/bjet.13310