Reliability Analysis of the Feedback Scale of a Course with Classical Test Theory and Generalizability Theory

Year 2020, Volume: 3 Issue: 2, 20 - 24, 25.10.2020

Giray Kolcu , Mukadder İnci Başer Kolcu

Abstract

Introduction: For the development of medical education within the framework of the concept of accountability, the preferred measurement tools for program evaluation should be reliable Reliability refers to the consistency of scores obtained with a certain measurement tool.
Objective: In this study, we aimed to evaluate the internal consistency reliability analysis of the qualitative research course feedback with classical test theory (Crohnbach alpha) and generalizability theory (G coefficient).
Method: In this study, feedback data of 46 participants from qualitative research courses implemented by the researcher were evaluated. The descriptive analyzes of the data of the feedback scale and reliability coefficients were determined according to the classical test theory and generalizability theory.
Results: In the evaluation of feedback by classical test theory, Crohnbach's alpha coefficient was calculated as 0.947 in the evaluation of the 20-item instrument of 47 participants. In the evaluation of the scale with generalizability theory,were estimated for predicting the single-surface crossed patterned variance components. G coefficient was calculated as 0,95.
Discussion: Reliability analysis of measurement tools preferred as data collection tools for program evaluation is important for accountability. In the literature, 0.70 and above is accepted as the universal reliability standard in reliability analyzes. In the reliability analysis of the scale in our study, it was considered acceptable according to the classical test theory and generalizability theory. We believe that the feedback scale evaluated in the scope of our study can be used as a reliable measurement tool in the evaluation of this training program.

Keywords

program evaluation, feedback, generalizbility theory, reliability

References

• 1. Melle E Van, Flynn L, Kaba A, Tavares W, Horsley T. Program Evaluation for Competency-based medical Education : Are we making a difference ? Int Conf Resid Educ. 2016;
• 2. Gandomkar R, Sandars J. The importance of context in medical education program evaluation. Med Teach [Internet]. 2018 Jan 2;40(1):106. Available from: https://doi.org/10.1080/0142159X.2017.1390215
• 3. Medicine DF of. Program Evaluation - Division of Medical Education - Dalhousie University [Internet]. [cited 2019 Sep 7]. Available from: https://medicine.dal.ca/departments/core-units/DME/education/programs-centres/program-evaluation.html
• 4. Francisco U of CS. Program Evaluation Policy (Medical Student) | UCSF Medical Education [Internet]. [cited 2019 Sep 7]. Available from: https://meded.ucsf.edu/policies-procedures/program-evaluation-policy-medical-student
• 5. Vassar M, Wheeler DL, Davison M, Franklin J. Program Evaluation in Medical Education: An Overview of the Utilization-focused Approach Journal of Educational Evaluation for Health Professions. J Educ Eval Heal Prof [Internet]. 2010 [cited 2019 Jan 4];7:1. Available from: http://jeehp.org/
• 6. Behçet P, Yrd O, Taha D. Eğitimde program geliştirme ve değerlendirme. Eğitimde program geliştirme ve değerlendirme. 2017.
• 7. Stufflebeam D. Evaluation models. New Dir Eval. 2001;2001(89):7–98.
• 8. Durning SJ, Hemmer P, Pangaro LN. The structure of program evaluation: An approach for evaluating a course, clerkship, or components of a residency or fellowship training program. Teach Learn Med. 2007;19(3):308–18.
• 9. Medicine SU of. Undergraduate Medical Educaion Program Evaluation Strategy [Internet]. 2016 [cited 2019 Jan 4]. Available from: https://medicine.usask.ca/documents/ugme/EvaluationFramework.pdf
• 10. Walsh K. Evaluation in medical education [Internet]. [cited 2019 Jan 4]. Available from: https://www.monash.edu/__data/assets/pdf_file/0007/935692/evaluation-in-medical-education.pdf
• 11. Cook DA, Hatala R. Validation of educational assessments: a primer for simulation and beyond. Adv Simul [Internet]. 2016;1(1):1–12. Available from: http://dx.doi.org/10.1186/s41077-016-0033-y
• 12. Şencan H. Sosyal ve Davranışsal Ölçümlerde Güvenilirlik ve Geçerlilik (Validity and Reliability in Social and Behavioral Measurements). 2005. 13. Baykul Y. Eğitimde ve Psikolojide Ölçme: Klasik Test Teorisi ve Uygulaması. Ankara: ÖSYM Yayınları; 2000.
• 14. Ercan İ, Kan İ. Ölçeklerde Güvenirlik ve Geçerlik. Uludağ Üniversitesi Tıp Fakültesi Derg. 2004;30(3):211–6.
• 15. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297–334.
• 16. Tavakol M, Brennan RL. Medical education assessment : a brief overview of concepts in generalizability theory. Int J Med Educ. 2013;4(4):221–2.
• 17. Castanelli DJ, Moonen-van Loon JMW, Jolly B, Weller JM. The reliability of a portfolio of workplace-based assessments in anesthesia training. Can J Anesth Can d’anesthésie [Internet]. 2019;66(2):193–200. Available from: https://doi.org/10.1007/s12630-018-1251-7
• 18. Güler N. Genellenebilirlik Kuramı ve SPSS ile GENOVA Programlarıyla Hesaplanan G ve K Çalışmalarına İlişkin Sonuçların Karşılaştırılması Generalizability Theory and Comparison of the Results of G and D Studies Computed by SPSS and GENOVA Packet Programs. Eğitim ve Bilim. 2009;34(154):93–104.
• 19. Cho E, Kim S. Cronbach’s Coefficient Alpha: Well Known but Poorly Understood. Organ Res Methods. 2015;18(2):207–30.
• 20. Atilgan H. Genellenebilirlik kurami ve puanlayicilar arasi güvenirlik için örnek bir uygulama. Egit ve Bilim. 2005;(7):95–108.
• 21. Güler N, Eroğlu Y, Akbaba S. Reliability of Criterion-Dependent Measurement Tools According To Generalizability Theory: Application in the Case of Eating Skills. 2014;14:217–32.
• 22. Shavelson RJ, Webb NM. Generalizability theory: A primer. Generalizability theory: A primer. Thousand Oaks, CA, US: Sage Publications, Inc; 1991. xiii, 137–xiii, 137. (Measurement methods for the social sciences series, Vol. 1.).
• 23. Cronbach JL, Gleser GC NH& RN. The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. New York: John Wiley and Sons; 1972.
• 24. İbrahim Turan, Ümit Şimşek HA. The Use and Analysis of Likert Scales and Likert - Type Items in Educational Giriş. Sak Univ J Educ. 2015;(30):186–203.
• 25. Mushquash, C., & O’Connor BP. SPSS, SAS, and MATLAB programs for generalizability theory analyses. Behav Res Methods. 2006;38(3):542–7.
• 26. Brennan RL. Generalizability theory. Generalizability theory. New York, NY, US: Springer-Verlag Publishing; 2001. xx, 538–xx, 538. (Statistics for social science and public policy.).
• 27. Gibson KA, Boyle P, Black DA, Cunningham M, Grimm MC, McNeil HP. Enhancing evaluation in an undergraduate medical education program. Acad Med. 2008;83(8):787–93. 28. Musick DW. A conceptual model for program evaluation in graduate medical education. Acad Med. 2006;81(8):759–65.
• 29. Kassab SE, Du X, Toft E, Cyprian F, Al-Moslih A, Schmidt H, et al. Measuring medical students’ professional competencies in a problem-based curriculum: a reliability study. BMC Med Educ [Internet]. 2019;19(1):155. Available from: https://doi.org/10.1186/s12909-019-1594-y
• 30. Bajwa NM, Nendaz MR, Galetto-Lacour A, Posfay-Barbe K, Yudkowsky R, Park YS. Can Professionalism Mini-Evaluation Exercise Scores Predict Medical Residency Performance? Validity Evidence Across Five Longitudinal Cohorts. Acad Med [Internet]. 9000;Publish Ah. Available from: https://journals.lww.com/academicmedicine/Fulltext/publishahead/Can_Professionalism_Mini_Evaluation_Exercise.97503.aspx
• 31. Monteiro S, Sullivan GM, Chan TM. Generalizability Theory Made Simple(r): An Introductory Primer to G-Studies. J Grad Med Educ [Internet]. 2019;11(4):365–70. Available from: http://www.jgme.org/doi/10.4300/JGME-D-19-00464.1
• 32. Taşdelen Teker G, Odabaşı O. Reliability of scores obtained from standardized patient and instructor assessments. Eur J Dent Educ [Internet]. 2019 May 1;23(2):88–94. Available from: https://doi.org/10.1111/eje.12406