Evaluation of ChatGPT as a Multiple-Choice Question Generator in Dental Traumatology

Year 2024, Volume: 6 Issue: 2, 235 - 238, 16.05.2024

Yağız Özbay

https://doi.org/10.37990/medr.1446396

Abstract

Aim: This study aims to evaluate the ability of ChatGPT-4 to generate clinical case-based multiple-choice questions in dental traumatology.
Material and Method: International Association of Dental Traumatology guidelines for the management of traumatic dental injuries were introduced to ChatGPT-4 as an information source and ChatGPT-4 was prompted to 20 generate questions in fractures and luxations, avulsion of permanent teeth, injuries in the primary dentition. Sixty questions in total were generated by ChatGPT and the questions were evaluated by 2 endodontists using a 3-point scale. The One-way analysis of variance and Post Hoc Tukey test were used to analyze the data and the significance was P<0.05.
Results: The average time to generate 20 questions was 1 min 55 sec. It was noted that 52% of the questions were usable without modification or with minor changes, while 28% were incorrect or completely useless.
Conclusion: Despite the flaws, ChatGPT can be useful for creating multiple-choice questions in dental traumatology after a rigorous evaluation, elimination, and development procedure.

Keywords

artificial Intelligence, clinical practice guidelines, dental education, endodontics, treatment planning

Ethical Statement

This study does not require an ethical approval

References

• Javaeed A. Assessment of higher ordered thinking in medical education: multiple choice questions and modified essay questions. MedEdPublish. 2018;7:128.
• Scully D. Constructing multiple-choice items to measure higher-order thinking. PARE. 2019;22:4.
• Scouller K. The influence of assessment method on students' learning approaches: multiple choice question examination versus assignment essay. Higher Education; Dordrecht. 1998;35:453-72.
• Rao SK, Kimball AB, Lehrhoff SR, et al. The impact of administrative burden on academic physicians: results of a hospital-wide physician survey. Acad Med. 2017;92:237-43.
• Chen L, Chen P, Lin Z. Artificial intelligence in education: a review. IEEE Access. 2020;8:75264-78.
• Cardoso MJ, Houssami N, Pozzi G, Séroussi B. Artificial intelligence (AI) in breast cancer care-leveraging multidisciplinary skills to improve care. Artif Intell Med. 2022;123:102215.
• Abd-Alrazaq A, AlSaad R, Alhuwail D, et al. Large language models in medical education: opportunities, challenges, and future directions. JMIR Med Educ. 2023;9:e48291.
• Mesko B, Topol EJ. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. NPJ Digit Med. 2023;6:120.
• Fatani B. ChatGPT for future medical and dental research. Cureus. 2023;15:e37285.
• Giannakopoulos K, Kavadella A, Stamatopoulos V, Kaklamanos E. Evaluation of generative artificial intelligence large language models ChatGPT, Google Bard, and Microsoft Bing Chat in supporting evidence-based dentistry: a comparative mixed-methods study. J Med Internet Res. 2023;25:e51580.
• Acar AH. Can natural language processing serve as a consultant in oral surgery?. J Stomatol Oral Maxillofac Surg. 2024;125:101724.
• Suarez A, Diaz-Flores Garcia V, Algar J, et al. Unveiling the ChatGPT phenomenon: evaluating the consistency and accuracy of endodontic question answers. Int Endod J. 2024;57:108-13.
• Ali K, Barhom N, Tamimi F, Duggal M. ChatGPT-A double-edged sword for healthcare education? Implications for assessments of dental students. Eur J Dent Educ. 2024;28:206-11.
• Cheung BHH, Lau GKK, Wong GTC, et al. ChatGPT versus human in generating medical graduate exam multiple choice questions-A multinational prospective study (Hong Kong S.A.R., Singapore, Ireland, and the United Kingdom). PLoS One. 2023;18:e0290691.
• Ayub I, Hamann D, Hamann CR, Davis MJ. Exploring the potential and limitations of chat generative pre-trained transformer (ChatGPT) in generating board-style dermatology questions: a qualitative analysis. Cureus. 2023;15:e43717.
• Kim HW, Shin DH, Kim J, et al. Assessing the performance of ChatGPT's responses to questions related to epilepsy: a cross-sectional study on natural language processing and medical information retrieval. Seizure. 2024;114:1-8.
• Spallek S, Birrell L, Kershaw S, et al. Can we use ChatGPT for mental health and substance use education? Examining its quality and potential harms. JMIR Med Educ. 2023;9:e51243.
• Bourguignon C, Cohenca N, Lauridsen E, et al. International Association of Dental Traumatology guidelines for the management of traumatic dental injuries: 1. fractures and luxations. Dent Traumatol. 2020;4:314-30.
• Levin L, Day PF, Hicks L, et al. International Association of Dental Traumatology guidelines for the management of traumatic dental injuries: general introduction. Dent Traumatol. 2020;4:309-13.
• Fouad AF, Abbott PV, Tsilingaridis G, et al. International Association of Dental Traumatology guidelines for the management of traumatic dental injuries: 2. avulsion of permanent teeth. Dent Traumatol. 2020;36:331-42.
• Day PF, Flores MT, O'Connell AC, et al. International Association of Dental Traumatology guidelines for the management of traumatic dental injuries: 3. injuries in the primary dentition. Dent Traumatol. 2020;36:343-59.
• Ngo A, Gupta S, Perrine O, et al. ChatGPT 3.5 fails to write appropriate multiple choice practice exam questions. Acad Pathol. 2023;11:100099.