EVALUATING THE SUCCESS OF ARTIFICIAL INTELLIGENCE CHATBOTS IN ANSWERING COMMON ORTHODONTIC QUESTIONS

Year 2023, Volume: 30 Issue: 4, 680 - 686, 30.12.2023

Sercan Taşkın , Mine Geçgelen Cesur , Mustafa Uzun

https://doi.org/10.17343/sdutfd.1369828

Abstract

Material and Method
The 15 questions commonly asked by patients
undergoing orthodontic treatment were searched
from the internet database and asked to ChatGPT3.5,
ChatGPT4, BING, and ZekAI applications on May
15, 2023. The answers given by the applications
were scored by two orthodontists, two orthodontic
residents, and two dentists on a Likert Scale from 1
(most adequate) to 5 (most inadequate).
Results
In the study, two orthodontists (MGC, MU) and two
orthodontic research assistants (AAA, ST) have
assigned significantly lower scores to the ChatGPT4
application compared to the BING application
(p<0.001). When the success scores assigned by
the participants to the applications were evaluated,
it was found that the BING application received
significantly lower success scores from one of the
dentists (ÇA) compared to the orthodontic research
assistants (AAA, ST) (p<0.001). One of the dentists
in the study (ÇA), on the other hand, evaluated the
ZekAI application as significantly more successful
compared to the orthodontist (MU), the other dentist
(MMU), and the orthodontic research assistant (ST)
(p<0.001). When all artificial intelligence chatbots
were compared among themselves, it was found
that ChatGPT4 provided the most desired responses
(p<0.001).
Conclusion
This research has substantiated, based on the
evaluation by participants, that the ChatGPT4
application represents the most reliable solution for
assisting clinicians in addressing prevalent orthodontic
queries, surpassing the ChatGPT3.5, BING, and
ZekAI applications in this regard.

Keywords

Artificial intelligence in orthodontics, chatbots in orthodontics, orthodontic patient education

YAPAY ZEKÂ DESTEKLİ SOHBET ROBOTLARININ YAYGIN ORTODONTİK SORULARI CEVAPLAMA BAŞARISININ DEĞERLENDİRİLMESİ

Abstract

Amaç
Bu çalışmada ortodontik tedavi gören hastaların ortodontistlerine
yaygın olarak sordukları soruların yapay
zekâ destekli sohbet robotları tarafından cevaplanma
başarısının değerlendirilmesi amaçlanmıştır.
Gereç ve Yöntem
Ortodontik tedavi gören hastaların yaygın olarak sordukları
15 soru internet veri tabanından araştırılarak
ChatGPT3.5, ChatGPT4, BING ve ZekAI uygulamalarına
15 Mayıs 2023’te sorulmuştur. Uygulamaların
verdikleri cevaplar iki ortodonti uzmanı, iki ortodonti
araştırma görevlisi ve iki diş hekimi tarafından 1 (en
yeterli)’den 5 (en yetersiz)’e kadar oluşturulan Likert
Skalası ölçeğine göre puanlanmıştır.
Bulgular
İstatistiksel değerlendirmeler sonucunda çalışmada
yer alan iki ortodonti uzmanı (MGC, MU) ve iki ortodonti
asistanı (AAA, ST) ChatGPT4 uygulamasına
BING uygulamasına göre istatistiksel olarak anlamlı
derecede düşük puan vermiştir (p<0,001). Hekimlerin
uygulamalara verdikleri başarı puanları değerlendirildiğinde,
BING uygulamasının çalışmada yer alan
diş hekimlerinden birinden (ÇA) aldığı başarı puanı
ortodonti araştırma görevlilerine (AAA, ST) göre istatistiksel
olarak anlamlı derecede düşük olmuştur
(p<0,001). Çalışmada yer alan diş hekimlerinden biri
(ÇA) ise ZekAI uygulamasını ortodonti uzmanı (MU),
diğer diş hekimi (MMU) ve ortodonti araştırma görevlisine
(ST) göre istatistiksel olarak anlamlı derecede
başarılı değerlendirmiştir (p<0,001). Tüm yapay zekâ
sohbet robotları kendi aralarında kıyaslandığında
ChatGPT4’ün en istenilen cevapları verdiği bulunmuştur
(p<0,001).
Sonuç
Bu çalışma, cevapları değerlendiren hekimlere göre
ChatGPT4 uygulamasının ChatGPT3.5, BING ve ZekAI
uygulamalarına göre yaygın ortodontik sorulara
cevap vermede hekimlere yardımcı olabilecek en güvenilir
uygulama olduğunu göstermiştir.

Keywords

Ortodontide hasta bilgilendirme, ortodontide yapay zekâ, yapay zekâ sohbet robotu

References

• 1. Russell S. Artificial intelligence a modern approach. Pearson Education; 2010.
• 2. Wong S. Artificial intelligence in radiology: how will we be affected? European radiology. 2019;141-143. https://doi. org/10.1007/s00330-018-5644-3
• 3. Andreu-Perez J, Deligianni F, Ravi D, Yang G. Artificial intelligence and robotics. arXiv preprint. 2018;1803.10813. https:// doi.org/10.48550/arXiv.1803.10813
• 4. Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, et al. Artificial intelligence in healthcare: past, present and future. Stroke and Vascular Neurology 2017;2:e000101. doi:10.1136/svn-2017- 000101
• 5. Aggarwal A, Tam CC, Wu D, Li X, Qiao S. Artificial Intelligence– Based Chatbots for Promoting Health Behavioral Changes: Systematic Review. Journal of Medical Internet Research. 2023;25:e40789. doi: 10.2196/40789.
• 6. Lyons R, Arepallli S, Fromal O, Choi J, Jain N. Artificial intelligence chatbot performance in triage of ophthalmic conditions. Canadian Journal of Ophthalmology. 2023;S0008- 4182(23)00234-X. doi: 10.1016/j.jcjo.2023.07.016.
• 7. Bhardwaz S, Kumar J. An Extensive Comparative Analysis of Chatbot Technologies-ChatGPT, Google BARD and Microsoft Bing. In 2nd International Conference on Applied Artificial Intelligence and Computing. India, 2023;673-679. doi: 10.1109/ ICAAIC56838.2023.10140214.
• 8. ZekAI. Curiosity Technology [Internet]. Cited: 15 Mayıs 2023. Available from: https://zekai.co/tr/
• 9. Skidmore K, Brook K, Thomson W, Harding W. Factors influencing treatment time in orthodontic patients. American Journal of Orthodontics and Dentofacial Orthopedics 2006;129(2):230-8. doi: 10.1016/j.ajodo.2005.10.003.
• 10. Pratt M, Kluemper G, Lindstrom A. Patient compliance with orthodontic retainers in the postretention phase. American Journal of Orthodontics and Dentofacial Orthopedics. 2011;140(2):196- 201. doi: 10.1016/j.ajodo.2010.02.035.
• 11. Gyawali R, Pokharel P, Giri J. Emergency appointments in orthodontics. APOS Trends Orthod. 2019;9(1):40-43. doi:10.25259/APOS-9-1-7
• 12. Verma S, Sharma R, Deb S, Maitra D. Artificial intelligence in marketing: Systematic review and future research direction. International Journal of Information Management Data Insights. 2021;1(1):100002. https://doi.org/10.1016/j.jjimei.2020.100002
• 13. Benichou L. Rôle de l'utilisation de l'intelligence artificielle ChatGPT dans la rédaction des articles scientifiques médicaux. The Role of Using ChatGPT AI in Writing Medical Scientific Articles. J Stomatol Oral Maxillofac Surg. 2023;124(5):101456. doi: 10.1016/j.jormas.2023.101456.
• 14. de Cock C, Milne-Ives M, van Velthoven M, Alturkistani A, Lam C, Mein E. Effectiveness of conversational agents (virtual assistants) in health care: protocol for a systematic review. JMIR research protocols. 2020;9(3):e16934.
• 15. Kessels R. Patients’ memory for medical information. Journal of the Royal Society of Medicine. 2003;96(5):219-222.
• 16. Monill-Gonza´lez A, Rovira-Calatayud L, d'Oliveria N, Ustrell- Torrent J. Artificial intelligence in orthodontics: where are we now? A scoping review. Orthod Craniofac Res. 2021;24: 6-15.
• 17. Ball M, Carla Smith N, Bakalar R. Personal health records: empowering consumers. J Healthc Inf Manag. 2007;21(1):77.
• 18. Clarke M, Moore J, Steege L, Koopman R, Belden J, Canfield S. Health information needs, sources, and barriers of primary care patients to achieve patient-centered care: A literature review. Health inf. Health Informatics J. 2016;22(4):992-1016.
• 19. Rinchuse D, Cozzani M. Effectiveness and efficiency in clinical orthodontic practice. International Orthodontics. 2015;13(4): 507–524.
• 20. Perez-Pino A, Yadav S, Upadhyay M, Cardarelli L, Tadinada A. The accuracy of artificial intelligence–based virtual assistants in responding to routinely asked questions about orthodontics. The Angle Orthodontist. 2023;93(4):427-432.
• 21. Balel Y. Can ChatGPT be used in oral and maxillofacial surgery? Journal of Stomatology, Oral and Maxillofacial Surgery. 2023;124(5):101471.
• 22. Muttanahally K, Tadinada A, Mago J, Vyas R. Usefulness of Artificial Intelligence-based Virtual Assistants in Oral and Maxillofacial Radiology Report Writing. World Journal of Dentistry. 2021;12(2):97-102.
• 23. Tustumi F, Andreollo N, Aguilar-Nascimento J. Future of the language models in healthcare: the role of chatGPT. Arquivos Brasileiros de Cirurgia Digestiva. 2023;356:e1721. doi: 10.1590/0102-672020230002e1727
• 24. Nobles A, Leas E, Caputi T, Zhu S, Strathdee S, Ayers J. Responses to addiction help-seeking from Alexa, Siri, Google Assistant, Cortana, and Bixby intelligent virtual assistants. NPJ digital medicine. 2020; 3(1):11.