Investigation of the Studies on Chatbots Designed for Educational Purposes

Year 2024, Volume: 22 Issue: 1, 299 - 336, 30.04.2024

Ahmet Yusuf Cevher , Serkan Yıldırım

https://doi.org/10.37217/tebd.1301877

Abstract

This study is based on a research group comprising 77 articles of various types retrieved from the Web of Science (WoS) database, aiming to uncover trends in research on educational chatbots. A qualitative research approach was adopted, employing a systematic review method. Data were obtained through document analysis, and content analysis and bibliometric analysis methods were used for the collected data. The findings indicated that publications on educational chatbots began in 2003 and rapidly increased in recent years. It was determined that China was the country with the highest number of publications, while the United States was the country with the most citations. Education and Information Technologies journal was found to be the most frequently published journal, with "chatbot" being the most preferred keyword. In many examined publications, there was either insufficient or incomplete discussion on the technology used in designing chatbots. Additionally, it was observed that most designed chatbots had a text-based interface and were primarily used for providing information. These robots were often developed using an artificial intelligence-based algorithm and served a complementary or supportive role during the learning process. Furthermore, it was noted that the effects of chatbots were predominantly examined on the variable of academic achievement. Lastly, it was observed that in many publications, information regarding the duration of chatbot implementation was not provided.

Keywords

Artificial intelligence, Chatbot, Systematic review, Bibliometric analysis, Content analysis

Eğitim Amaçlı Tasarlanan Sohbet Robotları Üzerine Yapılan Çalışmaların İncelenmesi

Abstract

Eğitim amaçlı sohbet robotları üzerine yapılan çalışmaların eğilimlerinin ortaya çıkarılması amacıyla yapılan bu araştırmanın çalışma grubunu Web of Science (WoS) veri tabanında taranan 77 adet makale türündeki yayın oluşturmaktadır. Araştırmada nitel araştırma tarzı benimsenerek sistematik derleme yöntemi kullanılmıştır. Çalışmaya ait veriler, doküman tarama yöntemiyle elde edilmiş, elde edilen verilere içerik analizi ve bibliyometrik analiz yöntemi uygulanmıştır. Bulgular, eğitim alanında kullanılan sohbet robotları üzerine yapılan yayınların 2003 yılında başladığını ve son yıllarda hızla arttığını göstermektedir. En fazla yayın yapılan ülkenin Çin olduğu ve en fazla atıf alan ülkenin ise Amerika Birleşik Devletleri olduğu belirlenmiştir. Education and Information Technologies dergisinin en fazla yayın yapılan dergi olduğu, en çok tercih edilen anahtar kelimenin ise "chatbot" olduğu tespit edilmiştir. İncelenen yayınların çoğunda, sohbet robotunun tasarımında kullanılan teknolojiden eksik ya da yetersiz bir şekilde bahsedildiği görülmüştür. Ayrıca, tasarlanan sohbet robotlarının çoğunlukla metin tabanlı bir arayüze sahip olduğu ve genellikle bilgi verme amacıyla kullanıldığı belirlenmiştir. Bu robotlar genellikle yapay zekâ tabanlı bir algoritma kullanılarak geliştirilmiş ve öğrenme sürecinde öğrenmeyi tamamlayıcı veya destekleyici bir rol oynamıştır. Ayrıca sohbet robotlarının etkilerinin en fazla başarı değişkeni üzerinde incelendiği gözlemlenmiştir. Son olarak, yayınların çoğunda sohbet robotlarının uygulama süresine dair bilgi verilmediği gözlemlenmiştir.

Keywords

Yapay zekâ, Sohbet robotu, Sistematik derleme, Bibliyometrik analiz, İçerik analizi

References

• Adamopoulou, E. & Moussiades, L. (2020). Chatbots: History, technology, and applications. Machine Learning with Applications, 2, 100006. https://doi.org/10.1016/j.mlwa.2020.100006
• Adiguzel, T., Kaya, M. H. & Cansu, F. K. (2023). Revolutionizing education with AI: Exploring the transformative potential of ChatGPT. Contemporary Educational Technology, 15(3), ep429. https://doi.org/10.30935/cedtech/13152
• Allen, I. E. & Seaman, J. (2014). Tracking online education in the United States. Babson Survey Research Group and Quahog Research Group, LLC. https://files.eric.ed.gov/fulltext/ED572777.pdf sayfasından erişilmiştir.
• Bjork, S., Offer, A. & Söderberg, G. (2014). Time series citation data: the Nobel prize in economics. Scientometrics, 98(1), 185–196. https://doi.org/10.1007/s11192-013-0989-5
• Bozkurt, A. (2013). Mega üniversitelerde öğrenci destek hizmetleri. Akademik Bilişim 2013 Konferansı’nda sunulmuş bildiri, Ocak 2013, Akdeniz Üniversitesi, Antalya. https://ab.org.tr/ab13/bildiri/126.pdf sayfasından erişilmiştir.
• Cevher, A. Y. & Yıldırım, S. (2020). Öğrenme stilleri konusunda yapılmış akademik çalışmaların incelenmesi: Sistematik derleme. HAYEF: Journal of Education (Online), 17(1), 20-50. https://doi.org/10.5152/hayef.2020.1922
• Chiu, T. K., Moorhouse, B. L., Chai, C. S. & Ismailov, M. (2023). Teacher support and student motivation to learn with Artificial Intelligence (AI) based chatbot. Interactive Learning Environments, 1-17. https://doi.org/10.1080/10494820.2023.2172044
• Clark, T. M. (2023). Investigating the Use of an Artificial Intelligence Chatbot with General Chemistry Exam Questions. Journal of Chemical Education, 100(5), 1905-1916. https://doi.org/10.1021/acs.jchemed.3c00027
• Cox, D. H. & Strange, C. C. (2010). Achieving student success: Effective student services in Canadian higher education. Montreal: McGill-Queen's Press-MQUP.
• Crawford, J., Cowling, M. & Allen, K. A. (2023). Leadership is needed for ethical ChatGPT: Character, assessment, and learning using artificial intelligence (AI). Journal of University Teaching & Learning Practice, 20(3), 2-19. https://doi.org/10.53761/1.20.3.02
• Falagas, M. E., Pitsouni, E. I., Malietzis, G. A. & Pappas, G. (2008). Comparison of PubMed, Scopus, web of science, and Google scholar: strengths and weaknesses. The FASEB Journal, 22(2), 338-342. https://doi.org/10.1096/fj.07-9492LSF
• Farajollahi, M. & Moenikia, M. (2010). The study of relation between students support services and distance students’ academic achievement. Procedia-Social and Behavioral Sciences, 2(2), 4451-4456. https://doi.org/10.1016/j.sbspro.2010.03.710
• Firat, M. (2020). Öğrenci destek servislerinde doğal dil işleme: GPT-3 örneği. International Conference of Strategic Research in Social Science and Education’da sunulmuş bildiri, Aralık 2020, Turkey. https://www.researchgate.net/publication/347929179 sayfasından erişilmiştir.
• Goktas, Y., Kucuk, S., Aydemir, M., Telli, E., Arpacik, O., Yildirim, G. & Reisoglu, I. (2012). Educational technology research trends in Turkey: A content analysis of the 2000-2009 decade. Educational Sciences: Theory and Practice, 12(1), 191-199. https://files.eric.ed.gov/fulltext/EJ978439.pdf sayfasından erişilmiştir.
• Gregori, E. B., Zhang, J., Galván-Fernández, C. & de Asís Fernández-Navarro, F. (2018). Learner support in MOOCs: Identifying variables linked to completion. Computers & Education, 122, 153-168. https://doi.org/10.1016/j.compedu.2018.03.014
• Heiser, J. F., Colby, K. M., Faught, W. S. & Parkison, R. C. (1979). Can psychiatrists distinguish a computer simulation of paranoia from the real thing? The limitations of turing-like tests as measures of the adequacy of simulations. Journal of Psychiatric Research, 15(3), 149–162. https://doi.org/10.1016/0022-3956(79)90008-6
• Hobert, S. & Wolff, R. M. (2019). Say hello to your new automated tutor- a structured literature review on pedagogical conversational agents. 14th International Conference on Wirtschaftsinformatik’da sunulmuş bildiri, Şubat 2019, Germany. https://core.ac.uk/download/pdf/301380749.pdf sayfasından erilmiştir.
• Horzum, M. B., Özkaya, M., Demirci, M. & Alpaslan, M. (2013). Review of Turkish distance education research. Inonu University Journal of the Faculty of Education, 14(2), 79-100. https://dergipark.org.tr/tr/download/article-file/92208 sayfasından erişilmiştir.
• Huang, W., Hew, K. F. & Fryer, L. K. (2022). Chatbots for language learning—Are they really useful? A systematic review of chatbot‐supported language learning. Journal of Computer Assisted Learning, 38(1), 237-257. https://doi.org/10.1111/jcal.12610
• Humphry, T. & Fuller, A. L. (2023). Potential ChatGPT use in undergraduate chemistry laboratories. Journal of Chemical Education, 100(4), 1434-1436. https://doi.org/10.1021/acs.jchemed.3c00006
• Hwang, G. J. & Chang, C. Y. (2021). A review of opportunities and challenges of chatbots in education. Interactive Learning Environments, 31(7), 4099-4112. https://doi.org/10.1080/10494820.2021.1952615
• Jeon, J. & Lee, S. (2023). Large language models in education: A focus on the complementary relationship between human teachers and ChatGPT. Education and Information Technologies, 28(12), 15873-15892. https://doi.org/10.1007/s10639-023-11834-1
• Karasozen, B., Bayram, G. & Zan, B. (2011). Wos ve Scopus veri tabanlarının karşılaştırılması. Türk Kütüphaneciliği, 25, 238‐260. https://www.researchgate.net/publication/290447910 sayfasından erişilmiştir.
• Keiper, M. C., Fried, G., Lupinek, J. & Nordstrom, H. (2023). Artificial intelligence in sport management education: Playing the AI game with ChatGPT. Journal of Hospitality, Leisure, Sport & Tourism Education, 33, 100456. https://doi.org/10.1016/j.jhlste.2023.100456
• Kohnke, L. (2023). L2 learners' perceptions of a chatbot as a potential independent language learning tool. International Journal of Mobile Learning and Organisation, 17(1-2), 214-226. https://www.inder scienceonline.com/doi/pdf/10.1504/IJMLO.2023.128339 sayfasından erişilmiştir.
• Kuhail, M. A., Alturki, N., Alramlawi, S. & Alhejori, K. (2022). Interacting with educational chatbots: A systematic review. Education and Information Technologies, 28, 973–1018. https://doi.org/10.1007/s10639-022-11177-3
• Kumar, J. A. (2021). Educational chatbots for project-based learning: investigating learning outcomes for a team-based design course. International Journal of Educational Technology in Higher Education, 18(1), 1-28. https://doi.org/10.1186/s41239-021-00302-w
• Kumtepe, E. G., Toprak, E., Öztürk, A., Büyükköse, G. T., Kılınç, H. & Menderis, İ. A. (2019). Açık ve uzaktan öğrenmede destek hizmetleri: Yerelden küresele bir model önerisi. Açıköğretim Uygulamaları ve Araştırmaları Dergisi, 5(3), 41-80. https://dergipark.org.tr/tr/download/article-file/853634 sayfasından erişilmiştir.
• Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P., ... & Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Journal of Clinical Epidemiology, 62(10), e1-e34. https://doi.org/10.1016/j.jclinepi.2009.06.006
• Martínez-Ruiz, T., Münch, J., García, F. & Piattini, M. (2012). Requirements and constructors for modeling variability in software processes, a systematic review. Software Quality Journal, 20(1), 229-260. https://doi.org/10.1007/s11219-011-9147-6
• Mitchell, P. & Grogono, P. (1993). Modeling techniques for tutoring systems. Computer Education, 20(1), 55-61. https://doi.org/10.1016/0360-1315(93)90070-Y
• Mohamed, A. M. (2023). Exploring the potential of an AI-based Chatbot (ChatGPT) in enhancing English as a Foreign Language (EFL) teaching: perceptions of EFL faculty members. Education and Information Technologies, 1-23. https://doi.org/10.1007/s10639-023-11917-z
• Moldt, J. A., Festl-Wietek, T., Madany Mamlouk, A., Nieselt, K., Fuhl, W. & Herrmann-Werner, A. (2023). Chatbots for future docs: exploring medical students’ attitudes and knowledge towards artificial intelligence and medical chatbots. Medical Education Online, 28(1), 2182659. https://doi.org/10.1080/10872981.2023.2182659
• Moore, M. G. (2003). Learner support. American Journal of Distance Education, 17(3), 141-143. https://doi.org/10.1207/S15389286AJDE1703_1
• Nichols, M. (2010). Student perceptions of support services and the influence of targeted interventions on retention in distance education. Distance Education, 31(1), 93-113. https://doi.org/10.1080/01587911003725048
• Nikolic, S., Daniel, S., Haque, R., Belkina, M., Hassan, G. M., Grundy, S., ... & Sandison, C. (2023). ChatGPT versus engineering education assessment: a multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity. European Journal of Engineering Education, 48(4), 559-614. https://doi.org/10.1080/03043797.2023.2213169
• Nimavat, K. & Champaneria, T. (2017). Chatbots: An overview types, architecture, tools and future possibilities International Journal for Scientific Research & Development, 5(7), 1019-1024. https://www.researchgate.net/publication/320307269 sayfasından erişilmiştir.
• Nov, O., Singh, N. & Mann, D. (2023). Putting ChatGPT’s medical advice to the (Turing) Test. arXiv. https://arxiv.org/abs/2301.10035 sayfasından erişilmiştir.
• Nuruzzaman, M. & Hussain, O. K. (2018). A survey on chatbot implementation in customer service industry through deep neural networks. IEEE 15th International Conference on e-Business Engineering’de sunulmuş bildiri, Ekim 2018, Xi'an, China. https://doi.org/10.1109/ICEBE.2018.00019
• Okonkwo, C. W. & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033. https://doi.org/10.1016/j.caeai.2021.100033
• Oracle Türkiye. (2014). Yapay zekâ nedir? https://www.oracle.com/tr/artificial-intelligence/what-is-ai/ sayfasından erişilmiştir.
• Pérez, J. Q., Daradoumis, T. & Puig, J. M. M. (2020). Rediscovering the use of chatbots in education: A systematic literature review. Computer Applications in Engineering Education, 28(6), 1549-1565. https://doi.org/10.1002/cae.22326
• Petticrew, M. & Roberts, H. (2008). Systematic reviews in the social science: A practical guide. UK: Blackwell.
• Rey-Martí, A., Ribeiro-Soriano, D. & Palacios-Marqués, D. (2016). A bibliometric analysis of social entrepreneurship. Journal of Business Research, 69(5), 1651–1655. https://doi.org/10.1016/j.jbusres.2015.10.033
• Reznik, L. (2009). General principles and purposes of computational intelligence. Systems Science and Cybernetics, 3, 198-222. https://books.google.com.tr/books?hl=tr&lr=&id=2-VRCwAAQBAJ&oi= fnd&pg=PA198&dq=%22General+Principles+and+Purposes+of+Computational+Intelligence%22&ots=m-LL1sYe2-&sig=D5eqAnG7gjezeKrTTpeHge3gpWU&redir_esc=y#v=onepage&q=% 22General%20Principles%20and%20Purposes%20of%20Computational%20Intelligence%22&f=false sayfasından erişilmiştir.
• Rumble, G. (2000). Student support in distance education in the 21st century: Learning from service management. Distance Education, 21(2), 216-235. https://doi.org/10.1080/0158791000210202
• Shum, H. Y., He, X. D. & Li, D. (2018). From Eliza to XiaoIce: challenges and opportunities with social chatbots. Frontiers of Information Technology & Electronic Engineering, 19(1), 10-26. https://doi.org/10.1631/FITEE.1700826
• Simpson, O. (2000). Supporting students in open and distance learning. London: Kogan Page. https://www.researchgate.net/publication/42795970 sayfasından erişilmiştir.
• Singh, R. (2018, Mayıs 2). AI and chatbots in education: What does the future hold? https://chatbotsmagazine.com/ai-and-chatbots-in-education-what-does-thefuturehold-9772f5c13960 sayfasından erişilmiştir.
• Skavronskaya, L., Hadinejad, A. & Cotterell, D. (2023). Reversing the threat of artificial intelligence to opportunity: a discussion of ChatGPT in tourism education. Journal of Teaching in Travel & Tourism, 23(2), 253-258. https://doi.org/10.1080/15313220.2023.2196658
• Smutny, P. & Schreiberova, P. (2020). Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Computers & Education, 151, 103862. https://doi.org/10.1016/j.compedu.2020.103862
• Song, D., Oh, E. Y. & Rice, M. (2017). Interacting with a conversational agent system for educational purposes in online courses. 2017 10th International Conference on Human System Interactions (HSI) içinde (s. 78-82), Temmuz, IEEE.
• Su, Y., Lin, Y. & Lai, C. (2023). Collaborating with ChatGPT in argumentative writing classrooms. Assessing Writing, 57, 100752. https://doi.org/10.1016/j.asw.2023.100752
• Sweitzer, J. L. (2005). Proper support improves online student success. Journal on Systemics, Cybernetics and Informatics, 3(5), 45-48. https://www.iiisci.org/journal/pdv/sci/pdfs/P550243.pdf sayfasından erişilmiştir.
• Tait, A. (2000). Planning student support for open and distance learning. Open Learning, 15(3), 287-299. https://doi.org/10.1080/713688410
• Tam, W., Huynh, T., Tang, A., Luong, S., Khatri, Y. & Zhou, W. (2023). Nursing education in the age of artificial intelligence powered Chatbots (AI-Chatbots): Are we ready yet? Nurse Education Today, 129, 105917. https://doi.org/10.1016/j.nedt.2023.105917
• Tamayo, P. A., Herrero, A., Martín, J., Navarro, C. & Tránchez, J. M. (2020). Design of a chatbot as a distance learning assistant. Open Praxis, 12(1), 145-153. https://doi.org/10.5944/openpraxis.12.1.1063
• Thorpe, M. (2010). Rethinking learner support: the challenge of collaborative online learning. Open Learning: The Journal of Open and Distance Learning, 17(2), 105-119. https://doi.org/10.1080/02680510220146887a
• Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R. & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10(1), 1-24. https://doi.org/10.1186/s40561-023-00237-x
• Topal, A. D., Eren, C. D. & Geçer, A. K. (2021). Chatbot application in a 5th grade science course. Education and Information Technologies, 26, 6241-6265. https://link.springer.com/article/10.1007/s10639-021-10627-8#citeas sayfasından erişilmiştir.
• Tsang, R. (2023). Practical applications of ChatGPT in undergraduate medical education. Journal of Medical Education and Curricular Development, 10, 23821205231178449. https://doi.org/10.1177/23821205231178449
• Turing, A. M. (1950). Computing machinery and intelligence. Mind, LIX 236, 433–460. https://doi.org/10.1093/mind/LIX.236.433
• Uçar, H. (2016). Uzaktan eğitimde motivasyon stratejilerinin öğrenenlerin ilgileri, motivasyonları, eylem yeterlikleri ve başarıları üzerine etkisi. (Doktora Tezi). http://tez.yok.gov.tr sayfasından erişilmiştir.
• Usun, S. (2004). Learner support services in distance education system (a case study of Turkey). Turkish Online Journal of Distance Education (TOJDE), 5(4), 31-39. https://files.eric.ed.gov/fulltext/ED494552.pdf sayfasından erişilmiştir.
• Verleger, M. & Pembridge, J. (2018). A pilot study integrating an AI-driven chatbot in an introductory programming course. 2018 IEEE Frontiers in Education Conference’da sunulmuş bildiri, Ekim 2018, San Jose, USA. https://doi.org/10.1109/FIE.2018.8659282
• Wallace, R. S. (2009). The anatomy of ALICE. Springer Netherlands.
• Wang, Y. H., Wang, W. N. & Yen, Y. H. (2005). An intelligent semantic agent for e-learning message communication. Journal of Information Science and Engineering, 21, 1031-1051. https://www.researchgate.net/publication/4140862 sayfasından erişilmiştir.
• Winkler, R. & Söllner, M. (2018). Unleashing the potential of chatbots in education: A state-of-theart analysis. Academy of Management Annual Meeting (AOM), 1, 15903. https://doi.org/10.5465/AMBPP.2018.15903abstract
• Wollny, S., Schneider, J., Di Mitri, D., Weidlich, J., Rittberger, M. & Drachsler, H. (2021). Are we there yet?-A systematic literature review on chatbots in education. Frontiers in Artificial Intelligence, 4, 654924. https://doi.org/10.3389/frai.2021.654924
• Xia, Q., Chiu, T. K., Chai, C. S. & Xie, K. (2023). The mediating effects of needs satisfaction on the relationships between prior knowledge and self‐regulated learning through artificial intelligence chatbot. British Journal of Educational Technology, 54(4), 813-1045. https://doi.org/10.1111/bjet.13305
• Yan, D. (2023). Impact of ChatGPT on learners in a L2 writing practicum: An exploratory investigation. Education and Information Technologies, 28, 13943-13967. https://doi.org/10.1007/s10639-023-11742-4
• Yıldırım, A. & Şimşek, H. (2006). Sosyal bilimlerde nitel araştırma yöntemleri (11. b.). Ankara: Seçkin.