How usable is Coursera? A usability analysis through eye-tracking and authentic tasks

Year 2022, Volume: 9 Issue: 4, 379 - 395, 01.07.2022

Hasan Tanış , Arif Akçay , Nursel Yılmaz , Mehmet Fatih Yiğit , Hakan Tüzün

https://doi.org/10.17275/per.22.96.9.4

Abstract

Massive Open Online Courses (MOOCs) are considered learning environments that eliminate many learning barriers. Online courses in MOOCs have become an opportunity for everyone during the lockdown of the COVID-19 outbreak. However, usability issues may cause problems such as high dropout rates and lack of learner's motivation. Therefore, in this study, the usability of Coursera, one of the most known MOOCs in the world, was evaluated. The evaluation was performed with ISO 9241-11 standard. The environment's effectiveness, efficiency, and satisfaction were evaluated with the authentic tasks requested to be done in Coursera. Additionally, the findings were supported by eye-tracking metrics such as fixation duration, fixation counts, heat maps, and gaze plots. Twelve individuals (six females, six males) participated in the authentic tasks, and three individuals (two females, one male) participated in the eye-tracking phase. Results of the study revealed that most participants successfully performed the authentic tasks and are generally satisfied with the usability of the environment. However, considering eye-tracking findings and Coursera Usage Satisfaction Survey, some usability problems such as inadequate language support and the difficulty of using the search feature emerged. In the end, possible reasons were discussed, and the suggestions were presented for usability improvements.

Keywords

usability, coursera, mooc, eye tracking

References

• Abran, A., Khelifi, A., Suryn, W., & Seffah, A. (2003). Usability meanings and interpretations in ISO standards. Software quality journal, 11(4), 325-338.
• Becker, S. A., & Mottay, F. E. (2001). A global perspective on web site usability. IEEE software, 18(1), 54-61.
• Conole, G. G. (2013). MOOCs as disruptive technologies: strategies for enhancing the learner experience and quality of MOOCs. Revista de Educación a Distancia, (39).
• Dix, A., Finlay, J., Abowd, G. D., & Beale, R. (2004). Human-computer interaction. Edingburh: Pearson Prentice Hall.
• Espada, J. P., Rodríguez, C. C., García-Díaz, V., & Crespo, R. G. (2014). Method for analysing the user experience in MOOC platforms. In Computers in Education (SIIE), International Symposium, 2014 (pp. 157-162). IEEE.
• Gütl, C., Rizzardini, R. H., Chang, V., & Morales, M. (2014). Attrition in MOOC: Lessons learned from drop-out students. In International Workshop on Learning Technology for Education in Cloud, 2014 (pp. 37-48). Springer International Publishing.
• Hone, K. S., & El Said, G. R. (2016). Exploring the factors affecting MOOC retention: A survey study. Computers & Education, 98, 157-168.
• International Organization for Standardization (ISO) (2018). ISO 9241-11: Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs)’, Part 11: Guidance on Usability Specification and Measures. Technical Report. Retrieved January 05, 2020 from https://www.iso.org/standard/63500.html
• Mackness, J., Mak, S. F. J., & Williams, R. (2010). The ideals and reality of participating in a MOOC. In L. Dirckinck-Holmfeld, V. Hodgson, C. Jones, M. de Laat, D. McConnell, & T. Ryberg (Eds.), Proceedings of the Seventh International Conference on Networked Learning (pp. 266-275).
• Mackness, J., Waite, M., Roberts, G., & Lovegrove, E. (2013). Learning in a small, task-oriented, connectivist MOOC: Pedagogical issues and implications for higher education. International Review of Research in Open & Distance Learning, 14, 140-159.
• McAuley, A., Stewart, B., Siemens, G., & Cormier, D. (2010). The MOOC model for digital practice.
• Nantel, J., & Glaser, E. (2008). The impact of language and culture on perceived website usability. Journal of Engineering and Technology Management, 25(1), 112-122.
• Nielsen, (2012). Usability 101: Introduction to Usability. Retrieved February 21, 2019 from https://www.nngroup.com/articles/usability-101-introduction-to-usability/
• Nielsen, J. (2000). Why you only need to test with 5 users. Retrieved February 21, 2019 from https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
• Nielsen, J. (2002). Top 10 Guidelines for Homepage Usability. Retrieved February 21, 2019 from https://www.nngroup.com/articles/top-ten-guidelines-for-homepage-usability/
• Nielsen, J., & Pernice, K. (2009). How to Conduct Eyetracking Studies. Retrieved May 16, 2019 from https://media.nngroup.com/media/reports/free/How_to_Conduct_Eyetracking_Studies.pdf
• Onah, D. F., Sinclair, J., & Boyatt, R. (2014). Dropout rates of massive open online courses: behavioural patterns. EDULEARN 14 Proceedings, 5825-5834.
• Park, J. H. (2007). Factors related to learner dropout in online learning. In F. M. Nafukho, T. H. Chermack, & C. M. Graham (Eds.), Proceedings of the 2007 academy of human resource development annual conference.
• Pireva, K., Imran, A. S., & Dalipi, F. (2015). User behaviour analysis on LMS and MOOC. IEEE Conference on e-Learning, e-Management and e-Services (IC3e), 2015 (pp. 21-26). IEEE.
• Rayner, K. (1998). Eye movements in reading and information processing. Psychological Bulletin, 85(3), 618-660.
• Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: how to plan, design and conduct effective tests (2. Edition). Indianapolis: Wiley Publishing, Inc.
• Shneiderman, B., & Hochheiser, H. (2001). Universal usability as a stimulus to advanced interface design. Behaviour & Information Technology, 20(5), 367-376.
• Tsironis, A., Katsanos, C., & Xenos, M. (2016). Comparative usability evaluation of three popular MOOC platforms. IEEE Global Engineering Education Conference (EDUCON), 2016 (pp. 608-612). IEEE.
• Wong, B., Nguyen, T. T., Chang, E., & Jayaratna, N. (2003). Usability metrics for e-learning. In Workshop on human computer interface for semantic web and web applications, 2003. Catania, Sicily, Italy: Springer-Verlag Berlin Heidelberg, 235–252.
• Xiao, J., Jiang, B., Xu, Z., & Wang, M. (2014). The usability research of learning resource design for MOOCs. In Teaching, Assessment and Learning (TALE), 2014 International Conference (pp. 277-282). IEEE.
• Yousef, A. M. F., Chatti, M. A., Schroeder, U., & Wosnitza, M. (2015). A usability evaluation of a blended MOOC environment: An experimental case study. The International Review of Research in Open and Distributed Learning, 16(2), 69-93.
• Zaharias, P., & Poylymenakou, A. (2009). Developing a usability evaluation method for e-learning applications: Beyond functional usability. International Journal of Human–Computer Interaction, 25(1), 75-98.