Comparative Study SPOC vs. MOOC for Socio-Technical Contents from Usability and User Satisfaction

Year 2019, , 4 - 20, 01.04.2019

Giovani Lemos De Carvalho Junıor Daniel Cebrian Robles Manuel Cebrian De La Serna Manuela Raposo Rıvas

https://doi.org/10.17718/tojde.557726

Cited By: 2

Abstract

According to literature, usability and user satisfaction directly influence learning achievement. However, few studies address those elements in MOOCs (massive open online courses). Relevant questions such as “Are drop-out rates similar between SPOCs (small private open courses) and MOOCs?” and “Are there significant usability and satisfaction differences between both formats?” may help improve initiatives and dissemination policies regarding open knowledge. This study aims to upgrade guidelines for the conversion of SPOCs into MOOCs, by exploring usability and satisfaction in both formats. A quantitative research based on a questionnaire was designed, and the validated SUSE (Satisfaction and Usability of Software in Education) instrument was applied to 7 courses of each type, with the same contents, and 5,192 students. The results revealed validity and reliability, with higher satisfaction scores on the open platform. Genders, regions of residence, and age groups also provided interesting findings: students are predominantly male, young, and from diverse geographic locations. Women, the younger, and residents of economically disadvantaged regions evaluated their experiences more positively. Both formats present significantly lower dropout rates (47.50% in MOOCs and 36.94% in SPOCs) and higher completion rates (52.50% in MOOCs and 63.06% in SPOCs) compared to those generally observed in free enrollment courses.

Keywords

MOOC, SPOC, usability, satisfaction, use studies, comparative studies

References

• Aleman de la Garza, L. Y., Sancho-Vinuesa, T., & Gómez Zermeno, M. G. (2015). Atypical: Analysis of a massive open online course (MOOC) with a terminal efficiency of 22.35%. Global Education Review, 2(3). 68-81. Retrieved from: http://ger.mercy.edu/index.php/ger/article/view/127 Alraimi, K. M., Zo, H., & Ciganek, A. P. (2015) Understanding the MOOCs continuance: the role of openness and reputation. Computers & Education, 80, 28-38. https://doi.org/10.1016/j.compedu.2014.08.006 Barak, M., Watted, A., & Haick, H. (2016) Motivation to learn in massive open online courses: Examining aspects of language and social engagement. Computers & Education, 94, 49-60. https://doi.org/10.1016/j.compedu.2015.11.010 Barba, P. G., Kennedy, G. E., & Ainley, M. D. (2016). The role of students' motivation and participation in predicting performance in a MOOC. Journal of Computer Assisted Learning, 32(3), 218-231. https://doi.org/10.1111/jcal.12130 Bevan, N. (1998). Quality and usability: A new framework. In Van-Veenendaal, E. & MCMullan, J. (Eds.), Achieving Software Product Quality. Netherlands: Tutein Nolthenius, 25-34. Retrieved from: https://goo.gl/Jtb71V Bozkurt, A., Akgun-Ozbek, E., & Zawacki-Richter, O. (2017). Trends and Patterns in Massive Open Online Courses: Review and Content Analysis of Research on MOOCs (2008-2015). International Review of Research in Open and Distributed Learning, 18(5), 118-147. https://doi.org/10.19173/irrodl.v18i5.3080 Canas-Bajo, J., Leikas, J., Jokinen, J., Canas, J. J., & Saariluoma, P. (2016). How older and younger people see technology in Northern and Southern Europe: Closing the generation gap. Gerontechnology 2016, 14(2), 110-117, https://doi.org/10.4017/gt.2016.14.2.010.00 Cabero-Almenara, J., & Llorente-Cejudo, M.d.C. (2017). Los MOOC: encontrando su camino. @tic. Revista d'innovació educativa, 18, 24-30. https://doi.org/10.7203/attic.18.9928 Chen, P.-S. D., Lambert, A. D., & Guidry, K. R. (2010). Engaging online learners: The impact of Web-based learning technology on college student engagement. Computers & Education, 54(4), 1222-1232. https://doi.org/10.1016/j.compedu.2009.11.008 Christensen, G., Steinmetz, A., Alcorn, B., Bennett, A., Woods, D., & Emanuel, E. J. (2013). The MOOC phenomenon: Who takes massive open online courses and why? SSRN Electronic Journal. (November 6, 2013). https://doi.org/10.2139/ssrn.2350964 Clow, D. (2013). MOOCs and the funnel of participation. In: Proceedings of the third international conference on learning analytics and knowledge (LAK 2013), Leuven, Belgium, April 8-12, 185-189. https://doi.org/10.1145/2460296.2460332 Cole, A. W., & Timmerman, C. E. (2015). What do current college students think about MOOCs? MERLOT Journal of Online Learning and Teaching, 11(2), 188-201. Retrieved from: https://goo.gl/oWTUVT Conole, G. (2013) Designing for learning in an open world. In Spector, M. & Lajoie, S. (Eds.), Explorations in the learning sciences, Instructional systems and performance technologies:, 4. https://doi.org/10.1007/978-1-4419-8517-0 Daradoumis, T., Bassi, R., Xhafa, F., & Caballe, S. (2013). A review on massive e-learning (MOOC) design, delivery and assessment. In Proceedings of the 2013 Eighth International Conference on P2P, Parallel, Grid, Cloud and Internet Computing (3PGCIC), 208-213. https://doi.org/10.1109/3PGCIC.2013.37 Djamasbi, S., Tullis, T., Hsu, J., Mazuera, E., Osberg, K., & Bosch, J. (2007). Gender preferences in web design: usability testing through eye tracking. In Proceedings of the 13th Americas Conference on Information Systems (AMCIS 2007), Keystone, Colorado, August, 1-8. Retrieved from: https://goo.gl/qCg4h7 Emanuel, E. J. (2013) MOOCs taken by educated few. Nature, 503, 342. (November 21, 2013). https://doi.org/10.1038/503342a Gamage, D., Fernando, S., & Perera, I. (2015). Factors leading to an effective MOOC from participants perspective. In 8th International Conference on Ubi-Media Computing, UMEDIA. Article number 7297460, 230-235.
• Garrido, M., Koepke, L., Andersen, S., Mena, A., Macapagal, M., & Dalvit, L. (2016). An examination of MOOC usage for professional workforce development outcomes in Colombia, the Philippines, & South Africa. Seattle: Technology & Social Change Group, University of Washington Information School. Glass, C. R., Shiokawa-Baklan, M. S., Saltarelli, A. J. (2016). Who takes MOOCs?: New Directions for Institutional Research. MOOCS and Higher Education: Implications for Institutional Research, 41-55. Gregory, R. L. (1998). Eye and brain: The psychology of seeing (5th ed.). New York, NY: Oxford University Press. Harrati, N., Bouchrikab, I., Taria, A., & Ladjailiab, A. (2016). Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis. Computers in Human Behavior, 61, 463-471. https://doi.org/10.1016/j.chb.2016.03.051 Hassan, Y., Martín Fernandez, F. J., & Iazza, G. (2004). Diseno web centrado en el usuario: usabilidad y arquitectura de la información. Hipertext.net, (2). Retrieved from: https://goo.gl/f8q3sG Ho, A. D., Reich, J., Nesterko, S., Seaton, D. T., Mullaney, T., Waldo, J., & Chuang, I. (2014). Harvard and MITx: The first year of open online courses. HarvardX and MITx Working Paper No. 1. Retrieved from: https://goo.gl/XHj6Uc Hood, N. E., & Littlejohn, A. (2016). MOOC Quality: The need for new measures. Journal for Learning for Development, 3(3), 28-42. https://goo.gl/djsu7x Ivanova, A., & Smirkarov, A. (2009). The new generations of students and the future of e-learning in higher education. In International Conference on e-Learning and the Knowledge Society - e-Learning’09. Berlin, Germany, August 31-September 01, 2009, 17-25. Retrieved from: https://goo.gl/CRTr5J Jordan, K. (2015). Massive open online course completion rates revisited: Assessment, length and attrition. International Review of Research in Open and Distributed Learning, 16(3), 341-358. https://doi.org/10.13140/RG.2.1.2119.6963 Katsanos, C., Tselios, N., & Xenos, M. (2012). Perceived usability evaluation of learning management systems: A first step towards standardization of the system usability scale in Greek. In Informatics (PCI), 16th Panhellenic Conference. IEEE, 302-307. https://doi.org/10.1109/PCi.2012.38 Khalil, H., & Ebner, M. (2014). MOOCs completion rates and possible methods to improve retention - A literature review. In Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2014, 1305-1313, Tampere, Finland: Association for the Advancement of Computing in Education (AACE). Retrieved from https://goo.gl/PcYAh5 Kimura, D. (1992). Sex Differences in the Brain. Scientific American, 267(3), 118-125. https://doi.org/10.1038/scientificamerican0992-118 Koller, D., Ng, A., Do, C., & Chen, Z. (2013). Retention and intention in massive open online courses: In depth. Educause Review, 48(3), 62-63. Retrieved from: https://goo.gl/qtSmnW Kumar, K. (2016). Emotional intelligence and achievement motivation: A correlation study. Indian Journal of Health and Wellbeing, 7(5), 546-549. Retrieved from: https://goo.gl/yYTPZA LeBar, M. (2014). MOOCs: Completion is not important. Forbes. (September 16, 2014). Retrieved from: https://goo.gl/2FBzNR Lin, C. J., & Hsieh, T-L. (2016). Exploring the design criteria of website interfaces for gender. International Journal of Industrial Ergonomics, 53, 306-311. https://doi.org/10.1016/j.ergon.2016.02.002 Liyanagunawardena, T. R., Adams, A. A., & Williams, S. A. (2013). MOOCs: a systematic study of the published literature 2008-2012. The International Review of Research in Open and Distance Learning, 14(3), 202-227. https://doi.org/10.19173/irrodl.v14i3.1455 Loureiro, A., & Messias, I. (2016). Competences and learning profiles of digital age's students. In Pinheiro, M. & Simões, D. (Eds.) Handbook of Research on Engaging Digital Natives in Higher Education Settings. Hershey, PA: Information Science Reference, 171-191. https://doi.org/10.4018/978-1-5225-0039-1.ch008 Monedero-Moya, J. J., Cebrian-Robles, D., & Desenne, P. (2015). Usability and satisfaction in multimedia annotation tools for MOOCs. Comunicar, 44, 55-62. https://doi.org/10.3916/C44-2015-06 Moss, G., Gunn, R., & Heller, J. (2006). Some men like it black, some women like it pink: Consumer implications of differences in male and female website design. Journal of Consumer Behavior, 5(4), July 2006, 328-341. https://doi.org/10.1002/cb.184 Online Course Report. (2017). State of the MOOC 2017: A Year of Privatized and Open Education Growth, Retrieved from: https://goo.gl/h2t5gT Orfanou, K., Tselios, N., & Katsanos, C. (2015). Perceived Usability Evaluation of Learning Management Systems: Empirical Evaluation of the System Usability Scale. International Review of Research in Open and Distributed Learning, 16(2). https://doi.org/10.19173/irrodl.v16i2.195 Palin, A. (2014). MOOCs: Young students from developing countries are still in the minority. Financial Times. (March 9, 2014). Retrieved from: https://goo.gl/xEQ9t0 Petrie, H., & Bevan, N. (2009). The evaluation of accessibility, usability and user experience. In Stepanidis, C. (Ed.) The Universal Access Handbook, CRC Press. https://doi.org/10.1201/9781420064995-c20 Read, T., Barcena, E., & Sedano, B. (2018) Current Trends in MOOC Research and Applications. In D. Jansen; L. Konings (Eds.) The 2018 OpenupEd Trend Report on MOOCs, 10-14. Maastricht, NL: EADTU. Retrieved from https://tinyurl.com/2018OpenupEdtrendreport Reich, J. (2014). MOOC completion and retention in the context of student intent. EDUCAUSE Review. (December 8, 2014). Retrieved from: https://goo.gl/7uVXDS Reich, J. (2015). Rebooting MOOC research. Science, 347(6217), 34-35. https://doi.org/10.1126/science.1261627 Rodriguez, R. L., Ramos, J. L. C., Silva, J. C. S., & Gomes, A. S. (2016). Discovery engagement patterns MOOCs through cluster analysis. Revista IEEE America Latina, 14, 4129-4135. https://doi.org/10.1109/TLA.2016.7785943 Sauro, J. (2011). Measuring Usability with the System Usability Scale (SUS). Retrieved from: https://goo.gl/iotQ5X Serrano-Angulo, J., & Cebrian-Robles, D. (2012). Usabilidad de la e-rúbrica mediante cuestionarios online con limesurvey. In C. Leite & M. Zabalza (Eds.), Ensino Superior: Inovação e qualidade na docência (pp. 467–486). Porto, Portugal: CIIE – Centro de Investigação e Intervenção Educativas. Serrano-Angulo, J., & Cebrian-Robles, D. (2014). Usabilidad y satisfacción de la e-Rúbrica. REDU. Revista de Docencia Universitaria, 12(1), 177-195. https//doi.org/10.4995/redu.2014.6426 Scanlon, E., McAndrew, P., & O’Shea, T. (2015). Designing for educational technology to enhance the experience of learners in distance education: How open educational resources, learning design and MOOCs are influencing learning. Journal of Interactive Media in Education, 6(1), 1-9. https://doi.org/10.5334/jime.al Shah, D. (2018). By the numbers: MOOCs in 2017: How has the MOOC space grown this year? Get the facts, figures, and pie charts. Retrieved from: https://goo.gl/f5Dsp5 Stokes, T. A., Guillan, D. J., & Braden, J. P. (2016). Establishing the link between usability and student satisfaction in adaptive online learning. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 60(1), 1976-1980. https://doi.org/10.1177/1541931213601450 Tedesco, D., Chadwick-Dias, A., & Tullis, T. (2004). Demographic differences in preferred web site content. Aging by design presentation at Bentley College, Waltham, Massachusetts. Retrieved from: https://goo.gl/HkooJ7 Teixeira, A., Mota, J., Garcia-Cabot, A., García-Lopez, E., & de-Marcos, L. (2016). A new competence-based approach for personalizing MOOCs in a mobile collaborative and networked environment. Revista Iberoamericana de Educación. 19(1), 143-160. https://doi.org/10.5944/ried.19.1.14578 Tsironis, A., Katsanos, C., & Xenos, M. (2016). Comparative usability evaluation of three popular MOOC platforms. In Global Engineering Education Conference (EDUCON), 2016 IEEE (pp. 608-612). https://doi.org/10.1109/EDUCON.2016.7474613 Whitehill, J., Mohan, K., Rosen, Y., Seaton, D. T., Tingley, D. (2017). Delving deeper into MOOC student dropout prediction. CoRR, arXiv:1702.06404v1. https://doi.org/10.1145/1235 Yan, A., Lee, M. J., & Ko, A. J. (2017). Predicting abandonment in online coding tutorials. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), Raleigh. Yousef, A. M. F., Chatti, M. A., Schroeder, U., & Wosnitza, M. (2015). A usability evaluation of a blended MOOC environment: An experimental case study. The International Review of Research in Open and Distributed Learning, 16(2) https://doi.org/10.19173/irrodl.v16i2.2032