Abstracts
Abstract
Although massive open online courses (MOOCs) have attracted much worldwide attention, scholars still understand little about the specific elements that students find engaging in these large open courses. This study offers a new original contribution by using a machine learning classifier to analyze 24,612 reflective sentences posted by 5,884 students, who participated in one or more of 18 highly rated MOOCs. Highly rated MOOCs were sampled because they exemplify good practices or teaching strategies. We selected highly rated MOOCs from Coursetalk, an open user-driven aggregator and discovery website that allows students to search and review various MOOCs. We defined a highly rated MOOC as a free online course that received an overall five-star course quality rating, and received at least 50 reviews from different learners within a specific subject area. We described six specific themes found across the entire data corpus: (a) structure and pace, (b) video, (c) instructor, (d) content and resources, (e) interaction and support, and (f) assignment and assessment. The findings of this study provide valuable insight into factors that students find engaging in large-scale open online courses.
Keywords:
- MOOCs,
- massive open online courses,
- engagement,
- text mining,
- machine learning
Download the article in PDF to read it. Download
Appendices
Bibliography
-
Altman, N. S. (1992). An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician, 46(3), 175-185. doi: 10.2307/2685209
10.1080/00031305.1992.10475879 Google Scholar Search this bibliographic reference on Google Scholar
-
Anderson, A., Huttenlocher, D., Kleinberg, J., & Leskovec, J. (2014). Engaging with massive online courses. Proceedings of the 23rd international conference on World Wide Web, 687-698. doi: 10.1145/2566486.2568042
10.1145/2566486.2568042 Google Scholar Search this bibliographic reference on Google Scholar
-
Ames, C. (1992). Achievement goals, motivational climate, and motivational processes. In G. Roberts (Ed.), Motivation in Sports and Exercise (pp. 161-176). Champaign, IL: Human Kinetics Books.
Google Scholar Search this bibliographic reference on Google Scholar
-
Baxter, J. A., & Haycock, J. (2014). Roles and student identities in online large course forums: Implications for practice. The International Review of Research in Open and Distributed Learning, 15(1), 20-40. doi: 10.19173/irrodl.v15i1.1593
10.19173/irrodl.v15i1.1593 Google Scholar Search this bibliographic reference on Google Scholar
-
Beldarrain, Y. (2006). Distance education trends: Integrating new technologies to foster student interaction and collaboration. Distance Education, 27(2), 139-153. doi: 10.1080/01587910600789498
10.1080/01587910600789498 Google Scholar Search this bibliographic reference on Google Scholar
-
Bishop, C. M. (2006). Pattern recognition and machine learning. Singapore: Springer.
Google Scholar Search this bibliographic reference on Google Scholar
-
Bonk, C. J., & Wisher, R. A. (2000). Applying collaborative and e-learning tools to military distance learning: A research framework. (Technical Report #1107), US Army Research Institute for the Behavioral and Social Sciences, Alexandria, VA.
10.21236/ada393677 Google Scholar Search this bibliographic reference on Google Scholar
-
Box, G. E. P., & Draper, N. R. (1987). Empirical model building and response surfaces. New York, NY: John Wiley & Sons.
Google Scholar Search this bibliographic reference on Google Scholar
-
Business Wire. (2014). Coursetalk and edX collaborate to integrate online course reviews platform. Retrieved from http://www.businesswire.com/news/home/20140417005360/en#.VCohj2eSx8F
Google Scholar Search this bibliographic reference on Google Scholar
-
Chiu, K. F., & Hew, K. F. (2018). Factors influencing peer learning and performance in MOOC asynchronous online discussion forum. Australasian Journal of Educational Technology, 34(4), 16-28. doi: 10.14742/ajet.3240
10.14742/ajet.3240 Google Scholar Search this bibliographic reference on Google Scholar
-
Coetzee D., Fox, A., Hearst, M.A., & Hartmann, B. (2014). Should your MOOC forum use a reputation system? In Proceedings of CSCW 2014, ACM Press, 1176-1187. doi: 10.1145/2531602.2531657
10.1145/2531602.2531657 Google Scholar Search this bibliographic reference on Google Scholar
-
Coffrin, C., de Barba, P., Corrin, L., & Kennedy, G. (2014). Visualizing patterns of student engagement and performance in MOOCs. In Proceedings of LAK 2014, ACM Press (2014), 83-92. doi: 10.1145/2567574.2567586
10.1145/2567574.2567586 Google Scholar Search this bibliographic reference on Google Scholar
-
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. doi: 10.1177/001316446002000104
Google Scholar Search this bibliographic reference on Google Scholar
-
Conole, G. (2013). MOOCs as disruptive technologies: Strategies for enhancing the learner experience and quality of MOOCs. RED, Revista de Educación a Distancia. Número 39. doi: 10.6018/red/50/2
10.6018/red/50/2 Google Scholar Search this bibliographic reference on Google Scholar
-
Cortes, C., & Vapnik, V. (1995). Support vector machine. Machine Learning, 20(3), 273-297. Retrieved from http://image.diku.dk/imagecanon/material/cortes_vapnik95.pdf
Google Scholar Search this bibliographic reference on Google Scholar
-
Cox, D. R. (1958). The regression analysis of binary sequences. Journal of the Royal Statistical Society. Series B (Methodological), 215-242. Retrieved from https://www.jstor.org/stable/2983890?seq=1#page_scan_tab_contents
Google Scholar Search this bibliographic reference on Google Scholar
-
Daniel, J. (2012). Making sense of MOOCs: Musings in a maze of myth, paradox, and possibility. Journal of Interactive Media in Education, 3. doi: 10.5334/2012-18
10.5334/2012-18 Google Scholar Search this bibliographic reference on Google Scholar
-
de Barba, P. G., Kennedy, G. E., & Ainley, M. D. (2016). The role of students' motivation and participation in predicting performance in a MOOC. Journal of Computer Assisted Learning, 32, 218-231. doi: 10.1111/jcal.12130
Google Scholar Search this bibliographic reference on Google Scholar
-
Dillahunt, T. R., Wang, B. Z., & Teasley, S. (2014). Democratizing higher education: Exploring MOOC use among those who cannot afford a formal education. International Review of Research in Open and Distance Learning, 15(5), 177-196. doi: 10.19173/irrodl.v15i5.1841
10.19173/irrodl.v15i5.1841 Google Scholar Search this bibliographic reference on Google Scholar
-
Elliot, A. J., & Church, M. A. (1997). A hierarchical model of approach and avoidance achievement motivation. Journal of Personality and Social Psychology, 72, 218-232. doi: 10.1037/0022-3514.72.1.218
10.1037//0022-3514.72.1.218 Google Scholar Search this bibliographic reference on Google Scholar
-
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. (2004). School engagement: Potential of the concept: State of the evidence. Review of Educational Research, 74, 59-119. doi: 10.3102/00346543074001059
Google Scholar Search this bibliographic reference on Google Scholar
-
Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. Annals of statistics, 1189-1232. Retrieved from https://statweb.stanford.edu/~jhf/ftp/trebst.pdf
Google Scholar Search this bibliographic reference on Google Scholar
-
Gore, H. (2014). Massive Open Online Courses (MOOCs) and their impact on academic library services: Exploring the issues and challenges. New Review of Academic Librarianship, 20(1), 4-28. doi: 10.1080/13614533.2013.851609
10.1080/13614533.2013.851609 Google Scholar Search this bibliographic reference on Google Scholar
-
Guo, P. J., Kim, J., & Rubin, R. (2014). How video production affects student engagement: An empirical study of MOOC videos. In Proceedings of the first ACM conference on Learning@ scale conference (pp. 41-50). ACM. doi: 10.1145/2556325.2566239
10.1145/2556325.2566239 Google Scholar Search this bibliographic reference on Google Scholar
-
Hazari, S., North, A., & Moreland, D. (2009). Investigating pedagogical value of wiki technology. Journal of Information Systems Education, 20(2), 187-198. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.476.714&rep=rep1&type=pdf
Google Scholar Search this bibliographic reference on Google Scholar
-
Hew, K. F. (2015). Towards a model of engaging online students: Lessons from MOOCs and four policy documents. International Journal of Information and Education Technology, 5(6), 425-431. doi: 10.7763/IJIET.2015.V5.543
10.7763/ijiet.2015.v5.543 Google Scholar Search this bibliographic reference on Google Scholar
-
Hew, K. F. (2016). Promoting engagement in online courses: What strategies can we learn from three highly rated MOOCs. British Journal of Educational Technology, 47(2), 320-341. doi: 10.1111/bjet.12235
10.1111/bjet.12235 Google Scholar Search this bibliographic reference on Google Scholar
-
Hew, K. F., & Cheung, W. S. (2014). Students' and Instructors' Use of Massive Open Online Courses (MOOCs): Motivations and Challenges. Educational Research Review, 12, 45-58. https://doi.org/10.1016/j.edurev.2014.05.001
10.1016/j.edurev.2014.05.001 Google Scholar Search this bibliographic reference on Google Scholar
-
Kearsley, G., & Shneiderman, B. (1998). Engagement theory: A framework for technology-based teaching and learning. Educational technology, 38(5), 20-23. Retrieved from https://www.jstor.org/stable/44428478?seq=1#page_scan_tab_contents
Google Scholar Search this bibliographic reference on Google Scholar
-
Keller, J. M. (1987). Development and use of the ARCS model of instructional design. Journal of Instructional Development, 10(3), 2-10. doi: 10.1007/BF02905780
Google Scholar Search this bibliographic reference on Google Scholar
-
Kizilcec, R., Piech, C., & Schneider, E. (2013). Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 170-179). New York, NY: ACM. doi: 10.1145/2460296.2460330
10.1145/2460296.2460330 Google Scholar Search this bibliographic reference on Google Scholar
-
Kloft, M., Stiehler, F., Zheng, Z., & Pinkwart,N. (2014). Predicting MOOC dropout over weeks using machine learning models. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 60-65. Association for Computational Linguistics. Retrieved from http://www.aclweb.org/anthology/W/W14/W14-41.pdf#page=67
10.3115/v1/w14-4111 Google Scholar Search this bibliographic reference on Google Scholar
-
Knowles, M. S., Holton, E. F., & Swanson, R. A. (2011). The adult learner: The definitive classic in adult education and human resource development. (7th ed.). New York: Elsevier Inc.
Google Scholar Search this bibliographic reference on Google Scholar
-
Knowlton, D. S. (2000). A theoretical framework for the online classroom: A defense and delineation of a student‐centered pedagogy. New Directions for Teaching and Learning, 2000(84), 5-14. doi: 10.1002/tl.841
Google Scholar Search this bibliographic reference on Google Scholar
-
Kolowich, S. (2013). The professors who make the MOOCs. Chronicle of Higher Education, 59(28), A20-A23. Retrieved from http://publicservicesalliance.org/wp-content/uploads/2013/03/The-Professors-Behind-the-MOOC-Hype-Technology-The-Chronicle-of-Higher-Education.pdf
Google Scholar Search this bibliographic reference on Google Scholar
-
Lombardi, M. M. (2013). The inside story: Campus decision making in the wake of the latest MOOC Tsunami. MERLOT Journal of Online Learning and Teaching, 9(2), 239-248. Retrieved from https://search.proquest.com/docview/1500421747?pq-origsite=gscholar
Google Scholar Search this bibliographic reference on Google Scholar
-
McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276-282. doi: 10.11613/BM.2012.031
10.11613/bm.2012.031 Google Scholar Search this bibliographic reference on Google Scholar
-
Meece, J. L., Blemenfeld, P. C., & Hoyle, R. H. (1988). Students' goal orientations and cognitive engagement in classroom activities. Journal of Educational Psychology, 80, 514-523. doi: 10.1037/0022-0663.80.4.514
10.1037//0022-0663.80.4.514 Google Scholar Search this bibliographic reference on Google Scholar
-
Murphy, K. P. (2012). Machine learning: a probabilistic perspective. Cambridge, MA: MIT press.
Google Scholar Search this bibliographic reference on Google Scholar
-
Robertson, S. (2004). Understanding inverse document frequency: On theoretical arguments for IDF. Journal of Documentation, 60(5), 503-520. doi: 10.1108/00220410410560582
10.1108/00220410410560582 Google Scholar Search this bibliographic reference on Google Scholar
-
Rodriguez. C. O. (2012). MOOCs and the AI-Stanford like courses: Two successful and distinct course formats for massive open online courses. European Journal of Open, Distance and E-Learning. Retrieved from http://www.eurodl.org/index.php?p=archives&year=2013&halfyear=2&article=516
Google Scholar Search this bibliographic reference on Google Scholar
-
Shah, D. (2016, December 25). By the numbers: MOOCS in 2016. Retrieved from https://www.class-central.com/report/mooc-stats-2016/
Google Scholar Search this bibliographic reference on Google Scholar
-
Shenton, A. K. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for Information, 22, 63-75. doi: 10.3233/EFI-2004-22201
10.3233/efi-2004-22201 Google Scholar Search this bibliographic reference on Google Scholar
-
Sims, R. (2003). Promises of interactivity: Aligning learner perceptions and expectations with strategies for flexible and online learning. Distance Education, 24(1), 87-103. doi: 10.1080/01587910303050
10.1080/01587910303050 Google Scholar Search this bibliographic reference on Google Scholar
-
Stehman, S. V. (1997). Selecting and interpreting measures of thematic classification accuracy. Remote sensing of Environment, 62(1), 77-89. doi: 10.1016/S0034-4257(97)00083-7
10.1016/s0034-4257(97)00083-7 Google Scholar Search this bibliographic reference on Google Scholar
-
Szpunar, K. K., Khan, N. Y., & Schacter, D. L. (2013). Interpolated memory tests reduce mind wandering and improve learning of online lectures. Proceedings of the National Academy of Sciences, 110(16), 6313-6317. doi: 10.1073/pnas.1221764110
10.1073/pnas.1221764110 Google Scholar Search this bibliographic reference on Google Scholar
-
Szpunar, K. K., Jing, G. H., & Schacter, D. L. (2014). Overcoming overconfidence in learning from video-recorded lectures: Implications of interpolated testing for online education. Journal of Applied Research in Memory and Cognition, 3(3), 161-164. doi: 10.1016/j.jarmac.2014.02.001
10.1016/j.jarmac.2014.02.001 Google Scholar Search this bibliographic reference on Google Scholar
-
Wanzer, M. B. (2002). Use of humor in the classroom: The good, the bad, and the not-so-funny things that teachers say and do. In J. L. Chesebro & J. C. McCroskey (Eds.), Communication for Teachers (pp. 116-125). Boston: Allyn & Bacon.
Google Scholar Search this bibliographic reference on Google Scholar
-
Warren, J., Rixner, S., Greiner, J. & Wong, S. (2014). Facilitating human interaction in an online programming course, in Proceedings of SIGCSE 2014, ACM Press, 665-670. doi: 10.1145/2538862.2538893
Google Scholar Search this bibliographic reference on Google Scholar
-
Wold, S., Esbensen, K., & Geladi, P. (1987). Principal component analysis. Chemometrics and Intelligent Laboratory Systems, 2(1-3), 37-52. doi: 10.1016/0169-7439(87)80084-9
10.1016/0169-7439(87)80084-9 Google Scholar Search this bibliographic reference on Google Scholar
-
Wu, H. C., Luk, R. W. P., Wong, K. F., & Kwok, K. L. (2008). Interpreting tf-idf term weights as making relevance decisions. ACM Transactions on Information Systems (TOIS), 26(3), 13. doi: 10.1145/1361684.1361686
10.1145/1361684.1361686 Google Scholar Search this bibliographic reference on Google Scholar
-
Young, J. R. (2013). What professors can learn from "hard core" MOOC students. Chronicle of Higher Education, 59(37), A4. Retrieved from https://www.chronicle.com/article/What-Professors-Can-Learn-From/139367
Google Scholar Search this bibliographic reference on Google Scholar

