Improving STEM MOOC evaluation requires an understanding of the current state of STEM MOOC evaluation, as perceived by all stakeholders. To this end, we investigated what kinds of information STEM MOOC instructors currently use to evaluate their courses and what kinds of information they feel would be valuable for that purpose. We conducted semi-structured interviews with 14 faculty members from a variety of fields and research institutions who had taught STEM MOOCs on edX, Coursera, or Udacity. Four major themes emerged related to instructors' desires: (1) to informally assess learners as an instructor might in a traditional classroom, (2) to assess learners’ attainment of personal learning goals, (3) to obtain in-depth qualitative feedback from learners, and (4) to access more detailed learner analytics regarding the use of course materials. These four themes contribute to a broader sentiment expressed by the instructors that they have access to a wide variety of quantitative data for use in evaluation, but are largely missing the qualitative information that plays a significant role in traditional evaluation. Finally, we provide our recommendations for MOOC evaluation criteria, based on these findings.
- MOOC Instructor
Download the article in PDF to read it.
- Baker, C. (2010). The impact of instructor immediacy and presence for online student affective learning, cognition, and motivation. The Journal of Educators Online, 7 (1).
- Davidson, E. J. (2005). Evaluating methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: SAGE Publications.
- Diamond, L. (2017, January 11). Online master of science in analytics degree to be offered for less than $10,000. Georgia Tech News Center. Retrieved from http://www.news.gatech.edu/2017/01/11/online-master-science-analytics-degree-be-offered-less-10000
- Douglas, K. A., Diefes-Dux, H. A., Bermel, P., Madhavan, K., Hicks, N. M., & Williams, T. V. (2017, June). Board# 32: NSF PRIME Project: Contextualized Evaluation of Advanced STEM MOOCs. In 2017 ASEE Annual Conference & Exposition. doi: 10.18260/1-2--27830
- Evans, S., & Myrick, J. G. (2015). How MOOC instructors view the pedagogy and purposes of massive open online courses. Distance Education, 36(3), 295-311. doi: 10.1080/01587919.2015.1081736
- Haavind, S., & Sistek-Chandler, C. (2015). The emergent role of the MOOC instructor: A qualitative study of trends toward improving future practice. International Journal on E-Learning, 14(3), 331-350.
- Hew, K. F., & Cheung, W. S. (2014). Students' and instructors' use of massive open online courses (MOOCs): Motivations and challenges. Educational Research Review, 12, 45-58. doi: 10.1016/j.edurev.2014.05.001
- Hollands, F. M., & Tirthali, D. (2014). MOOCs: Expectations and reality. Full report. Center for Benefit-Cost Studies of Education, Teacher's College, Columbia University, NY. Retrieved from https://files.eric.ed.gov/fulltext/ED547237.pdf
- Knox, J., Ross, J., Sinclair, C., Macleod, H., & Bayen, S. (2014). MOOC feedback: Pleasing all the people. In S. D. Krause & C. Lowe (Eds.), Invasion of the MOOCs: The promises and perils of massive open online courses (pp. 98-104). Anderson, South Carolina: Parlor Press.
- Koller, D., Ng, A., & Chen, Z. (2013). Retention and intention in massive open online courses: In depth. Educase Review, 48(3), 62-63.
- Liyanagunawardena, T. R., Lundqvist, K. Ø., & Williams, S. A. (2015). Who are with us: MOOC learners on a FutureLearn course. British Journal of Educational Technology, 46(3), 557-569. doi: 10.1111/bjet.12261
- London, J., & Young, C. (2016). The role of massive open online courses (MOOCs) in engineering education: Faculty perspectives on its potential and suggested research directions. International Journal of Engineering Education, 32(4), 1788-1800.
- Najafi, H., Rolheiser, C., Harrison, L., & Håklev, S. (2015). University of Toronto instructors' experiences with developing MOOCs. International Review of Research in Open and Distance Learning, 16(3), 233-255.
- National Academy of Engineering. (2004). The engineer of 2020: Visions of engineering in the new century. Washington, DC: The National Academies Press. doi: 10.17226/10999
- National Science and Technology Council, Committee on Technology, Subcommitee on Nanoscale Science, Engineering, and Technology, National Nanotechnology Initiative. (2008). Strategy for nanotechnology-related environmental, health, and safety research. Arlington, Virginia: National Nanotechnology Coordination Office. Retrieved from https://www.nano.gov/sites/default/files/pub_resource/nni_ehs_research_strategy.pdf
- Patton, M. Q. (2002). Qualitative evaluation and research methods (3rd ed.). Thousand Oaks, CA: SAGE Publications.
- Rubin, H. J., & Rubin, I. S. (2005). Qualitative interviewing: The art of hearing data (2nd ed.). Thousand Oaks, CA: SAGE Publications.
- Scriven, M. (1983). Evaluation ideologies. In D. L. Stufflebeam, G. F. Madaus, & T. Kellaghan (Eds.), Evaluation models: Viewpoints on educational and human services evaluation (pp. 258-260). Springer Science & Business Media.
- Scriven, M. (1991). Evaluation thesaurus (4th ed.). Thousand Oaks, CA: SAGE Publications.
- Scriven, M. (2015, August 15). Key evaluation checklist. Western Michigan University Evaluation Center. Retrieved from http://www.michaelscriven.info/images/MS_KEC_8-15-15.doc
- Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation: Theories of practice (Revised). Thousand Oaks, CA: SAGE Publications.
- Shah, D. (2016, December 25). By the numbers: MOOCs in 2016. Class Central. Retrieved from https://www.class-central.com/report/mooc-stats-2016/
- Stephens-Martinez, K., Hearst, M. A., & Fox, A. (2014). Monitoring MOOCs: Which Information Sources Do Instructors Value? In Proceedings of the first ACM conference on Learning @ scale conference (pp. 79-88). ACM. doi: 10.1145/2556325.2566246
- UNESCO. (2016). Education 2030: Incheon Declaration and Framework for Action for the implementation of Sustainable Development Goal 4. UNESCO Digital Library. Retrieved from https://unesdoc.unesco.org/ark:/48223/pf0000245656.
- Zheng, S., Rosson, M. B., Shih, P. C., & Carroll, J. M. (2015). Understanding student motivation, behaviors and perceptions in MOOCs. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 1882-1895). Vancouver, CA: ACM. doi: 10.1145/2702613.2702628