A newly published, two-year assessment of the massive open online courses (MOOCs) created by HarvardX and MITx through their edX online-learning partnership describes in depth the evolving features of 68 courses offered by the two institutions. The analysis, conducted by the HarvardX Research Committee and MIT’s Office of Digital Learning, delivers three distinct, valuable kinds of insights into the field:
- a fuller, enriched characterization of MOOCs, whose users have evolved significantly as more courses have become available;
- hints of emerging advanced research that may provide insights into the effectiveness of different teaching approaches and the actual learning achieved by MOOC users; and
- the applications of these inquiries and their insights to on-campus, classroom courses. (Improvements in residential pedagogy and learning have been part of the agenda since edX was launched nearly three years ago, but have been somewhat overshadowed in the hoopla surrounding the debut of the online courses themselves.)
From “Enrollment” to Participation and Certification
The report aggregates data through last September 21 on 1.7 million “course entries” in the 68 MOOCs studied (involving about 1 million unique participants, who on average engage with 1.7 courses each; some 300,000 unique participants account for almost 1 million of those total course counts), encompassing 10 million participant-hours (about six hours per participant) and 1.1 billion logged events (diverse interactions with the courses, materials, etc.).
The basic census of MOOC users has evolved in a revealing way: through this February, HarvardX was using course registrants, then numbering 2 million, as its leading indicator of reach. In the HarvardX monthly newsletter for March, the metric shifted to participants (then 980,000), with this distinction:
By registrants, we mean those who have expressed interest in the course by enrolling [a simple, online procedure], but have not [taken] any subsequent action. Participants have engaged in some way with course material (i.e., by logging into the course at lease once). In a typical course, there are about twice as many registrants as participants.
The researchers highlight several evolving trends.
• A new normal. The initial excitement about MOOCs may be calming down a bit. For the 11 courses studied that offered repeat versions, participation declined 43 percent, on average, from the first to the second offering. Moreover, given the proliferation of course offerings, “course instructors and administrators who hope to reach the relatively high numbers of participants from our Year 1 data should not expect to do so without a strategy for recruitment and visibility.” (And of course the universe is much larger than Harvard and MIT: edX currently lists 468 courses from all its participating institutions.)
Moreover, in focusing on participants—people who engage in some way with a course after initial registration—the researchers make two observations about “non-entrants,” who register but never subsequently do anything in the course:
- they numbered 1.3 million during the two years under study; and
- the proportion of registrants who never engage (never participate in the course) rose from 37 percent in the first year to 47 percent in the second.
Among those who do engage, “There is…a moderate increase in the percentage of explorers who access half or more of course chapters, from 14 percent to 19 percent.”
• Questions of intent. About one-third of participants responded to a survey about their intentions in taking a course; of these, 57 percent said they intended to earn a certificate of completion, and 24 percent of these respondents did so. As for the 43 percent of respondents who were unsure or did not intend to earn a certificate, just 8 percent did earn the certificate.
This is, of course, a fundamental difference between a free MOOC and a course taken on campus as part of a structured educational program leading to a degree: many MOOC participants are, in the researchers’ language, “uninterested in certification.”
• Teacher training. Among the one-fifth of participants who answered survey questions about past experience as a professional teacher or instructor, 39 percent identified themselves as past or present teachers—suggesting a large pool of educators, whose use of MOOCs may mean that the online courses’ reach is amplified in the participants’ own classrooms or course plans.
Discussing the report, Peter K. Bol, Carswell professor of East Asian languages and civilizations (who developed the 10-part ChinaX course with William F. Kirby, Chang professor of China studies and Spangler Family professor of business administration), offered an illustrative anecdote about this “knock-on effect.” A Chinese history professor of his acquaintance at another institution is using ChinaX as, in effect, the text for her course: her students view the lecture segments, but don’t do the online assessments; and then in class, they pursue directed, in-depth discussions of selected topics. Such adoptions of HarvardX material, he said, have “multiplied manyfold” the pedagogical reach of the online contents.
(As a side note, such students would access the course by registering for ChinaX on the edX platform; because they do not complete all the online assignments and assessments, they would be tallied as “participants” but not among those who become “certified” as having completed the course. In his role as vice provost for advances in learning, overseeing HarvardX and other pedagogical initiatives, Bol keeps a close eye on such data. [Editor’s note: Bol is also a director of Harvard Magazine Inc.])
• Computer science at the summit. The researchers divided the courses into four intellectual categories: computer science; science, technology, engineering, and mathematics (STEM); humanities, history, religion, design, and education; and government, and health and social sciences. (These categories, based on the HarvardX and MITx courses offered, do not align precisely with a classic liberal-arts division such as humanities, social sciences, natural sciences—the last, in turn, perhaps further divided into biological and physical sciences—and engineering and applied sciences.) Much of the initial impetus for MOOCs at MIT—and at for-profit MOOC enterprises such as Coursera and Udacity—came from computer science, and this review of HarvardX and MITx courses shows much larger participation in computer-science courses: 68,000 versus 19,000 for offerings in the other disciplinary areas. As the report notes, “the standout enrollments of CS courses…continue to put the Massive in MOOCs more than other curricular areas.” (In fairness, of course, a professor whose HarvardX humanities course reaches thousands of online learners, for instance, may thereby teach the equivalent of an entire career’s worth of enrollments in her campus-based seminar and small-lecture classes.)
Interestingly, computer-science and STEM students had certification (course-completion) rates of only about half those recorded by students studying MOOCs in the other fields.
• Willingness to pay. Participants can pay $25 to $250 to verify their identities for their coursework and certificates (the report covered 13 course offerings that enabled verification). A survey of those who do so reveals that they are slightly older, more educated, more likely to be from the United States, and a higher proportion of males than students who elect not to verify their work.
The incentive to pay for verification is not, as yet, strong: acceptance of MOOCs for general course credit toward a degree at an unaffiliated institution, or as a professional, job-related credential, is minimal so far. Harvard Business School’s HBX, launched on a different model, requires payment of substantial course fees by the students, who are interested in demonstrating mastery of basic business skills and principles. (It will be interesting to see which model Harvard Medical School adopts in its forthcoming online venture; the school has a large for-fee business providing continuing education to physicians.)
As a strategic matter, edX and Harvard and MIT will have to determine how much to trade off the goal of maximizing access to their online courses, versus either generating revenue or encouraging participants to pay something to provide their own incentive to complete a course once they enroll.
• Lifelong learners? As might be expected of MOOCs created by faculty members from elite, selective institutions like Harvard and MIT, they aim high. The report reveals that among the participants in the 1.7 million course units logged, the proportion with bachelor’s degrees ranges from 61 percent in STEM offerings and 63 percent in computer-science classes to 76 percent for those studying in humanities and associated fields, and 81 percent in the government, health, and social-science area. Elsewhere, the report notes, “no course has a majority of non-bachelor’s participants” (emphasis added).
For those who hope that MOOCs will spread higher education to underserved populations in the United States or around the world, it is thus fair to say, the promise is so far not being definitively met by the high-level HarvardX and MITx offerings—some of which deliberately aim for graduate-level students. Basic-skills and required entry-level courses still hold huge potential, but, not surprisingly, are not the focus of the Harvard and MIT professors deploying courses on the edX platform.
Beyond Description
As this report notes, it and its predecessors have been “descriptive”: organized information on course enrollments and now participants, mined from the online registration and class data. In the first year of research, the descriptions were limited by:
- the small number of courses;
- the highly heterogeneous disciplines represented among those courses; and
- the fundamental problem of the differing intent among initial registrants—and their subsequent levels of engagement with the online contents.
The second-year report, though providing much richer descriptions of participants in Harvard and MIT MOOCs, reemphasizes these limitations and the resulting “challenge for evaluation” stemming from diverse “participant intentions and backgrounds”:
Certification may be consequential and desired for some but unnecessary and irrelevant for others. An online course that certifies few participants may bring considerable benefits to teachers or residential students. Our findings suggest opportunities to prioritize particular goals for particular populations, for example, teacher usage of modules, or certificate acquisition by underrepresented students. With open access comes a disparate audience with disparate goals. Our existing research has now described this well.
Of interest now are hints that the HarvardX researchers are getting beyond what one of them described last year as “fishing in the exhaust” (that descriptive mining of raw enrollment and participation data) and progressing toward meaningful understanding of learning and the effectiveness of pedagogical techniques. Such research might involve observation of course participants as they engage with the online contents and tools; assessing students’ level of knowledge before and after a course; testing of alternative approaches (prompted or unprompted lessons, for instance, or frequent quizzes); and so on. As the new report puts it, “advancing research on learning can proceed by focusing research strategically, to better understand the most promising mechanism through which open online courses can advance learning.”
The HarvardX research team has multiple studies under way now. Bol noted the broad array of interesting research questions—including those associated with generating revenue: how to maintain the massive, open character of online courses while generating funds to cover the costs of producing them and enabling deans to compensate faculty members for the time they invest in doing so.
Some of the new studies apparently involve real experimentation and increasingly powerful research methods. A recent technical paper (with the formidable title “Addressing Common Analytic Challenges to Randomized Experiments in MOOCs: Attrition and Zero-Inflation”) gets at the problem of diverse intent and behaviors among students who enroll and then participate in online courses: a first step toward “A-B” testing of alternative teaching techniques.
Clearly, that next step toward more sophisticated assessment is a key toward achieving edX’s goals of (as the report puts it) “access, residential impact, and research” on learning.
Into the Classroom
The second of those objectives—residential impact: carrying discoveries from online learning into campus-based classrooms—goes beyond the report’s self-described focus, quoted above, on online learning proper. The larger goal, within HarvardX and beyond (including programs such as the Harvard Initiative for Learning and Teaching—now, like HarvardX, reporting to Bol as vice provost) is to understand learning and improve teaching across the board, however practiced.
A final, “anecdotal” section of the new MOOC report points toward such applications, aiming to “help communicate the breadth of the impact that the HarvardX and MITx initiatives have had on the development, visibility, and usage of pedagogical resources and innovations on campus.” At this point, it observes, more than 83 percent of MIT undergraduates have used a residential MITx system in one or more classes. “At Harvard,” it says, “many residential courses that have HarvardX counterparts are using HarvardX resources to support residential instruction.”
At one level, Bol pointed to the “reusability or recirculation of HarvardX content.” It is not the case, he said, that professors need to “flip” an entire course (recording all lectures for students to view before they meet, and then using class time for discussions and problem sets). Rather, they can adopt online units or modules to illustrate and enliven certain parts of their instruction.
Enabling such adoptions is not a simple matter technically or intellectually, he noted. How can faculty members access such teaching units, for a few minutes of content, without having their students register through edX for a full course? (A new course platform being phased into use at Harvard will ease that process, he said.) Complicating the challenge: “As the amount of learning content we create explodes,” he said, “we have to be sure people can find it.” Some online teachers have laboriously created unique “finding aids” to the visual contents of their courses, making it easier for people elsewhere and in other fields to use their materials. Bol said HarvardX had engaged the library system to help think through techniques for enabling such searches more readily and systematically: an intellectual problem, given the differences between textual and video materials.
The report broadens its field of interest still further, from MOOC participants to “other stakeholders, including teachers-as-learners, our residential students, our course instructors, and our institutions.” Among the applications of the MOOC experiments to classroom settings, the report outlines student use of online-course resources (assessments, videos, and “feedback technologies”). For course instructors, “design, development, and delivery of online materials can lead to deeper understanding of the content and new proficiency with technology-mediated teaching.”
Most broadly, the report notes a spillover effect on “residential faculty and students who are not directly involved with the MOOC effort at all”:
Through a critical mass of converging conversations about innovations in teaching and learning…MOOCs have become one of many alternative perspectives that offer everyone on campus new opportunities to see teaching and learning with new eyes.
The HarvardX and MITx MOOCs have called many longstanding features of residential courses into question: the rigidity of the semester schedule, the lengthy periods between assignment submission and feedback, and the rarity of sharing syllabi, lecture notes, lesson plans, assessments, and rubrics across faculty teaching common courses, to name only a few examples. Even if MOOCs are not a flawless answer to these questions, they deserve credit for helping to clarify issues worth addressing and inspiring campus-wide conversations about the future of learning at our institutions.
Pursuing the conversation, underpinning it with credible research and rigor, and integrating its application not only to online courses, but to learning and teaching generally: these are the high-risk, but potentially high-reward, outcomes toward which the investments in learning science, pedagogy, and educational technology ought to be directed. Perhaps they are beginning to be.
“The diverse contributions of MOOCs have been obscured by a common expectation that MOOC certification would grow exponentially in number and value, especially among those without access to conventional higher education,” the researchers observe. “That they have not yet delivered on this promise should not blind us to their other benefits.”
Indeed, with that ambition put into perspective—particularly for the kinds of courses Harvard and MIT professors are most likely to develop—the HarvardX experiment might contribute further to “strategies that capitalize on synergies” among educators and students. In this sense, the report concludes, “Participation, exploration, certification, teacher enrollment, and residential enhancement are less independent dials to turn up and down but interacting ingredients in a recipe for online and residential advances in learning.”
Read the HarvardX news announcement here.
