MITx and HarvardX Research Findings – 1

While nature was dumping mounds of snow on the region (albeit less than promised), Harvard and MIT decided to release a comparable amount of data to the public that provides some important insight that can inform the next round of discussions surrounding MOOCs, discussions that promise to go beyond the inflated expectations that began 2013 and the disillusion-based cynicism that ended that same year.

Many of us have pointed out that both the numbers and vocabulary we use to describe the success or failure of MOOCs (notably “enrollments” of 100,000+ students and “drop-out rates” of 95%) lack the kind of depth (or even meaning) needed to make sense of what students are actually doing when they participate in a MOOC class (other than signing up for it and either earning or not earning a certificate).

But with a set of working papers released by the two founders of edX: an overview analyzing the behavior of students in six HarvardX courses and eleven MITx ones, as well as separate papers digging into the details of each individual course, we now have numbers we can use to build a better portrait of the complex behavior of students involved with massive open learning.

Some of this data simply confirms what other research has demonstrated, especially with regard to the demographics of a MOOC student body (which tends to be older and better educated than the 18-22 year old undergraduates who generally take a Harvard or  MIT class).  Gender tends to skew male for science and technology classes, and closer to 50:50 for humanities classes.  And, as many have already noted, the number of people that click the button to “enroll” in a MOOC class is 10-20 times higher than the number of people who get certified.

But given that (unlike a traditional college course) there is no cost to signing up for a MOOC with no intention of completing it, those enrollment numbers are subject to a range of confounding factors (my favorite being the “Colbert Bump,” when an appearance by Anant Agarwal of edX on the Colbert Report led to a tripling of enrollments – but not a tripling of graduation rates).

More importantly, how students spread out in terms of activity and success after they hit that enroll button tells us much more than can simple starting and finishing statistics.  For instance, each of the new HarvardX/MITx reports tries to categorize student behavior with charts such as these (drawn from the two courses I’m most familiar with: Ancient Greek Hero and Justice – both from Harvard):

The dots on these diagrams map student activity (measured by the number of course chapters accessed)  against success (based on final grade), and as the enormous scattering on both course charts indicates, there does not seem to be any type of behavior unrepresented by some course takers within the student body of a MOOC.

Regions of high density correspond to more general categories of behavior, from those who watched less than half the course videos and did not earn a certificate (which the researchers behind the report refer to as “Viewers”), those who watched more than half the videos but did not pass (called “Explorers”) as well as “Certified” students (those who passed the course), many of whom seemed to have earned their certificate without engaging with all of the material.

An extreme case of the latter phenomenon is what the report calls an “Optimizer” (someone who does the absolute minimum to earn a certificate).  Such certificate hounds, however, seem to be outliers.  More interesting categories include those who watch most or all of the videos who never take a single test (which the report refers to as “Listeners,” although “Auditors” would probably be a better label) and “Completionists” who interact with all of the course material, including assessments, earning a final score far higher than was necessary to pass.

Personally, I would raise the Listener/Auditor threshold to at least .25/25% on the y-axis since it wouldn’t be unusual for someone primarily interested in auditing to try a quiz every now and then, even if they had no intention of earning a certificate.  Similarly, “Completionist” should really include everyone who engaged with most or all of the material and passed the course (especially for a course like HeroesX which included more challenging assessments than other MOOC classes – including Justice).  In fact, those dense bars in the top right corner of each chart could be thought of as encompassing those who got the most out of the course by dedicating themselves to learning, with a passing grade just a byproduct of that choice.

This brings up another important variable the researchers behind these reports stressed again and again: that individual courses have individual natures, meaning they cannot all be lumped into one bucket and used as the basis to pass judgment on “MOOCs as a whole” (whatever that means).

To take one example drawn from the charts above, assessments in HeroesX was deeply embedded in the course material, which is why all data points fall below the diagonal (since it was impossible to take an assessment without at least opening one of the course modules).  JusticeX, in contrast, had separate links to lessons and graded assessments, which is why it was possible for some students (many, in fact) to earn a high score on tests without accessing all (or even most) of the learning content.

There are more insights to be gained and more results to discuss that come out of this important research now available to the public (more than can be contained in a single post).  So tune in next time as we dig deeper into what the numbers tell us about a mode of teaching and learning we are only beginning to understand.