MOOC Grading

One of the more challenging aspects of working with a MOOC provider to improve assessments (or, as I prefer to call them: “Active Learning Components”) is what all this assessment (sorry “Active Learning”) is supposed to add up to.

After all, the way to improve MOOC quizzes and exams is by applying appropriate elements of the test-design toolkit used by those creating professional examinations like the A+ IT certification, the NCLEX nurse licensure exam, or the SAT.  And while peer-grading adds some challenges to assigning essays within a MOOC, ways of overcoming those challenges are going to be built around a modified version of the same process used to systematically score essays like the AP History or English exams.

But while improving the quality of individual test components is straightforward (even if it involves a fair amount of work), how those components should contribute to overall MOOC grading is anything but obvious.

For the most part, course developers have approached this issue by organizing MOOC grading and establishing a cut (pass/fail) scores in a way that ensures anyone who puts the appropriate time into the course will pass.  But as I learned last year, these low cut scores (often combined with assignments much easier than you would find in an average classroom) mean that it is also not that difficult to pass even if you did not learn what the course was trying to teach.

Personally, I think this is an OK tradeoff.  For motivated students (as I was last year) are taking the MOOC to learn which means passing is just a byproduct of taking the course seriously.  And if someone decides to stop listening to lectures, reading, or putting their learning to work by turning in assignments as soon as they get over the pass/fail threshold, who are they screwing over (other than themselves)?

But like most aspects of the MOOC experiment, this first cut at using MOOC grading as a form of motivation should be seen as part of a work in progress.  It makes a lot of sense that a MOOC should not be so hard that someone who gives it their all is likely to fail.  But since different students enter these learning experiences with different motivations (or, as my HarvardX colleague Justin Reich explains and quantifies it, different intentions), it seems as though assessment/active learning can be designed to motivate students with different types of intention to do just that little bit more.

For example, the first MOOC I ever took (Coursera’s Think Again: How to Reason and Argue) based grading on four reasonably challenging exams.  And while the pass/fail score was relatively low, the ability to pass with distinction by earning a higher grade was the motivating factor for me treating Think Again like a real college course (setting my whole Degree of Freedom project in motion) rather than as the equivalent of the Great Courses CDs I listen to in the car.

Also, the very last assignment in that class (contribution to an argument contest, which served as one of the first crowdsourced MOOC assignments) was ungraded.  But for the hundreds of students who participated in that project, it was a chance to have some fun, contribute to the community, and put our learning to work which hundreds of us took part in, even though no grade was involved.

So for those entering a MOOC with the intention of completing it, it is likely that other types of ungraded assignments (like challenging peer-graded essays or other crowdsource group projects) might be a welcome addition.  After all, most students of this type are already doing more than what is required to earn a certificate.  So why not give them additional challenges that don’t interfere with people with less ambitious goals from also passing the course?

It gets a bit trickier coming up with similar strategies for students who might just be auditing or dabbling (or students who might want to earn a certificate, but have a threshold of how much time they are ready to dedicate to the course).  Should quizzes and exams be made so much more challenging that there is a greater likelihood someone committed to the course can still fail?

Perhaps, although a better MOOC grading strategy might be to give students many chances to get over the pass/fail hurdle (by providing multiple alternative exams where only our highest grade counts – another feature I just remembered from Think Again) which would make examinations an important part of both assessment and learning within a course.

This issue will be top of mind during the last few months of my Fellowship at HarvardX.  And given the crowd-based nature of the MOOC, I welcome any thoughts, ideas, and suggestions from readers since the more we can make assessment a vital part of the MOOC learning experience, the more answers we will have for those still wondering what the fuss is all about regarding massive open learning.