Measuring MOOC Learning

Given the response I could expect if I proposed to a major publication my desire to write a 300-part series on a single subject (even one as important as the revolution now underway in online learning), I’m a big fan of what blogs allow individuals to accomplish on their own.

That said, their reverse chronological nature means that interesting content can quickly get buried as it gets pushed further and further into the past.

This was brought home to me when an older post attracted an interesting comment by Paul Morris, an independent learner from the UK who got his degrees via Britain’s Open University system that’s been providing access to accredited free learning for years.

As Paul describes, under the Open University system progress is measured in hours with the expectation that a full-year course (the equivalent of two full-semester courses in an American college system) are based on 30 weeks of work with an expected commitment of fifteen hours of effort per week.   Doing the math, this would bring a single semester-length course in at over 200 hours of expected study.

The use of time as a yardstick (or meter stick in this case) of rigor/demand on students seems reasonable.  Although, if you accept this method of measurement, this means that almost every MOOC course (certainly the ones I’ve taken so far – never mind my non-MOOC classes) would fall far short of what Open University would consider a full semester worth of learning.

Now there are other means to lay claim to having completed a college-level class such as outcomes.

My colleague and doppelganger Scott Young, for example, decided that his ability to successfully complete the same exam taken by MIT students at the end of a course (with all tests taken under the same conditions of MITers sitting in a proctored classroom) would count as passing a full-semester class. Which means that regardless of how many hours it took him to learn the material required to pass such exams, tests rather than hours of work would be his method for assigning himself credit.

Scott’s system is far from unique, especially if you look at major industries such as IT where success is based on taking rigorous, validated certification exams where passing the test is the measure of progress, rather than the number of hours required to master the material needed to sit the exam.

For my Degree of Freedom project, I’ve been using a couple of means to declare a semester-length course completed.  One is straightforward: if I do all the work associated with an online class (such as a MOOC that includes lectures, reading, homework, quizzes and other assignment) this “counts” as having finished one of the classes in my 32-course degree lineup.

But the other measure is the more amorphous concept of learning, i.e., did I learn what I would have if I took a comparable course within a brick-and-mortar university setting?

Since I already have a college degree, I’ve got a point of comparison between the amount I remember learning in a traditional college class vs. what I’m getting out of a MOOC or other free-learning experience.  And for classes like Coursera’s Think Again (a logic course I can compare to one I took on the same subject years ago when I attended university), it’s pretty clear to me that the amount one learns if you put the time into completing all of the material in that MOOC course was the same as what I learned in a comparable classroom course taken previously.

Now there are some cases where the material covered in an online class seems lighter than what I would expect from a traditional full-semester one.  But I also recall classes I took in college that were decidedly easier (and less demanding of my time) than others.

Also, as I’ve noted a number of times before, the means of measuring learning (such as automatically graded quizzes and peer-graded essays) within a MOOC course certainly feel like less than what students have to do when enrolled in a traditional university program.

I suppose if you showed up at my door this morning demanding I take a challenging final exam or write a closed-book essay on one of the subjects I studied during my freshman year, I would probably struggle with that assignment.

But if I was given time to prepare (by letting me know what to expect and  giving me time to study), I suspect I would do much better (just as any sophomore would have a better chance of success if given time to get ready for a test on a subject he or she studied during their freshman year).

More importantly, the courses I’ve been taking have exposed me to new ways of thinking meaning that if I was given some sort of open book assignment to demonstrate my learning, I would have the knowledge and experience to succeed in demonstrating that a class did indeed deliver the goods.

This analysis brings up a host of issues related to what students are accomplishing regardless of where their studies are taking place.  For today’s students all have access to far more information and resources than did any of us who went to college before the Internet age.  Which means that recall (once the key metric for success in instruments like closed-book exams) are less important than background knowledge coupled with the ability to perform research, think critically and synthesize material into something original and thoughtful.

So might we be better off focusing on MOOCs and other tools that will teach students these critical skills?  Or perhaps we should be preparing students at a younger age to perform research, think critically and write well since those who have a grounding in these vital skills are the ones most likely to succeed, regardless of where or how their learning take place.