Assessment is such an important topic with regard to MOOCs and other new learning models that I plan to devote a week to the subject in April. But for now, I’d like to provide an argument why testing is both the greatest vulnerability for moving MOOCs forward, as well as the ripest area for innovation in online learning.
To begin with, if you run down the list of course components discussed this week (which so far covered lectures, homework, reading, and discussion/community) very few of these require complete re-imagination when a course moves from the physical classroom to the Web.
Lectures (typically 50-90 minutes long in a traditional class) are already broken down into smaller increments in most virtual courses (from the 1-2 minute micro-lectures found in the Udacity classes I’ve enrolled in, to strings of linked 5-15 lectures that are the preferred model for courses I’ve taken from Coursera and EdX, to the strict 30-minute standard of Great Courses audio classes).
Theoretically, lectures might eventually be split still further and fit into some sort of adaptive branching model. But, as the leaders of both Coursera and EdX pointed out at an MIT meeting I attended last night, these options present technical and pedagogical challenges that may make them less beneficial than one might think in terms of delivering learning.
But regardless of how you slice, dice and arrange the lecture, the best courses will always involve an expert with talented presentation skills communicating information to students (whether face-to-face or via screen or iPod). While some courses I’ve taken separate from this project eliminate lectures entirely in favor of guided reading and online communication, these do not reflect the model for the most compelling multi-media options currently generating such excitement (and concern) within the educational community.
And speaking of reading, as far as I can tell this still involves students absorbing text-based materials (whether delivered to them on paper or on screen). The notion of how much reading should be associated with a college-level course is an important point to debate, particularly as we evaluate whether a MOOC or other online experience is truly as rigorous as its’ classroom equivalent. And such a conversation should include whether one mission of the college course is to enforce reading of a certain amount or certain set of texts.
But these are debates over quantity, not over the qualitative nature of a learning component (in this case reading). Similarly, even the technological options supporting online equivalents of classroom discussion (including online discussion forums, social networks and audio/video conferencing technologies) mostly bring up issues relating to numbers (i.e., how to get enough students to contribute and interact enough without swamping online forums to the point where they become unusable).
But testing – or any technique for evaluation (embodied in homework, quizzes, exams, graded writing assignments or class projects) is something that can’t be simply moved from a professor’s paper file to the screen without thinking through issues such as:
- What is the purpose behind a specific evaluation exercise (be it homework, a quiz or high-stakes exam)? Is it to ensure comprehension of a reading assignment or lecture point? Measure recall of factual knowledge? Give students the ability to demonstrate critical thinking or other higher-order skills?
- What assessment techniques are practical in the context of an online class delivered to large numbers of students?
- How do we know whether those chosen assessments are actually measuring what we want measured?
One advantage of the traditional classroom is that there exists an authority (the professor) who acts as the setter of standards. And regardless of whether this authority is perfectly consistent or wildly arbitrary in how he or she grades assignments (or asks his or her TAs to do perform such grading), the dozens (or, in some cases, hundreds) of students enrolled in a college class know what they’re getting into when they select a teacher.
But once grades are assigned based on machine-scored multiple-choice quizzes or peer-reviewed writing assignments (the two most popular assessment techniques I’ve seen so far in the MOOC classes I’ve taken), to whom can a student turn if they feel that a test question stinks (or is wrong) or if they don’t feel their peers adequately graded their work?
And if students don’t feel that this week’s assessments are being handled rigorously, how seriously will they take next week’s? Finally, if it’s an open question whether material taught has actually been learned, how strong an argument can we really make regarding the comparability between a MOOC and a brick-and-mortar college class?
Fortunately, many problems related to how to give consistent testing to large diverse populations have been addressed (if not entirely solved) within industries such as the standardized educational testing, professional licensure and certification. And in a few weeks time, we’ll take a look at how we might be able to apply some of the science and solutions they’ve been working on to this vital aspect of the online learning experience.
How about asking for your money back?
Assessment is probably the biggest hurdle for any online course. Any automated system is going to rely on closed format questions for the foreseeable future (the ability of automated systems to handle real language well enough to assess academic writing is a good few years away). Closed form questions (eg multiple choice) can be effective in assessing objective knowledge (although they tend to favour recall unless real effort is put into the question writing) but it is difficult to envisage how they could work with more subjective (ie typically liberal arts) material.
The alternative being offered on a number of courses is the use of peer assessment. There are a couple of issues with this method. Having used it in school and college course I have myself taught, i have almost always been disappointed by the quality of feedback given even where detailed guidance is given. A typical feedback given to a classmate on a piece of work which had taken (so far) 12 hours to produce – ‘really good’. When pressed (the course specification required students to amend their products based on feedback, so it was fairly important to get something usefully critical) they changed it to ‘really good colours’. Even my best and most fluent students rarely managed more than a couple of sentences and their observations were so trivial as to be of little value. The problem was that the didn’t have a firm enough grasp of the material to be able to offer insightful comment and the didn’t have the experience or training to offer (or accept) critical feedback. I’ve seen little to suggest that the majority of students enrolled on MOOCs will be any better at assessing or giving feedback.
Working through a MOOC now that uses peer assessment and I am immediately concerned even before the first assessment comes through. Initial discussions reveal that many (if not most) of those posting don’t even understand what the course is about (an academic writing course for college level students) and are focused on creative and expressive writing. Moreover, a significant proportion do not have English as their first language and, from their postings, don’t have an adequate grasp in order to proceed. I’m not prejudiced against non-native English speakers but it’s not unreasonable to expect any student to have a good grasp of the language before starting a college level writing course. When I expressed the view that feedback criticising early posts for failing to ‘express the soul’ of the writer were probably misguided in this context I was voted down and roundly condemned for taking a narrow view of writing. This doesn’t fill me with confidence.
We should, of course, as academics take a scientific approach to this. We cannot trust our own instincts or anecdotal evidence. So what does current research say about the accuracy of peer review. My reading of the research seems quite optimistic so I will give it a try. In addition we can verify the results for ourselves. Set up the peer assessment process and then take samples and see to what extent the instructor grades agree with the peer grades. We can also apply modifications based on grading to the curve. Of course that assumes that all students will tend to be biased in the same direction. To deal with this you can use Calibrated Peer Assessment as Coursera are meant to be doing. Ask each student to grade a sample assignment and estimate that student’s ability to grade and tendency to be biased based on that sample. Use that information to modify that student’s grades in the course. Again, this technique should be verified by sampling.
One thing I am concerned with is that this is a measurement problem and a good familiarity with measurement science and statistics is necessary to deal with the issue. I am not convinced that all who are commenting have these skills.
I agree with Paul that assessment is one of the biggest areas that needs improvement with regard to both testing content and the technology/methodology used to implement assessment within a MOOC. But like Brian, I’m optimistic that present shortcomings can be overcome, especially since I’ve seen similar challenges overcome during the years I lived in the world of prefessional exam development.
Most teachers have mastered the artistry of testing, but testing science is not often implemented outside of the world of standardized assessment (such as educational testing, certification and licensure) since classroom testing is graded by an acknowledged authority (the teacher). But since MOOCs are bringing the once intimate classroom to mass scale, we need to take a look at some of the science that has let testing work at scale to see if we can find solutions to current challenges with massive courses.
This issue is so important that I’m planning to dedicate a week to discussing the topic – which will be the week after this coming one (starting April 8).
In addition I think we should separate assessment for the purpose of giving a MOOC certificate from assessment for college credit. Giving a MOOC certificate is a bit like a certificate of achievement at a workplace training event. People like to know that you attended the training event and actively took part in the learning process. Given other evidence, people may also deduce that you would have learned something for it. This is not serious accreditation and it may be relatively easy to automate it or otherwise get the marginal cost to a very low level. However, awarding college credits is another thing. this may require some serious assessment and most people accept that it will cost something. For the moment we may need more than peer assessment for this. In the future such serious assessment may be a combination of objective tests (MC quizzes), peer-assessments augmented by instructor verified assessment.
I believe that no matter what discipline a person is attempting to learn or master there will always be the need for testing or certification. If you look at the large IT institutions most of them have a stringent certification test to validate the comprehension of the student. Until such a system of validation is put into place their will always be mostly skeptics of a student’s commitment to learning on their own. I personally have taken several Lynda.com courses and received my certificate of completion and it was a terrific add on to my existing understanding of Excel; however, probably could not pass Microsoft’s Office certification test.
We live in a world of instant gratification and the word FREE is the age old marketing concept to getting an individual to try out a product and MOOCS are no different from this marketing lure. It starts with 160,000 students to enter in a basic sign up and at the end 1% could maybe be graded for their efforts.
Time is another huge factor in this equation, as stated on this web-site, is it really possible to mastering a discipline in a quarter of the normal time to accomplish a bachelor degree, probably not; however, if the focus of the learning were more around direct learning of a profession rather than electives it might be possible to create profession certification.
What surprises me is that business is not getting more involved in this concept, because of the jobs and skills gap that seems to exist in the market place. My idea would be be to have companies post jobs that really breakdown the skills requirement and compensation of the work, then an individual could take the required courses and a certification to fulfill the requirements of the position. Of course, this concept is attempting to help drive an economy versus just learning for leisure. I think that the skills of today’s business world have become some much more technical and precise that general degree programs seem to have a gap between university learned information and marketplace skills.
I think that’s a fantastic idea Dan. Get companies to offer learning courses based on the skills they want applicants to have. Eventually, multiple companies will realise that they are duplicating each others work and may farm this out to a single provider of learning for that content.
I completed a Coursera MOOC just before Christmas, learning a programming language.
The assessment approach was peer review/marking and I found the experience very positive.
It worked because the assignments were coding exercises and had clear goals/outcomes to be achieved on each exercise.
There was some freedom to the way you did your coding, but to get the points you had to achieve the minimum goals/outcomes and your peers knew exactly what the assessment criteria were.
So there were two positives:
1. It was easy to assess/mark for peers.
2. It allowed some freedom and creativity in the way the projects were coded or some freedom to express creativity, but still meet the assessment criteria.
The college had obviously spent some time setting up the environment so that students could code, test and submit their projects, then later get them marked by peers.
Since completing that course, I tried two others that didn’t have the same approach and I didn’t complete them.