As many readers know, my background in professional test design has left me sensitive to MOOC quizzes and exams that sometimes seem thrown together as afterthoughts.
This is because in the best designed courses, instruction and assessment (whether in the form of quizzes, final exams, graded papers and homework assignments) work hand in hand to deliver and reinforce learning (with lectures and reading the primary vehicle to deliver information and assessment the means by which students internalize that learning by putting it to work).
In a best case scenario, a course plan places equal weight on instruction and assessment. For instance, when I was on a team that designed a standard for teaching Digital Literacy many years ago, we created a course curriculum and exam blueprint at the same time in order to ensure every learning objective we included could be both taught and measured.
The primary limitation in MOOC assessment is that the scale of massive classes means tests and other assignments need to be graded by someone other than the professor teaching the course (or his or her teaching team). This translates to tests that are machine scoreable (i.e., multiple choice), and assignments that are either self- or peer-graded (the preferred method of evaluating open-ended assignments such as papers or lab reports).
Given the weight automated exams have to pull in most MOOC classes, professors and course designers should do everything they can to make these assessment tools as challenging and effective as possible. At the very least, items in those tests, which typically include not just multiple-choice but also multiple response (multiple-choice with more than one correct answer), true/false, and open response (fill-in-the-blank) items should follow these rules for quality item development.
But the prevalence of true/false items (which are no-no’s in professional testing) in so many of my MOOC exams attests to the fact that these rules (and one suspects other methods for maximizing the effectiveness of assessment) are not given the attention they deserve in the MOOC development process. But beyond the vague complaints of a testing dweeb, is there some scientific means to determine whether we’re talking about a genuine problem (rather than a pet peeve)?
If I had access to test data, I could perform some of the analysis described in this piece. But absent Coursera or edX sending me terabyte-size databases to play around with, I’ve come up with my own experiment based on translating test questions in existing MOOC exam into what I call an Obviousity Index.
This index takes advantage of the fact that many tests created by people without item-writing training tend to include the same mistakes, notably: (1) the longest and most detailed answer turns out to be the correct one; and (2) “All of the above” or “None of the Above” answers tend to be correct (and when both appear in a question, “All of the above” predominates).
The use of true/false (or equivalents like “Yes/No”) also presents problem, especially on Pass/Fail courses with low pass scores (since random guessing on true/false items can earn you 50% on your way towards this low cutoff mark).
While other test factors are important (such as not including obviously wrong answers as distracters), these are subject to qualitative judgment, so I have not included them in my Obviousity Index which will be determined by following just these steps for every graded assessment in a MOOC course:
- Answer every multiple choice question by selecting the longest answer
- Answer “All of the above” whenever it appears (this rule takes precedent over Rule #1)
- On any True/False or equivalent question, flip a coin to select an answer (Heads for True/Yes, Tails for False/No)
- Do not answer any question that is in the multiple-response or open-response (fill-in-the-blank) format
After going through these steps, I’ll calculate the total percentage grade I would have received in a course if I blindly followed the rules above without paying any attention to the substance of the test questions (or course content for that matter).
To simplify this analysis (and my life), I’m only planning to apply this technique to courses (ideally from more than one institution/vendor) where the final grade is based entirely on how you perform on automatically graded quizzes and exams.
So tune in tomorrow to see what the Obviousity Index reveals (if anything).
Many courses give the impression that the designers get to the end of refactoring their existing course materials then have no real idea how to replace the written papers that are the mainstay of their in-course assessment. As the courses are not for credit, a couple of quick multiple choice quizzes seem the easiest solution–after all, nobody really cares how highly they score, do they? Once one adds in the desire to keep students engaged it is hardly surprising that few courses have taxing assessments.