“Shouldn’t there be a dot on that ‘i’?” – Can machines help humans mark MOOCs?
The issue of how a mark MOOCs is a moot point at the moment.
As Europeans race to play catch up with their U.S. counterparts, (no educational body left behind!?), two glaring questions rise to the fore in many articles about these massive open online courses.
These are:
- How do institutions make money from them?
- Will MOOC students be able to gain credits for offline courses?
The answer to both these questions, in my view, ultimately rests on how the courses are evaluated.
To date, I have completed two MOOCs – Edinburgh University’s E-Learning and Digital Cultures (edcmooc), and the Open University’s Open Education (#h817open).
For the first, the assessment was done by peer evaluation of a digital artefact with students asked to mark their fellows on a scale of zero to two (0 = “You have got to be kidding me”; 2 = “I bow to your awesomeness”).
Effectively, anyone who submitted a video, Prezi, blog or other clickable virtual product gained a ‘Statement of Accomplishment’; but, only those awarded a grade of 1.5 or 2.0 earned the prestigious ‘with distinction’ wording on their downloadable certificate.
Before those in the latter category rushed to celebrate, they would have been well-advised to read the small print on the certificate which read :
“Please note: The online offering of this class does not reflect the entire curriculum offered to students enrolled at the University of Edinburgh. It does not affirm that the student was enrolled at the University of Edinburgh or confer a University of Edinburgh degree, grade or credit. The course did not verify the identity of the student”.
In plain English, this tells us: Don’t pretend you are on a par with real Edinburgh University academics; hell, we can’t even prove you are who you say you are.
Martin Weller
For the OU course, there were no certificates; the only way you could prove to doubters that you weren’t one of the dropouts was to apply for, receive and display a badge to show a) Understanding of OERs (Open Educational Resources) b) Understanding of MOOCs c) Course completion.
The decision as to who had earned these badges fell to course guru, and all round decent guy, Martin Weller.
Needless to say, no-one could delude themselves into thinking that these awards (created in Cloudworks) were equivalent to an Open University degree, grade or credit.
All this is fine and dandy for ageing lifelong learners, like yours truly, who are seeking greater wisdom rather than looking for ways to sex up their CV.
Nevertheless, I have a sneaking suspicion that the novelty value of MOOCs will quickly wear off for many participants when it dawns on them that there is no meaningful feedback or tangible reward for all the heavy-duty brainpower they have expended.
How can MOOCs boost credibility and maintain their appeal on this thorny issue of assessment?
Given that those registered for these free courses runs into the thousands rather than the hundreds, the potential workload for even the most dedicated academic is daunting and, arguably, beyond the realm of possibility.
Peer assessment is one alternative but, as this tends to be a bit of a lottery, it is not always popular with students and is unlikely to cut much ice with most educational institutions either.
Another possibility is to devise some automated computer-assisted marking scheme.
Prolific Melbourne-based teacher, writer and freelance journalist, Stephen Downes, has just pondered upon these very questions on his illuminating ‘Half an Hour blog’. He confesses at the outset “I can’t see myself marking 5,000 term papers, and a similar number of exams” , and who can blame him?
As a self-declared “specialist in online learning and new media technologies”, Downes is a man who should be able to shed some light on this topic, but, although he offers some interesting examples of methods that have been tried, he doesn’t say much more than to observe that there are risks whatever approach you take.
Let’s hear it for the old-school method!
If you look at the recent Position Statement on Machine Scoring by the National Council of Teachers of English (NCTE), you are left in no doubt that schemes that propose a non-human grading of written work ruffle quite a few feathers.
The opening paragraph of this statement sets the tone: “Writing is a highly complex ability developed over years of practice, across a wide range of tasks and contexts, and with copious, meaningful feedback. Students must have this kind of sustained experience to meet the demands of higher education, the needs of a 21st-century workforce, the challenges of civic participation, and the realization of full, meaningful lives”.
After reading this, you will not be surprised to learn that they totally reject the notion that any artificial intelligence could be up to the job of grading what they describe as “rich, multifaceted” tasks.
NCTE dismiss the idea that machines can gauge the subtleties of written texts, and state that the process would merely reward grammatical accuracy/lexical complexity. This may therefore penalize the type of innovative or creative styles that good university-level students should be aspiring to. If, for example, someone uses repetition to hammer home a point, would a computer be able to pick up the fact that this was done deliberately.
NCTE also foresee that test-takers would soon devise “machine-tricking strategies”, and it’s easy to see how such techniques could be rapidly circulated on the net.
You begin to see what a thorny issue this whole marking business is.
I think it is inevitable that, sooner or later, participants will be asked to pay for some, or all, of the ‘expert’ support offered on these massive courses. When that day arrives, the honeymoon period of MOOCs will be well and truly over.
At this point, we are likely to see an escalation of learner-generated remixes of open educational resources that by-pass institutionalised credit-based systems entirely.
Of course, you wouldn’t have any paperwork at the end of such ‘courses’ but they would fulfill the goal of improving world knowledge and, correct me if I’m mistaken, isn’t that what education is meant to be for?
I took the same U of Edinburgh course. I already have two Masters, but don’t want to spend any more money on a traditional education. I’m currently taking two MOOCs from Harvard. If I pass (a big if), these MOOCs will look good on my resume. I don’t need any more college credits. But there are slight discussions in the U.S. about allowing for college credits. It’s almost entirely about money, not education. If you take 57 out of 60 credits from a college, and 3 from a MOOC, then the college is out of the amount of money for that course. Now imagine if all your students take 3 MOOCs, and get credit, they could start losing some serious money. Also, professors are worried about their jobs, in the respect that, why would students in some crummy college want to trudge to a crummy lecture at 9 a.m., when they could get an excellent free online course from an excellent professor at an excellent university?
I’ve never known any university that doesn’t charge for transferred credit — even the OU itself does. There’s not necessarily any financial threat to the existence of a small university, as they can try to find a price-point where both the institution and the student benefit.
In fact, one of the current motivators for the uptake in (non-free) assessed xMOOC courses in the US is the claim that many community colleges are oversubscribed, and the use of outside electronic tutoring won’t reduce the student numbers at all; it will simply increase the student turnover. Everyone, in theory is a winner.
The other way that cheap (again, free is unlikely for credit-bearing courses) online courses might benefit universities is that they reduce the fear of committing to the cost of a full degree programme, at the same time as making the student personally invested in the idea of getting a degree — i.e. you’ve got the first semester or two out of the way for a couple of hundred rather than a couple of thousand, and you’ve now started on your degree, so you want to finish it.
Think of it as an educational “gateway drug”… during my time studying with the Open University in the UK, I encountered a fair few people who started with the intention of doing a couple of modules for personal interest, but ended up continuing on to get a degree, just for the sake of getting a degree.
Yes, there’s a huge paradox in the current situation – on the one hand institutions are offering these free, open courses in an apparently altruistic way while, as you say, other considerations are “almost entirely about money, not education”.
Almost all the xMoocs in US are allowing credit for a small fee US$39-79. Not for all courses yet, but going that way. 2 states are ramrodding legislation to make it mandatory for their state institutions to except MOOC credit. If this trends as fast as the whole MOOC movement; then it is happening now.
All the major US MOOC will certify the id of the MOOC attendee for a small fee. So the credit part is happening. The comfortably tenured really need to take their heads out of the sand and keep an ear to the ground.
Reblogged this on Organisational Learning and Development and commented:
Great little Blog post on what I think is one of the biggest issues facing MOOC’s and the like. That is how does one show that one learnt anything, or that you are competent in the materials. From an organisational point of view I really dont care if you have done 100 MOOC’s on a range of subject, unless there has been proper assessment (and sorry peer review of your ‘work’ just doesnt cut it for me) then dont even both putting it in your resume as it is meaningless unless you can actually show that someone is willing to say that you are competent.
One of the potential funding models for moocs was to run the course free to all and then charge for assessment & certificate. So anyone interested can study and any that want a certificate can pay and thus fund a marker to mark them.
Accreditation, accreditation, accreditation (to misquote T Blair)