My Job Is Safe For Now
Guest post: hilzoy (who is a philosophy professor)
Department of Louise brings scary, scary news: people are not just developing, but actually marketing and using, software that supposedly enables computers to grade (or, more accurately, "grade") essays. According to one of the people who is selling this stuff, their biggest problem is that no one believes that computers can possibly do a good job grading essays:
"That's the biggest obstacle for this technology," said Frank Catalano, a senior vice president for Pearson Assessments and Testing, whose Intelligent Essay Assessor is used in middle schools and the military alike. "It's not its accuracy. It's not its suitability. It's the believability that it can do the things it already can do."
I can see how this would be a problem. Take me, for instance: I am not an ignorant Luddite, but when I ask myself: self, honestly, do you think a computer could do a good job of grading a student's essay? the answer is: no. Not a chance. Maybe at some distant point in the future, when all of us except for those fortunate few who can afford beachfront property in Antarctica are living in climate-controlled biodomes, they might be able to, but not just now. Oddly enough, I'm right:
"The software is not flawless, even its most ardent supporters admit.
When the University of California at Davis tried out such technology a couple years back, lecturer Andy Jones decided to try to trick e-Rater.
Prompted to write on workplace injuries, Jones instead input a letter of recommendation, substituting "risk of personal injury" for the student's name.
"My thinking was, 'This is ridiculous, I'm sure it will get a zero,'" he said.
He got a five out of six.
A second time around, Jones scattered "chimpanzee" throughout the essay, guessing unusual words would yield him a higher score.
He got a six.
In Brent's class, sophomore Brady Didion submitted drafts of his papers numerous times to ensure his final version included everything the computer wanted.
"What you're learning, really, is how to cheat the program," he said."
Just the lesson we want to give those impressionable young minds entrusted to our care. But that hasn't stopped people from using it:
"Watertown, S.D., students are among those who now have their writing-assessment tests scored by computer.
Lesli Hanson, an assistant superintendent in Watertown, said students like taking the test by computer and teachers are relieved to end an annual ritual that kept two dozen people holed up for three days to score 1,500 tests.
"It almost got to be torture," she said.
Some 80 percent of Indiana's 60,000 11th-graders have their English assessment scored by computer, and another 10,000 ninth-graders are taking part in a trial in which computers assess some routine written assignments.
Stan Jones, Indiana's commissioner of higher education, said the technology isn't as good as a teacher but cuts turnaround time, trims costs and allows overworked teachers to give written assignments without fearing the workload."
The question is, if those assignments are going to be graded by programs that give a perfect score to a letter of recommendation with the word 'chimpanzee' scattered throughout, why would you bother giving written assignments at all? And if you really don't care about the grading, why go to the trouble of buying software when it's so much easier to paint the letters A through F on dice and roll them? Sheesh.
For some no doubt quaint and technologically unsophisticated reason, I think that teaching my students to write well is the most important thing I can do for them, bar none. Long after they have forgotten exactly what the Categorical Imperative is, or what the main objections to utilitarianism are, the kind of intellectual confidence and precision that really learning to write gives you will linger on. So I think: if I can teach them that, I have done my job. And if, in addition, I teach them that it's possible to make arguments for moral claims, I can die happy. The idea of handing either task over to software that can't distinguish a letter of recommendation for someone called "risk of personal injury" from an essay on workplace safety is -- well, I can't even figure out how to say how awful it is. And I should say that I hate, hate, hate grading.
Of course, lucky me has a good job at a wealthy research university. So if I were to say that I just didn't have time to grade essays, I would be lying. Possibly the teachers in Indiana have the time too, and they're just lazy. Possibly, though, they really don't have the time. Grading essays is hard work, especially if you try to do right by your students. There are rough drafts and rewrites, lengthy attempts to explain what went wrong with a student's argument and how it might have been improved, and so on. I have been doing this for a while, and it still takes me about 45 minutes per 5-7 page paper, if I'm working efficiently. (Note to future professors: this is not only the right thing to do; it also does a lot for student morale. When they get over a page of comments that are plainly written about their paper, not mass-produced, they are much less likely to think you're biassed.)
The point is: it does take a lot of time. And eleventh grade teachers have a lot more students than I do. Maybe they really, honestly can't grade their students' papers. If that's the case, it's time to think about hiring some more teachers. And if there isn't enough money, it's time to think about finding some, through higher taxes if need be. Because if essays are being graded by this software, kids are being badly shortchanged.
(Cross-posted at Obsidian Wings.)