Please visit the new home of Majikthise at bigthink.com/blogs/focal-point.

« Oh Give Me A Home, Where the Caribou Roam... | Main | Thank you, guest bloggers! »

May 17, 2005

My Job Is Safe For Now

Guest post: hilzoy (who is a philosophy professor)

Department of Louise brings scary, scary news: people are not just developing, but actually marketing and using, software that supposedly enables computers to grade (or, more accurately, "grade") essays. According to one of the people who is selling this stuff, their biggest problem is that no one believes that computers can possibly do a good job grading essays:

"That's the biggest obstacle for this technology," said Frank Catalano, a senior vice president for Pearson Assessments and Testing, whose Intelligent Essay Assessor is used in middle schools and the military alike. "It's not its accuracy. It's not its suitability. It's the believability that it can do the things it already can do."

I can see how this would be a problem. Take me, for instance: I am not an ignorant Luddite, but when I ask myself: self, honestly, do you think a computer could do a good job of grading a student's essay? the answer is: no. Not a chance. Maybe at some distant point in the future, when all of us except for those fortunate few who can afford beachfront property in Antarctica are living in climate-controlled biodomes, they might be able to, but not just now. Oddly enough, I'm right:

"The software is not flawless, even its most ardent supporters admit.

When the University of California at Davis tried out such technology a couple years back, lecturer Andy Jones decided to try to trick e-Rater.

Prompted to write on workplace injuries, Jones instead input a letter of recommendation, substituting "risk of personal injury" for the student's name.

"My thinking was, 'This is ridiculous, I'm sure it will get a zero,'" he said.

He got a five out of six.

A second time around, Jones scattered "chimpanzee" throughout the essay, guessing unusual words would yield him a higher score.

He got a six.

In Brent's class, sophomore Brady Didion submitted drafts of his papers numerous times to ensure his final version included everything the computer wanted.

"What you're learning, really, is how to cheat the program," he said."

Just the lesson we want to give those impressionable young minds entrusted to our care. But that hasn't stopped people from using it:

"Watertown, S.D., students are among those who now have their writing-assessment tests scored by computer.

Lesli Hanson, an assistant superintendent in Watertown, said students like taking the test by computer and teachers are relieved to end an annual ritual that kept two dozen people holed up for three days to score 1,500 tests.

"It almost got to be torture," she said.

Some 80 percent of Indiana's 60,000 11th-graders have their English assessment scored by computer, and another 10,000 ninth-graders are taking part in a trial in which computers assess some routine written assignments.

Stan Jones, Indiana's commissioner of higher education, said the technology isn't as good as a teacher but cuts turnaround time, trims costs and allows overworked teachers to give written assignments without fearing the workload."

The question is, if those assignments are going to be graded by programs that give a perfect score to a letter of recommendation with the word 'chimpanzee' scattered throughout, why would you bother giving written assignments at all? And if you really don't care about the grading, why go to the trouble of buying software when it's so much easier to paint the letters A through F on dice and roll them? Sheesh.

For some no doubt quaint and technologically unsophisticated reason, I think that teaching my students to write well is the most important thing I can do for them, bar none. Long after they have forgotten exactly what the Categorical Imperative is, or what the main objections to utilitarianism are, the kind of intellectual confidence and precision that really learning to write gives you will linger on. So I think: if I can teach them that, I have done my job. And if, in addition, I teach them that it's possible to make arguments for moral claims, I can die happy. The idea of handing either task over to software that can't distinguish a letter of recommendation for someone called "risk of personal injury" from an essay on workplace safety is -- well, I can't even figure out how to say how awful it is. And I should say that I hate, hate, hate grading.

Of course, lucky me has a good job at a wealthy research university. So if I were to say that I just didn't have time to grade essays, I would be lying. Possibly the teachers in Indiana have the time too, and they're just lazy. Possibly, though, they really don't have the time. Grading essays is hard work, especially if you try to do right by your students. There are rough drafts and rewrites, lengthy attempts to explain what went wrong with a student's argument and how it might have been improved, and so on. I have been doing this for a while, and it still takes me about 45 minutes per 5-7 page paper, if I'm working efficiently. (Note to future professors: this is not only the right thing to do; it also does a lot for student morale. When they get over a page of comments that are plainly written about their paper, not mass-produced, they are much less likely to think you're biassed.)

The point is: it does take a lot of time. And eleventh grade teachers have a lot more students than I do. Maybe they really, honestly can't grade their students' papers. If that's the case, it's time to think about hiring some more teachers. And if there isn't enough money, it's time to think about finding some, through higher taxes if need be. Because if essays are being graded by this software, kids are being badly shortchanged.

(Cross-posted at Obsidian Wings.)

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341c61e653ef00d83423028b53ef

Listed below are links to weblogs that reference My Job Is Safe For Now:

» Introducing the grade-O-matic 3000 from Lucky Jim
hilzoy, guest-posting at Majikthise, casts a critical glance at computer grading of student essays. As much as I share hilzoy's skepticism about the efficacy of this software, I must admit that I felt a little thrill at the thought of outsourcing my ... [Read More]

» “True genius resides in the capacity for evaluation of uncertain, hazardous, and conflicting information.” ~ Sir Winston Churchill from ESL Lesson Plan
It's sometimes hard to be fair and even harder to remain objective. What do you have to say about the controversial subject of grading student papers? [Read More]

» “True genius resides in the capacity for evaluation of uncertain, hazardous, and conflicting information.” ~ Sir Winston Churchill from ESL Lesson Plan
It's sometimes hard to be fair and even harder to remain objective. What do you have to say about the controversial subject of grading student papers? [Read More]

» “True genius resides in the capacity for evaluation of uncertain, hazardous, and conflicting information.” ~ Sir Winston Churchill from ESL Lesson Plan
It's sometimes hard to be fair and even harder to remain objective. What do you have to say about the controversial subject of grading student papers? [Read More]

» "True genius resides in the capacity for evaluation of uncertain, hazardous, and conflicting information." ~ Sir Winston Churchill from ESL Lesson Plan
It's sometimes hard to be fair and even harder to remain objective. What do you have to say about the controversial subject of grading student papers? [Read More]

» Evaluating Student Work in ESL from ESL Lesson Plan
It's sometimes hard to be fair and even harder to remain objective. What do you have to say about the controversial subject of grading student papers? [Read More]

» Evaluating Student Work in ESL from ESL Lesson Plan
It's sometimes hard to be fair and even harder to remain objective. What do you have to say about the controversial subject of grading student papers? [Read More]

» Evaluating Student Work in ESL from ESL Lesson Plan
It's sometimes hard to be fair and even harder to remain objective. What do you have to say about the controversial subject of grading student papers? [Read More]

» Evaluating Student Work in ESL from ESL Lesson Plan
It's sometimes hard to be fair and even harder to remain objective. What do you have to say about the controversial subject of grading student papers? [Read More]

Comments

Nope, sorry --- it's more important to keep taxes down. Let people learn how to write correctly on their own dime. Plus, you don't want the kids to get to uppity. They might read something about how their standard of living is declining, and their nation is regressing. Best just to let them write about workplace safety for monkeys.

I don't know what you guys are talking about. My essay got 6/6 points, while your boring academic prose only got 4/6. I am better then you. You are just jealous. I will show you part of my essay so you can marvel at my writing skills.

I highly chimpanzee this risk of personal injury because this risk of personal injury has a high chimpanzee work ethic, and a brilliant mind. Chimpanzee risk of personal injury also bananas a balanced quantum physics mental state where they can handle chimpanzee stress. Chimpanzee is a fast learner, and will easily quantum mechanics the risk of personal injury to your space shuttle language. Language is this risk of personal injurys chimpanzee, the same way quantum mechanics is this hospises space shuttle.

See, you guys are just jealous that the computer picks out better writing then you would.

Why is chimpanzee such a high scoring word. Is it something to do with scrabble scores?

Just goes to prove the old saying, "To error is human, to really screw things up takes a good computer."

I wonder if the computer is using a 'range of knowledge' algorythm. Quantum mechanics, chimpanzee, personal injury and Space Shuttle cover a wide range. MSWord called the first sentence a fragment...

"Possibly the teachers in Indiana have the time too, and they're just lazy."

Many are...I speak as a victim of Indianapolis Public Schools, and as an employee of a BIG university.

P.S. We have some good ones, too...they're just plain overworked.

The grammar checker in Word is pretty much worse than useless. No one seriously thinks this software works, do they? Come on, you're pulling my leg. Please say you are.

That was the fun fun of it. Heck, that did not even qualify as a fragment. The entire chimpanzee of that first personal injury made no Space Shuttle whatsoever. Word at least was critical.

I did this years ago. I got tired of correcting short answer and essay questions, so I used mark sense forms and required students to write answers using only the first five letters of the alphabet.

I have no problem with computers grading essays. After all, the essays are generated by computers as well

The comments to this entry are closed.