Is AI the answer to student cheating on tests? Not that you’d know it from a recent study of AI detection system Proctorio at the University of Twente in the Netherlands.

Proctorio tracks students’ eye movements and body language while taking exams to flag “suspicious” behavior. So, as Vice tells it, 30 computer science student volunteers were told to take a first year exam that that system supervised. Six were told to cheat, five were told to act suspiciously without really cheating, and the rest were told to just write the test:

The results confirmed that Proctorio is not good at catching cheaters. The system did not flag any of the cheaters as cheating. Some used virtual machines, a known vulnerability to Proctorio’s system. The study says the software did flag those students as an “irregularity” but also flagged other honest students with the same irregularity. Similarly, some cheaters used audio calls, but Proctorio did not flag their audio as abnormal, but did flag the audio of students taking tests in noisy environments as abnormal.

Aaron Gordon, “Scientists Asked Students to Try to Fool Anti-Cheating Software. They Did.” at VICE (September 8, 2022) The paper is open access.

Student creativity proved too much for the system. The researchers concluded, “the software is “best compared to taking a placebo: it has some positive influence, not because it works but because people believe that it works, or that it might work.” (VICE)

Detecting cheating, at any level of education, a bit of a cat-and-mouse game. AI, far from being a simple fix, makes passing off the work of others as one’s own (plagiarism) easier. In a 2019 article in Nature, Debora Weber-Wulff stressed that “Software cannot determine plagiarism; it can only point to some cases of matching text. The systems can be useful for flagging up problems, but not for discriminating between originality and plagiarism. That decision must be taken by a person.”

Here’s an example: Recently, some computer science pros discovered how to use translation software to largely disguise plagiarism from machine detection. The translation software did not produce a word-for-word rendition of the original — which is what a machine detection system would look for.

The cheaters later got caught because the software is not doing any thinking. So its word choices were weird: “[T]he software can’t tell the user whether the phrase would sound odd to an English-speaking colleague. For example, ‘remaining energy’ became ‘leftover vitality’; ‘cloud computing’ became ‘haze figuring’; and ‘signal to noise’ became ‘flag to commotion.’” (Mind Matters News, August 6, 2021) Well, someone was bound to eventually read the stuff and notice the weirdness.

Sometimes, software detection systems detect plagiarism when it did not happen. In one case, the software rejected a paper for repeated citations, which were required by the format. Again, this is a function of the fact that the system is not thinking.

Why neither AI nor profs can detect one type of cheating today

Perhaps a bigger concern today is the way the internet has made one type of cheating much easier — and neither AI nor profs can easily detect it under normal circumstances — the essay writer for hire. Dave Tomar, many years a ghostwriter of student essays and author of The Complete Guide to Contract Cheating in Higher Education (Academic Influence, 2022), claims that 40% of students cheat at least once.

Once Google became the world’s search engine in 2000, the old system of just buying or copy-pasting an essay was obsolete. So the new system he worked in quickly arose. Students now hire a writer to produce a unique, original essay:

This isn’t the first time Tomar has gone public about the brain-for-hire business. In 2010, writing under the pseudonym “Ed Dante,” he offered to explain himself in Chronicle of Higher Education: The “man who writes your students’ papers tells his story,” His article, the “Shadow Scholar,” was something of a publishing event — reputedly the most commented-on article in that publication’s history. He wrote an earlier book about it (Bloomsbury, 2012). After he left that trade, he freelanced for variety of publications.

News, “Brain for hire: the internet makes academic cheating much easier” at Mind Matters News (July 7, 2022)

Tomar argues that many students are underqualified for university, in debt, and desperate. These are the sorts of problems that Weber-Wulff urges her colleagues to address as a social issue as much as an honesty issue.

At any rate, if creativity always wins out over time against the machine, we can’t address cheating without taking into account the human dimension.

You may also wish to read: Brain for hire: The internet makes academic cheating much easier. Dave Tomar, who wrote essays for students for hire for a decade, then wrote a book about it, thinks 40% of students cheat at least once. Having moved from essay writer for hire to plagiarism detector, Tomar stresses that poor skills and high debt have made many students desperate enough to cheat.