AI vs. Academia

Can anyone tell who wrote it?

Andrey A. Popkov and Tyson S. Barrett, “AI vs academia: Experimental study on AI text detectors’ accuracy in behavioral health academic writing,” Accountability in Research, 2024.

When academic writing looks like AI

Academic writing follows conventions. Articles share structures, use specialized vocabulary, and rely on familiar phrases to describe methods, results, and limitations.

During a literature review for behavioral health research, those patterns began to look familiar in another way: they resembled the structured language of early AI-generated text. A few published papers were submitted to an online AI detector. The detector flagged them as AI-generated—even though they had been written before ChatGPT.

ZeroGPT result for a 2016 article: 35.35% AI GPT, with the flagged passages highlighted in yellow
Published in 2016. In 2023 ZeroGPT marked 35% of it as AI.

At the same time, journals were beginning to discuss policies for AI use in academic writing. Those policies raised a practical question: could the available tools reliably distinguish AI-generated language from academic writing?

Testing the premise

We tested a free and a commercial AI detector against behavioral health research articles published before GPT and text generated by ChatGPT and Claude. A semi-automated workflow collected the texts, submitted them through the detectors' websites, and recorded their assessments, following the same steps a person using the tools would take.

The free detector identified a median of 27.2% of the human-written academic text as AI-generated. The commercial tool performed better, but it also made errors and had particular difficulty detecting text generated by Claude.

Human writing could be flagged as AI-generated, while AI-generated text could be missed. Either error matters when a detector's assessment informs a decision about an author or a manuscript.

Beyond the text

Considering the risks of falsely accusing scholars and missing AI content, perhaps it is time to move the focus from the text itself as hand-typed or AI-assisted to what is behind that text - the core ideas, the data, the analysis, the thought processes, and the conclusions that truly define academic merit.

Publication: Andrey A. Popkov and Tyson S. Barrett, “AI vs academia: Experimental study on AI text detectors’ accuracy in behavioral health academic writing,” Accountability in Research, 2024. Read the full article.