When academic writing looks like AI
Academic writing follows conventions. Articles share structures, use specialized vocabulary, and rely on familiar phrases to describe methods, results, and limitations.
During a literature review for behavioral health research, those patterns began to look familiar in another way: they resembled the structured language of early AI-generated text. A few published papers were submitted to an online AI detector. The detector flagged them as AI-generated—even though they had been written before ChatGPT.

At the same time, journals were beginning to discuss policies for AI use in academic writing. Those policies raised a practical question: could the available tools reliably distinguish AI-generated language from academic writing?
Testing the premise
We tested a free and a commercial AI detector against behavioral health research articles published before GPT and text generated by ChatGPT and Claude. A semi-automated workflow collected the texts, submitted them through the detectors' websites, and recorded their assessments, following the same steps a person using the tools would take.
The free detector identified a median of 27.2% of the human-written academic text as AI-generated. The commercial tool performed better, but it also made errors and had particular difficulty detecting text generated by Claude.
Human writing could be flagged as AI-generated, while AI-generated text could be missed. Either error matters when a detector's assessment informs a decision about an author or a manuscript.
Beyond the text
Considering the risks of falsely accusing scholars and missing AI content, perhaps it is time to move the focus from the text itself as hand-typed or AI-assisted to what is behind that text - the core ideas, the data, the analysis, the thought processes, and the conclusions that truly define academic merit.
Publication: Andrey A. Popkov and Tyson S. Barrett, “AI vs academia: Experimental study on AI text detectors’ accuracy in behavioral health academic writing,” Accountability in Research, 2024. Read the full article.