The greater problem of AI detectors compared to AI [Updated 14-06-2023]

Update 14-06-2023: Copyleaks confirms the following, and I quote: “…we actually don’t claim to be able to flag poetry, songs, haikus as accurately as traditional content at this time.”

As a teacher, I have many reasons to want AI detectors to work. If only to know that a student actually wrote their papers themselves. However, this doesn’t mean I trust AI detectors without a question…

Let’s take a moment to look at AI detectors. The ones I personally like are Copyleaks and Crossplag. Both you copy the content, paste it into their system and you will get a percentage of likelihood of it being AI written. And the interesting part, the results are completely different while comparing literally the same texts. So, how can this be? Well, you would actually need to ask the creators for that answer.

However, the fact of these massive differences reveals a notable problem, the fact that AI detectors do in fact already have a problem detecting if a text is written by AI. And there needs to be added another important fact here, we are only at the start of the AI ‘generation’. It’s very likely that AI will evolve in the coming years, with the AI’s writing becoming more and more human-like over time. And in this regard, AI detectors will only get bigger problems over time. In fact, there will likely come a time when AI detectors can’t recognize if a text is written by AI or a human.

And this problem already exists at several subjects already. A notable example would be poems. While checking out a total of 250 random poems through Copyleaks, all of them were recognized by Copyleaks as AI written. In comparison, less than 10% (19 poems to be exact) were recognized as AI written by Crossplag. Which is closer to truth? Well, Crossplag would be the answer, as quite some of the poems date back hundreds of years…

The reason why poems are detected so often by Copyleaks has likely to do with patterns of writing. In case of most forms of poems, there’s some kind of structure. That’s eventually why we like the sound of a good poem. AI detectors like Copyleaks are very clearly checking on that.

And it’s important to note here, not just poets keep to such structured writing, so do many professional writers, people on the autism spectrum, people with social (pragmatic) communication disorder, and quite a few more people.

Just imagine for a moment, you’re autistic or have SCD. Now you’ve worked hours and hours for some assignment that your teacher gave you. After so much time and effort, you deliver it to your teacher. Then your teacher gives you a failing grade. You’re surprised and ask, “why”. And the response of your teacher is, “because you’ve used AI”. You would get frustrated and annoyed because you knew that you put so much time and effort in your work, all destroyed because some AI detector incorrectly claimed your work was written by an AI.

This is the reality that we are actually entering right now. And the thing is, this is easily resolved in the education system. After all, studies already prove the lacking benefit of homework and writing papers. You could in fact just skip them as a teacher and you won’t need AI detectors, to begin with.

However, this is very different here online. Just imagine writing a blog and all of your content is recognized as AI and you get attacked over it on social media. Because it’s a highly regarded AI detector, such as Copyleaks, people believe you faked your content and you will get shunned and people stop reading your blog altogether. Furthermore, there’s a very high likelihood that you will be blocked by ad networks.

This is the problem we are actually seeing already. And this is not even the worst of it, as AI detectors give different results just based on what you copy. As a notable example, the medical disclaimers in my blog posts, they get flagged as AI written by Copyleaks. However, if I just add the first paragraph after the disclaimer, it’s no longer detected as AI written, not even the disclaimer itself. This shouldn’t be possible if an AI detector was doing what it should be doing.

That’s also why I think AI is less of a problem than AI detectors are. The reality being, those who need AI detectors could get around them by simply adjusting their ways of working, like the teaching example I gave. There’s no actual benefit of them as things are. However, AI does have a lot of benefits in the right applications. As a notable example, there are AI chat bots that help people who experience loneliness.

From my point of view, I would really prefer if AI detectors like Copyleaks were more honest about the fact that their detections are not perfect. As things stand now, there might be some use to AI detectors when people understand that they are only a helpful tool to identify AI, but not the complete solution. You will need to know if there aren’t possibly reasons why there is a false positive. And, sadly, there’s too little attention to this fact…