The limitations of AI detection tools and the importance of human analysis

During these past days and nights, I have been receiving some harassment over my way of writing. This is due to my writing being flagged by Copyleaks as possibly AI-written. Copyleaks confirmed through Twitter that their system isn’t as good at verifying whether poems and songs are AI-written or human-written. This information is stated in their FAQ (Frequently Asked Questions), although it is hidden in a PDF file on their website. Being a young person, I learned in one of my first computer lessons to never click on suspicious files on the internet.

This is what Copyleaks’ FAQ says precisely:

Following this, I could provide a long-winded explanation about how my age, location, first language, and disabilities increase the chances of my writing being detected as AI-written. However, the truth is, I could just as well direct you to the Stanford University article called “AI-Detectors Biased Against Non-Native English Writers” or Jack Caulfield’s “How Do AI Detectors Work?“. From my point of view, there’s something more interesting to consider here.

That is whether you could do what AI detectors do yourself. After all, most AI detectors are programmed to analyze the perplexity and burstiness of someone’s writing. However, you can also do this analysis yourself. In fact, when you input most of Copyleaks’ FAQ into their own software, it identifies it as AI-written. If you were to examine it yourself, you would come to the same conclusion.

An interesting characteristic of most AI writing is the use of short paragraphs, similar to the way I’m writing right now. The reason my text won’t be recognized as AI-written is that I purposely increased the perplexity and the burstiness of my writing. When it comes to long-winded texts, this is not hard to do. This is also why AI detection tools often recommend using long texts rather than short ones.

The fact is, you don’t actually need AI detection tools to determine if a text is AI-written. I can provide an example from Copyleaks’ own website, specifically a testimonial about Safe Search Kids, a blog that provides a Google safe search function. However, if you have a Google account as a child, this function would usually be enabled by default, rendering the website’s purpose pointless.

Here is the popular list of articles on their blog right now:

Did you notice what I noticed? The titles have strange capitalization. If the content creator was from a country where English is not the native language, especially a German-speaking country, this would make sense. However, Copyleaks’ website confirms that the owner of Safe Search Kids is American, which raises questions about the capitalization.

The issue becomes even more evident when checking their content. They have a top 10 list on their website without clarifying the numbers 1 to 10 in any proper way. Any human writer would have noticed this, while an AI wouldn’t. Moreover, on their “about us” page, the same capitalization problem occurs in their actual content when they capitalize “safe search” without any reason to do so. They also switch between writing “safe search” and “SafeSearch,” which only an AI would do, as it would be aware that it’s specifically “Google SafeSearch,” regardless of the normal way of writing “safe search” is just “safe search.”

Additionally, as I mentioned before, if you examine their content, you’ll find that everything is written in short paragraphs. In fact, some paragraphs are so short that it becomes challenging to read the content on their website. It doesn’t feel like their content was written to provide information but rather as a way to improve their search engine optimization and increase their website’s visibility on Google, thereby boosting their ad revenue.

However, the most telling sign of AI is the lack of understanding exhibited in the articles. Here on my blog, there are specific words that I wrote which an AI wouldn’t understand as being relevant to me. A notable example is my poem about my difficult past, titled “Rising Above: A Survivor’s Journey,” where I specifically mention “a newfound family.” The fact that I was adopted isn’t a secret, but I considered it an important detail to include in this poem. An AI wouldn’t comprehend the significance of including this detail. Even if you tried to make an AI do so, it would likely repeat it through the poem because it wouldn’t grasp its importance. Although, perhaps it’s because I don’t know how to make it generate accurate content, as my experience with AI is limited to trying ChatGPT, as I’m not particularly fond of AI and only tried it for this blog post.

Going back to the subject of AI detection itself, laziness is a significant problem for humans in regards to AI detection. Tools like Copyleaks, Crossplag, and ZeroGPT are meant to assist you, but they are precisely that—tools. It’s similar to using a shovel or a spade to dig a hole; they won’t automatically dig the hole for you. You need to put effort into it as well. That’s the major lesson many people need to learn. AI detection may not be perfect, but it doesn’t need to be if you take the time and effort to examine the evidence yourself.

Moreover, I also believe that AI detection tools should not be publicly available. Apart from the fact that they are still in development and can result in false positives and negative consequences, they also inadvertently help AI adapt and become more human-like over time. By being publicly accessible, they are essentially defeating their own purpose.