1.5 million people have already taken part

A machine is intelligent if it can make us believe in communication that it is a human being. The great British mathematician and computer scientist Alan Turing (1912-1954) established this yardstick for artificial intelligence back in the 1950s – at that time it was still a purely hypothetical mind game.

With the ever-improving capabilities of text-generating AI like ChatGPT, Turing’s idea is more relevant now than ever. As part of a research project, the AI ​​company AI21 has started what it says is the largest Turing test experiment to date – in the form of a game that every user can try.

The test has been online since mid-April, and now the Israeli company has the first ones results announced. According to AI21, 1.5 million people have already tried the “Social Turing Game” called “Human or not?”

The experimental arrangement of “Human or not?‘ is very simple: you should chat for two minutes with a randomly assigned partner, then you should decide whether you communicated with a machine or another human test subject. The chatbot is based on Large Language Models (LLM) such as GPT-4 from OpenAI or Jurassic-2 from AI21 Labs.

As the results so far show, 68 percent of the players correctly decided after the two-minute conversation whether they were chatting with a human or a bot. According to AI21 Labs, this was the result of the evaluation of the first two million rounds.

It was easier for the testers to correctly identify a fellow human being. Here 73 percent guessed correctly. If the subjects were dealing with a bot, they were only 60 percent correct.

AI21 Labs has also broken down the results by country. Users from France did best with an average of 71.3 percent correct assignments. India is at the bottom with 63.5 percent. Germany is tied with the Netherlands in third place (68.1 percent).

Women did marginally better than men, at 67.7 versus 67.4 percent.

Incidentally, “Human or not?” differs fundamentally from the actual Turing test in one respect: In the classic arrangement, a test person communicates blindly with two interlocutors. One is a human and the other is a machine. Now the subject has to decide who is who. If this is not successful, the test is passed and the machine is considered equal to humans.

Whether the Turing test has already been passed is considered controversial, despite allegedly successful tests (e.g. with the chatbot Eugene Goostman).