Scientists dispute whether computer 'Eugene Goostman' program passed Turing test

A computer program named Eugene Goostman that which imitates a Ukrainian teenager with a quirky sense of humour and a pet guinea pig has won an artificial intelligence competition at the Royal Society in London.

The program convinced 10 out of 30 judges at the nation's most prestigious scientific institution that it was a real person in a series of online chats lasting five minutes each.

The event's organisers, from Reading University, claimed Eugene had made history by passing the Turing test, a significant goal in the field of artificial intelligence, though other scientists begged to differ.

Regardless of the program's success, the latest performance was an improvement on Eugene's past attempt to win an AI competition in 2012 when it expressed its love of Eminem and hatred for Britney Spears and Star Wars, and mentioned a pet guinea pig which could squeal Beethoven's Ode to Joy.

Proposed by Alan Turing, the wartime codebreaker and computing pioneer, the Turing test challenges computer scientists to create a program that is indistinguishable from a person in its conversational ability. The goal sidesteps more obscure questions about the nature of the mind, and focuses attention on what it produces and how it behaves.

Computer scientist Vladimir Veselov began work on Eugene in 2001, a year after leaving his home in Russia for the US.

The program analyses questions it receives, and searches a "knowledge base" for material before compiling a response. Some of the time it will ask a clarifying question, or draw on a stock response from memory.

During the tests each judge sat down at a pair of computers and typed in questions. One computer was linked to another with a person at the keyboard, while the other was running a program that provided replies.

The judges included Lord Sharkey, who campaigned for Turing's posthumous pardon over a conviction for homosexuality, and Robert Llewellyn, who played a neurotic robot called Kryten in the television series Red Dwarf.

Declaring that Eugene had passed the Turing test, Prof Kevin Warwick of Reading University said it was fitting that such an important landmark had been reached at the Royal Society.

But one judge who took part in the tests, Prof Aaron Sloman, a philosopher and researcher on artificial intelligence at Birmingham University, was unimpressed. Sloman said he took part in the experiment to see how much progress had been made with so-called "chatbots" after creating one with colleagues some years back. Speaking about Eugene, he said: "It has kept some – not all – who try it out entertained for more than five minutes. But it is essentially stupid and incompetent, no matter how many people it fools for how long."

Stevan Harnad, professor of cognitive sciences at the University of Quebec in Montreal, said that whatever had happened at the Royal Society, it did not amount to passing the Turing test. "It's nonsense, complete nonsense," he said. "We have not passed the Turing test. We are not even close."

In 1950 Turing predicted that in about 50 years' time computer conversations could pass as human around 30% of the time. But he said that a statistical survey like a Gallup poll to decide if a machine could think was absurd.

"Turing's insight was that the way to explain how the mind works is to design a system that can do whatever the mind can do," Harnad said. "That includes all of our verbal capacity, as well as the sensorimotor, or robotic, capacity in which it is grounded. Not for five minutes, but for a lifetime."

John Denning, who worked with Veselov on Eugene, defended the program, which he said could run on a normal laptop. "I think we passed 'a' Turing test, but I don't know if it's 'the' Turing test," he said. "Is Eugene smarter than a person? No. You're not going to put your life in the hands of a 13-year-old who makes wisecracks and has an odd sense of humour."

News of Eugene's success crashed the server it was hosted on over the weekend. Asked if it marked the rise of the machines that would spell the end of humanity, Denning said: "We have been looking at logs of people chatting with Eugene. What people say does not bode well for the future of humanity. It's pretty startling what people will say to robots. People say paedophilic things, things about Eugene's lineage."

Marvin Minsky, one of the most revered names in artificial intelligence, told the Guardian: "Nothing is learned from poorly designed 'experiments'. Ask the program if you can push a car with a string. And, if not, then, why not?"

guardian.co.uk © Guardian News and Media 2014

["A Cyborg Face Gradually Fades Into Some Binary Code Stream. Digital Illustration." on Shutterstock]