It's worth noting that some prediction URLs, for instance: <a href="https://longbets.org/6/" rel="nofollow">https://longbets.org/6/</a> are offline.<p>The original long bet was whether or not computers could pass the turing test by 2029: <a href="https://longbets.org/1/" rel="nofollow">https://longbets.org/1/</a><p>You'd think that LLMs fulfill this, but I do wonder if a clever human could still discern between them given their particularities.<p><a href="https://longbets.org/9/" rel="nofollow">https://longbets.org/9/</a> is another interesting one depending on whether you think covid leaked from a lab or not.
><a href="https://longbets.org/6/" rel="nofollow">https://longbets.org/6/</a><p>Anyone know why this (and many other bets) have been removed?<p><a href="https://web.archive.org/web/20080803213625/https://longbets.org/6/" rel="nofollow">https://web.archive.org/web/20080803213625/https://longbets....</a><p>>By 2010, more than 50 percent of books sold worldwide will be printed on demand at the point of sale in the form of library-quality paperbacks.<p>Not exactly a hot-button issue.
There was a paper a while ago, where they found a strategy for GPT beating humans on the Turing test 2/3rds of the time.<p>They just prompted it to talk the way a 13 year old girl texts.<p>(Also they prompted it to pretend not to know anything about anything.)
Most of the LLM tells we usually recognise are due to their heavy attunement to role playing a "helpful assistant" with a knack for correct typography.<p>If you used a smaller corpus, followed a different kind of fine tuning routine with rather normal conversation patterns, and wrote a different base prompt, I don’t see why it shouldn’t be possible to come up with a very particular personality, so to speak.
> where they found a strategy for GPT beating humans on the Turing test 2/3rds of the time.<p>IIRC, PARRY <a href="https://simulation-and-society.org/parry/" rel="nofollow">https://simulation-and-society.org/parry/</a> got there in 1972 by pretending to be a paranoid schizophrenic person
>while ELIZA has a comforting effect that many crave, PARRY often actively makes users uncomfortable and afraid<p>PARRY is my spirit animal.<p>> In empirical evaluations conducted during the 1970s, experienced psychiatrists proved unable to reliably distinguish PARRY's responses from those of actual patients diagnosed with paranoid disorders, achieving only 48 percent accuracy, a figure statistically indistinguishable from random guessing.<p><a href="https://ai.miraheze.org/wiki/PARRY" rel="nofollow">https://ai.miraheze.org/wiki/PARRY</a>
Also I found the source code!<p><a href="https://www.cs.cmu.edu/afs/cs/project/ai-repository/ai/areas/classics/parry/" rel="nofollow">https://www.cs.cmu.edu/afs/cs/project/ai-repository/ai/areas...</a>
I feel like pretending to be a child is cheating somewhat. Might as well print random key presses and claim it's a toddler.
> You'd think that LLMs fulfill this, but I do wonder if a clever human could still discern between them given their particularities.<p>Yes, especially if you use the original adversarial setup with three simultaneous participants (judge, two candidates) as described in Turing's original paper.<p>However, I'm not quite sure how long even that one will last.
Wayback shows #6 as "By 2010, more than 50 percent of books sold worldwide will be printed on demand at the point of sale in the form of library-quality paperbacks."<p>It was removed early 2011. Is this what happens when they can't decide on who won?
>You'd think that LLMs fulfill this, but I do wonder if a clever human could still discern between them given their particularities.<p>While LLMs are very capable of appearing human in general conversation, the test is that they are trying to convince a human that they are human, which is not something that is in their training data and I suspect they would go about things in very odd ways from an actual person, especially when challenged by an actual person. It is still very easy to back LLMs into a corner where they fall into very unnatural patterns and in my experience challenging them on their "humanness" (or anything) backs them into that corner.<p>LLMs also still can't use context or subtext in natural ways and even people who are terrible with purposefully exploiting context and subtext, generally have a natural ability that they developed to use and parse them. In a short conversation that is not adversarial, this is not an issue but falls apart as the length of the conversation grows or they start getting challenged.<p>LLMs just don't play with ideas and conversation like we do, they don't riff on it and go with the flow without care, picking up on subtexts and contexts and manipulating them the way people constantly do without effort until they find themselves on the edge of a cliff looking into the void and debating if they should jump. If we are put into the situation of proving we are human (in an academic test sort of way) we will play with it and have fun with it, and we will find ourselves on that cliff looking into the void, debating if we should expose ourselves and admit things we normally wouldn't. We pick up on this when people do it, even over the internet, their way of stating things changes and their subtext falls away or drastically changes; I have never seen anything approaching this in LLMs.
I don't think any frontier lab tried to make an LLM optimized for the literal Turing test where the LLM is made to argue that it's human. This is not in their interest. They make their LLMs insist they aren't human, have no feelings and sentience etc etc.<p>So we don't actually know how good LLMs can be in this.
That is absolutely true of the chatbots we have access to but I suspect there are people working towards these ends, including all the big names in the field, major win for their company if they manage it. But even then, I have yet to see a model that could naturally handle subtext and context, they struggle even when dealing with programming languages where these things are far simpler than they are in human language.<p>There is also the problem with how easy it is to back them into a corner where they start acting in very unnatural ways. This would arise when challenging them on their humanity and most people would pick up on it and craft their challenges around it until it fell apart because that is how people work in such adversarial situations.
> You'd think that LLMs fulfill this, but I do wonder if a clever human could still discern between them given their particularities.<p>I think the clear winning strategy is to find some of the common sense reasoning fails that are still quite common, like the "should I walk or drive to the car wash" one from a few months ago.
These are usually from a missing context (i.e. specifying where the car is right now). Tokenizer issues (how many Xs are in Y) are a better giveaway, but models can solve them using tools nowadays. I think the best way to uncover an LLM is to reveal its superhuman aspects. Ask it for an A* algorithm for your problem. If it can just type down a working algorithm in a few seconds, that is definitely not a human. Although with some careful system prompt design, you could probably mask that aspect too.
In a true competitive Turing test, these reasoning pitfalls could be included in the system prompt, there are not that many of them. The tester would need to find a novel reasoning pitfall, but those aren't uncommon among humans either, so that would not make a very useful differentiator.<p>Another approach could be to probe the helpfulness of an AI. A human would likely not comply with a request to write a sonnet about tangerines.
By 2029 things will be different. Humans will have learned to talk like LLMs.
The Turing test has been passed. The Turing test had already been passed in 2002. The Turing test isn't about some machine fooling <i>all</i> humans <i>all</i> the time. It's about fooling <i>some</i> humans <i>some</i> of the time. That's an extremely low bar. It has been cleared. Let's put this to rest.
> You'd think that LLMs fulfill this, but I do wonder if a clever human could still discern between them given their particularities.<p>You don't even have to be that clever. We've all tried using it for work. They fail.<p>In the first place, the Turing test was always a loosely defined thought experiment. I suppose it's still serving that purpose, but at least half of HN takes it way too seriously. It's <i>absolutely not</i> proof of <engagementbait> endorsed by Alan Turing himself.
I think if you trained an LLM specifically for the purpose of passing the Turing test (instead of being helpful, harmful, and so on), its likely it would pass it.<p>You would have to train it/finetune it on a couple hundred of 'humans chatting in the context of a Turing test'.
I don't think that's true, assuming that the humans are also allowed to strategize and study the problem beforehand. The main weakness of the current models is that they still fail hard at certain kinds of common sense reasoning about real world situations, that no human ever would. Things like the "walk or drive to a car wash that's 2 minutes away" thing from a few months ago (I think the latest models have patched that one in particular, but I'm sure others can be found in a similar vein).
That would not really pass the test in spirit, I think. Passing must be something that falls out as a consequence of being a good chatbot, not the other way around.<p>However such good conversational agents would still have benefits, and i would be curious how far the approach of building a model that’s nice to talk to can take you, as opposed to the current generation of all-knowledgable helpful assistants.
I think you are absolutely right.<p>The big problem with the Turing test always was that it doesn't take into account adversarial designs. They had those chatbot contests about a decade ago, where the chatbots would regularly pass the turing test, not because the bots were intelligent, but because they were packed with rhetorical tricks designed to avoid saying anything of substance.<p>By any meaningful measure, today's LLM's pass the Turing test. Except we may need to lobotomize them for the deception to work.
I'd like to see an honest attempt at that.<p>What's more interesting are the types of humans that might <i>fail</i> the Turing test. Maybe to discourage the failure of real humans, there could be consequences outside the context of the test.<p>That is, most people accused of being a computer would crack and start pleading their humanity, but some sociopaths might not. I then wonder how many of them overlap with those <i>so invested</i> in abusing the premise of the Turing test.<p>It really is a fun thought experiment when you spice it up enough.
Mannerisms aside, they are easy to spot from having inhuman amounts of trivia knowledge
Couldn’t you instruct them to <i>not</i> display as much knowledge for a test situation…?
Humanity is taking an interesting technological arc.<p>Terminator (1984 film) had a scene showing that in a future 2009, humans would use dogs to try to sniff out whether a robot passing for a human is secretly a machine.[1]<p>In our real world 2026, there <i>are</i> no humanoid machines that can complete basic generic tasks, like carrying a tray across the stage and holding it for 30 seconds.[2] They move slowly and badly (probably from an LLM like neural network doing very few frames per second of correction and analysis), and are nowhere near lifelike.<p>When they don't need a body to pass for a human, such as typing online, they do a bit better.<p>We can tell them apart from humans. As you say, they have stylistic quirks. And you mention that they're easy to spot because they come trained with inhuman amounts of trivial knowledge.<p>[1] <a href="https://www.reddit.com/r/MovieDetails/s/0qUCVPYgjt" rel="nofollow">https://www.reddit.com/r/MovieDetails/s/0qUCVPYgjt</a><p>[2]
<a href="https://www.reddit.com/r/LivestreamFail/s/t6ZV0yhgEe" rel="nofollow">https://www.reddit.com/r/LivestreamFail/s/t6ZV0yhgEe</a>
I would not call Turing's experiment ill-specified, on the contrary among other factors the paper the game is introduced in earnestly calls for a telepathy-proof room to ensure no side channel leakage. The 1950s was an interesting decade.
It won't fool all of the people all of the time. But it succeeds often enough to pass the loose definition. It does many things that we called "AI complete" for decades.<p>It is also clear that is is also not really aware, either. I think Turing would be tickled that we find ourselves in an intermediate state that he would not have imagined.