Some thoughts about Anthropic's new cryptanalysis results

(blog.cryptographyengineering.com)

66 points by supermatou3 hours ago

6 comments

simonw2 hours ago
This is good:> If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that. The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...]> On the other hand: if you think that models are super-intelligent or that AGI is already here, you should also stop thinking that. Working with these tools is like swimming in a pond where the ground drops off sharply. One minute you’re wading comfortably and there’s support under your feet. Then suddenly you cross a specific line, and you’re back to swimming on your own.
- dboreham1 hour ago
 I'm also getting irritated with the “glorified autocomplete” comments. Since nobody can post such comments and also use the tools I'm using, I'm wondering if the phenomenon is due to people only having experience with the free version of whatever it is they're trying to use?
 - dgellow1 hour ago
 The “glorified autocomplete” framing isn’t to take literally. It’s a way to remove the mystic and whole anthropomorphization of AI. It’s saying they aren’t sentient or entities we are interacting with, even if that’s how the output presents itself. Instead they are “just” stochastic models
 - simonw1 hour ago
 Some people use it to demystify, but a whole lot of people seem to be using it to dismiss the technology entirely.Personally I like to remind people that these things are next-token predictors, but then emphasize how truly astonishing the results we can get out of sufficiently advanced next-token predictors are.
 - wren69911 hour ago
 The "next-token predictor" framing is also a bit shaky. It's an accurate description of pre-training, where next-token prediction is a useful learning objective to force the model to learn higher-level representations. It's wildly misleading for a model put through an RL post-training campaign. The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different.
 thesz35 minutes ago
 <pre><code> > The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different. </code></pre> <a href="https://arxiv.org/abs/2504.13837" rel="nofollow">https://arxiv.org/abs/2504.13837</a>"Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns. While RLVR-trained models outperform their base models at small k (e.g., k = 1), the base models achieve a higher pass@k score when k is large. Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model. "
 wren69913 minutes ago
 Ooh. This looks like an interesting paper and there were a couple of things in the intro that I found counter-intuitive. It'll take me a while to digest the whole thing.> Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base modelOn the face of it this seems unsurprising given the policy gradient term directly minimises this difference.I don't have a good feel for how the output of an RLVR-trained model concretely differs from the base model. My guess would be there are a fairly small number of "forks" where the training creates a token flip that sends the model down a more useful path.The fact that the straight paths between the forks resemble the base model would again be unsurprising since (a) those are exactly the right context to continue to elicit more output that's relevant to solving the problem (so not penalised by RLVR), and (b) preservation drops naturally out of the policy gradient term you add to limit catastrophic forgetting in the base model.Low perplexity could be explained by the relative sparsity of the forks in the output stream, and/or by forks already having high entropy in the base model.I think there's room in that forks-and-straights characterisation for the RLVR'd model to be doing something that looks a lot like computation, while having low perplexity vs the base model. I don't see anything in my admittedly incredibly shallow skim of the paper that refutes that.
 adw6 minutes ago
 This paper is less dramatic than you think it is and really just re-explains what RLVR does.Let's stipulate that what pretraining does is train next token prediction over a gigantic corpus. You can then sample from this distribution repeatedly (cf the Large Language Monkeys paper) and count how often it passes some deterministic verifier.What GRPO-style RLVR does is precisely this, but then reward the trajectories which passed the verifier. These distributions are _by construction_ within the accessible output space of the pretrained model; you're reweighting the distribution so that pass@k goes up, because that's (for applications like programming) very useful. RLVR is about making sampling more efficient; the only new information being added to the system is the presence of the verifier, and note that you only get a reward when the verifier passes, so there's essentially no mechanism for "teaching new facts" here.
 simonw1 hour ago
 Right, but it's still useful to think of these models in terms of next-tokens because it helps explain that they look at every token that came before and use that to put out the next one.You can get into RL as part of explaining why it's so unnervingly good at picking a next token.
 wren699148 minutes ago
 That's true. The fact that an LLM is a pure function of (all previous tokens) -> (next token), with internal state like KV cache only existing for optimisation purposes, is pretty mind-blowing.I guess it was more the "predictor" part I had issue with. There's a tendency to reach for statistical or probabilistic terminology to describe things that aren't usefully understood in those terms. For example in the "Speed Always Wins" LLM technical survey (<a href="https://arxiv.org/pdf/2508.09834" rel="nofollow">https://arxiv.org/pdf/2508.09834</a>):> The gate is a crucial component to bring sparsity in MoE models. For a batch of input token representations X ∈ RT×D, the gate function G determines the probabilities of dispatching token xi to each expert e...which is nonsense: the gate simply, directly, selects the experts. There's nothing probabilistic about it.
 - slopinthebag22 minutes ago
 I think this is smart, it seems to me that the best way to use these models its to approach them as a next-token predictor instead of an intelligent entity. That's how I've gotten the best results from them and allows me to avoid some of the pitfalls people fall into by anthropromising them.
 - ameliaquining1 hour ago
 I'm curious, what are you hoping to convey by reminding people that LLMs are next-token predictors? They are, of course, but most people without an AI background won't fully understand what that means, so I assume you're using it at least partly as a proxy for something else.
 simonw1 hour ago
 I think understanding how this stuff works is really important. For technical people it gives them a useful starting point for understanding it all. For less technical people it's crucial to help them understand that it's not some weird new magical science-fiction AI - it's still computer programs that turn text into numbers and do stuff with the numbers and turn those back into text.It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next.
 dgellow1 hour ago
 While you’re active in this thread, I just want to say thank you for all your writing, you’re such a reliable source of sanity in that crazy new world :)
 ameliaquining1 hour ago
 I don't think the next-token-predictor thing should increase anyone's confidence that LLMs aren't conscious or can't escape the control of their operators. A very closely analogous argument would "prove" that humans aren't conscious or can't do [insert task here] either. (No, I'm not saying that any of this is true of today's LLMs, I'm saying this particular argument doesn't work.)I recommend this explanation: <a href="https://www.astralcodexten.com/p/next-token-predictor-is-an-ais-job" rel="nofollow">https://www.astralcodexten.com/p/next-token-predictor-is-an-...</a>
 dgellow57 minutes ago
 You can say that for any argument regarding consciousness, because we don’t have an actual, all encompassing definition of what consciousness is. In general I don’t think comparison with humans makes much sense, we should be able to discuss LLMs without always falling back to “but what about humans” (sorry for the caricature)
 cyanydeez3 minutes ago
 I think it's important to understand the humans _can_ do what LLMs do: predict next tokens from prior ones.But LLMs are only operating on text and humans are only operating on <waves hands>
 ameliaquining52 minutes ago
 Shouldn't that imply that agnosticism is the proper view, rather than asserting that something is impossible on a next-token-predictor architecture?(Note: I don't actually think the consciousness question is the most important one in the near term. Where I think this line of reasoning gets really dangerous is when people use it to assert that LLMs can't or won't engage in certain behaviors no matter much they advance; this doesn't have anything to do with consciousness.)
 slopinthebag15 minutes ago
 I think you can reliably assert that X != Y without having a complete definition of Y, as long as you can identify at least one property or condition that Y possesses which X violates.So for consciousness and LLMs it could be Qualia, lack of semantic understanding, lack of continuity in time, lack of a high degree of integrated causal feedback, etc.Or perhaps those are just features of human consciousness but not integral to consciousness as a whole. To me this then implies panpsychism to some degree, which I'm alright with too.
 eterm58 minutes ago
 Then people took these next-token predictors and fed their output in to weapons systems. Irresponsible but perhaps also inevitable.
 - argee1 hour ago
 Exactly. They ARE "glorified autocomplete" in an ontological sense. That says nothing about capability or outcome. The people who come out swinging against that characterization usually ignore the whole ontological argument (which is...the entire point) and go after an outcome-based strawman.
 - ameliaquining43 minutes ago
 Isn't the outcomes question the one that people actually care about in most contexts?
 argee31 minutes ago
 Depends. "Most" implies majority, and the majority of people are using these tools not for programming but in contexts where ontology is more relevant than capability (not that capability is irrelevant, but most people care, or are tricked into caring, far more about the former).
 ameliaquining25 minutes ago
 Sorry, what contexts are these?
 argee9 minutes ago
 Sorry, this discussion is too incommensurable for me.
 - cyanydeez12 minutes ago
 I like glorified copy/paste frankensteined with find/replace.
 - mostlylurks39 minutes ago
 The models themselves are indeed glorified autocomplete in terms of what they actually do (with things like agentic coding harnesses being required as a wrapper around them to make that internal autocomplete something more useful). Many people use this fact to critique LLMs, but many other common instances of people pointing out LLMs' apparent lack of intelligence actually come from people not understanding that the model is a glorified autocomplete underneath whatever interface people access them through, and the interface isn't providing the underlying model all the information they assume it would, making it seem less intelligent than it actually is.
simonw2 hours ago
> both outputs of Claude Mythos, their (still) unreleased advanced modelThat sentence gives the impression that Mythos might be released in the future. That's clearly not going to happen - it's already "released" in as much as selected, trusted partners can access it, and the rest of us get it in the form of Fable - which is Mythos but with filters that downgrade you if you try to use it for anything even remotely related to cybersecurity or biology.(The other day Fable 5 downgraded me to Opus after I asked it to explain the difference between tusks and teeth.)
- free_bip2 hours ago
 Genuine question here, why would you ask Fable to explain the difference between tusks and teeth? That's a task that can probably be handled by Haiku.
 - simonw2 hours ago
 It's the default when I pop open the Claude iPhone app, I usually don't bother to switch it.
john_strinlai2 hours ago
>They [anthropic] appear to have just told it to get some results and then strapped its nose to the grindstone until it found some.it is fun how well this works.i cant find the link immediately (will look and edit with it), but somewhere in the "hello there the jacobian conjecture is false thanx" thread, someone brought up a different conjecture breakthrough where the prompts were basically just repeated "no, keep going" until a result was found.edit: <a href="https://chatgpt.com/share/6a60b2eb-0b64-83ee-9c76-7931ca1de063" rel="nofollow">https://chatgpt.com/share/6a60b2eb-0b64-83ee-9c76-7931ca1de0...</a>i especially like "you should do a breakthrough". each prompt is less than ~20 words. makes me really question the whole "prompt engineering" stuff.
- alwa1 minute ago
 Oh boy. The human here is effectively assuming the role of a Magic 8 Ball…What a weird species of halting problem…
- throwup2382 hours ago
 > i especially like "you should do a breakthrough". each prompt is less than ~20 words. makes me really question the whole "prompt engineering" stuff.This is well past prompt engineering and into process engineering like six sigma. Just like in an early industrial revolution factory, we’re all still figuring out what works in the process of making stuff except this is so early that even simple things like “make this screw standardized” (or “no, keep going” in this case) is really high impact.The degrees of freedom an LLM has is so large that we're going to be exploring their capabilities for decades, especially if they continue to get better. This is why IMO experts are always going to be better at LLMs in their field because they can force them LLM into processes (think prompt engineering -> CC dynamic workflows) that follow their work processes and get much better results out of them than “keep going.”
- adamzenith1 hour ago
 In response to your edit, you should check out Terry tao's chat gpt logs about the recent Jacobian result. The models are smart enough to brute force some things, but can cut to the meat much faster with good prompting
 - john_strinlai1 hour ago
 i read his, too. his replies are indeed more directed, but also quite short, unstructured, and natural sounding. if i recall, maybe 1 or 2 of his prompts exceeded 50(ish) words.in my head, the comparison is the multi-paragraph prompts (borderline essays) i would read in various communities on reddit and similar forums, that people (often self-proclaimed "prompt engineers") said were "required" to get good output. or some of the prompts ive read in various logs that are like a thousand words of setup.even looking back at the first prompts i was sending when i started to use chatgpt were (in hindsight) crazy long and full of unnecessary guidance/caveats/"ignore xyz"/etc.
- dist-epoch1 hour ago
 That second person stated that for many years they tried that particular graph problem on various AI models, starting with o1 and o3.Its quite likely they now found the counterexample with a more serious prompt, and then for virality re-tried a few times with meme-prompts like "you should do a breakthrough", knowing that the model is capable of solving this particular one. Worst case the meme-prompts don't work and they share the real one they initially used.
 - john_strinlai1 hour ago
 >Its quite likely they now found the counterexample with a more serious prompt, and then for virality re-tried a few times with meme-promptsi am not sure why this is "quite likely". it'd be pretty silly to get a mathematical breakthrough and then hide it for an undisclosed amount of time to get a few more likes on a tweet, when the impressive part is the breakthrough.not saying your theory is impossible, but i think the simple answer is that the model is just smarter than o1 and o3.and, in any case, the model ended up getting the result with the meme prompt and "keep going", which was what i find fun. just like how the crypto results were from prompts of, more or less, "keep going", and that's pretty damn cool.
 - dist-epoch1 hour ago
 You can open 10 tabs and prompt 10 times. We are talking about a few hours delay.I agree with you that obviously no prompt engineering was needed just "solve this problem", but imagine it was you doing this problem with every model, wouldn't you have tested a new model with the best prompt you had from previous iterations, maybe with some partial previous results in it, exactly to maximize your probability for a mathematical breakthrough?
mkagenius2 hours ago
I also just finished writing a note - <a href="https://mkagenius.substack.com/p/notes-on-mythos-breaking-aes" rel="nofollow">https://mkagenius.substack.com/p/notes-on-mythos-breaking-ae...</a><a href="https://news.ycombinator.com/item?id=49099977">https://news.ycombinator.com/item?id=49099977</a>
throawayonthe2 hours ago
> I asked Claude for its thoughts, and it doesn’t mince words: “what makes this genuinely interesting — and, frankly, a little embarrassing for the field — is that none of the ingredients are exotic.” The TL;DR is that someone just did a much more thorough job applying all of our known tools. In short: the sort of things that attack AIs are wonderful at.did i just read two summaries/TLDRs (in a row) of the already-two-sentence summary right above?
- Ar-Curunir1 hour ago
 There a constructive way to leave feedback you know