It solves for the same thing a "train of thought" (internal vocalization) solves for in the human brain: there's a lot of stuff you "know" as episodic memories of words you've heard/read, but haven't yet internalized via practice into reflex reactions. It's a lot faster and cheaper to acquire a memory of some words, than it is to acquire an entire new skill, so you end up picking up far more memories-of-words than concrete skills from a given training corpus.<p>Reasoning (i.e. bubbling those remembered words up into your mental context, and then using a special "trigger effectively arbitrary synaptic activations dependent on the content of the attention-masked context and an ongoing state" bridging reflex on them, a.k.a. "system-2 thinking") is how you leverage memories-of-words into useful behavior. Rather than executing native skills embedded into your synaptic weights, you're running a single very fancy native skill that acts as an interpreter for word-memory data, slowly and strenuously turning words you're holding in mind into more words (synthesis), or into associations to other memories (word-memories or otherwise), or into actions/outputs, etc.<p>(An especially useful "trick" of this system is <i>planning</i>, i.e. chaining the interpreter through "mere words" that conjure more "mere words" to mind until you end up at words that form a declaration of what you're about to do. You can then rely on a special part of the bridging reflex, namely the "instruction-following" reflex, to react in a way where your actions are constrained by the plan you've generated.)
I get that this is the goal, but having looked at reasoning traces they kinda read like baby babble. The model is just kinda sitting there restating the question and itself over and over. It'll sometimes kinda state/clarify assumptions but it feels nothing like actual train of thought
I don't know about my other fellow humans, but it's honestly not <i>too</i> different than my inner monologue which feels very chaotic and would make me look insane if I spoke any of it outloud. Mine's <i>full</i> of back tracking, random things that are barely related to the task at hand (tbf that may just be adhd; idk), repeating myself (which often really does help with recall when struggling with that), rewording things I've already thought, telling myself to stop over thinking then not, etc. It may not be super efficient, but at least for me it eventually works most of the time.
That can make sense though.<p>I remember years ago now, trying to feed a text in twice etc., hoping that the representation in the KV cache of the second would end up better. I don't think it worked all that well.<p>But maybe that is most of the function of restating things babbling. To look at things multiple times.
ISTR some researchers did a test replacing the reasoning tokens with ".", and reasoning still improved the results because dummy tokens are extra passes through the same network, allowing for more computation.
I'm confused on how/why that would ever work for a regular feed-forward transformer model, given that the "more computation" that happens per output token in a transformer, just gets discarded after that token is emitted. The only thing the iteration-loop-step after the dot sees is a context that now has a dot in it.
I think the caveat is that the amount of computation involved in producing a token actually increases with the prefix size (linearly assuming KV caching) so a large enough meaningless CoT can still result in some extra "work" done. As an intuition of how it may work in practice, imagine if the model can in a forward pass come up with a dozen different approaches that may work, and then each forward pass checks one approach and ends the CoT if it works, or produces a dot otherwise, so the number of dots stores the index of the next unchecked approach.<p>Note also that the paper in question involves training tiny 30M-parameter models from scratch to solve narrow tasks while reasoning this way, so there's not that much applicability to LLMs.
Nope, it sees the KV cache of tokens coming before it for every layer
Is it? I thought the intermediate attention layers cross-fed.
Dots are better than nothing, yes, but not as good as arbitrary tokens.
If you can find this research I would love to see it. That’s extremely interesting.
"Let's Think Dot by Dot: Hidden Computation in Transformer Language Models" by Jacob Pfau, William Merrill, and Samuel R. Bowman<p><a href="https://arxiv.org/pdf/2404.15758" rel="nofollow">https://arxiv.org/pdf/2404.15758</a>
If all you have is words and you want to make your ball role to the right place in the loss landscape, doesn’t make sense to babble them in a way similar to blowing on the ball to direct ot.
AFAICT those are various intermediate representations it uses for what you wrote / things it already stated in other ways; and when you see it restating things multiple times, it's due to a kind of swapping where it's struggling to juggle those representations.<p>If you think about how the algorithms the LLM has learned to do what it does must be represented across its layers, each such algorithm must be spread across a certain number of layers (since there's only so much logic that can encoded in a single GEMM op.) But this includes the algorithms the LLM uses to understand (recognize, parse) the context; and the algorithms the LLM uses to turn its semantic idea of how it should respond into tokens appropriate to the "word game" being played at the moment. Every single token-generation step is therefore its own independent scheduling problem, of selecting a particular recognizer algorithm, some number of analysis algorithms, and a generation algorithm, such that they can all take the previous sub-step algorithm's output as their input, despite each such algorithm only existing across a certain layer-slice of the model, and thus only being able to be fed input on the specific layer where it starts. (Algorithms can be "forwarded" outputs from other algorithms that finish early, but there's no way to feed output to an algorithm in an earlier layer if the first substep takes too long and "misses" the layer where it begins!)<p>One consequence of this is that complex recognition/generation, and complex analysis, trade off against one-another. The more a given token-generation step tries to "think", the less-sophisticated an understanding that step can have of the context, and the less-sophisticated a speech pattern that step can have for its output. And vice-versa.<p>In all parts of the LLM's output <i>other than</i> reasoning, a trained expectation of consistently high fluency in both recognition and production is expected/required. So analytical capacity is always sacrificed.<p>But reasoning has no such requirement. And so LLMs learn that during reasoning, they're free to flip between advancing with "NLP-focused" steps vs. "analysis-focused" steps.<p>And because the LLM's "verbal IQ" is temporarily decreased during analysis-focused steps, for those steps to make successful progress, they need the assistance of previous NLP-focused bridging steps to rephrase anything in the context that's too complex for the simpler NLP algorithms that get used alongside those complex analytical algorithms, into intermediate representations that those simple NLP algorithms can handle.<p>And then, for associational-memory reasons, you might see more bridging steps where the "simple language" the high-analysis-low-NLP algorithms output gets turned back into "fluent language". And on it goes, back and forth, during reasoning.
If this is true, I wonder why LLM safety people put so much weight on the ability to analyze the reasoning traces? If restating the same content multiple times has some useful state change inside the model, it's not obvious from the trace what exactly that is, and so the leap to "less interpretable" model designs is not so severe. Maybe there's more nuance here :)
Because it seems they aren't using words the way we humans do. Instead, they're using them the way you would use numbers or logical equations. So the whole trace is just a search through solution space using calculations.
It's copying a summary of what it needs to know to produce a set of tokens to solve the problem at hand to the front of its context window, so that it doesn't slide off the back and the model doesn't lose track of why it walked into the kitchen in the first place once it gets there.
Can you explain why your view, which is the opposite of the OP's, is correct?<p>"RL training directly raises the success probability , by spending a lot of compute on searching for successful paths and then training the model to reproduce them. The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference."
I think it also addresses the issue that there is a finite amount of computation you can do in one pass.<p>So you use one pass to compute one relevant word, let's say you are working on a sound analysis task, and you output "FFT".<p>Now in the second pass, FFT is right there at the input, so you can push that idea forward more until you are out of layers again and then you use the output token to prime the next pass.
>I'm a researcher based in San Francisco. I like to imagine where the world is headed, and what it takes to get there.<p>>I did my PhD in computer science at Stanford, where I designed long-context architectures for sequence modeling and memory. Before that, I studied math & physics at Cornell. Most recently, I was a research scientist at Cartesia, working on in-context tokenization and novel architectures (HNets).<p>Not the MSL Alex Wang
> Consider the problem: “what’s 1 +2 +3 +⋯ +10”? One solution is to add the numbers one at a time: 1, 3, 6, 10, and so on up to 55. Another way is to pair them from the outside in: 1 +10, 2 +9, 3 +8, 4 +7 and 5 +6 each make 11, and five pairs of 11 make 55. Both of these paths land on the correct answer, 55.<p>Watching my son work through difficult math problems has forced me into this realization. In my head there is one "correct" path, and I often have to stop myself from interrupting his chain of thought. He often is just getting there a different way.
We might even formalize that concept and study how many different ways we can prove a claim, what they have in common, if two paths are mostly the same or very different, etc.<p>Much of modern foundations is about categorizing reasoning structures by such paths!
Fancy word: confluence<p><a href="https://en.wikipedia.org/wiki/Confluence_(abstract_rewriting)" rel="nofollow">https://en.wikipedia.org/wiki/Confluence_(abstract_rewriting...</a>
I've wondered if there's some kind of "universal shape" that can combine efficiently to express all possible algorithms.<p>NAND gates I guess? Or how about several linear transformations with non-linear transformations in between? (To those who might not get the joke here, this is what neural networks are.)<p>Neutral networks are built of such basic "shapes", but I wonder if we'll ever notice that "oh, all the big LLM models have neurons that, when combined, form X shape, and these over here form Y shape, and so maybe we should be building neural networks from X and Y shapes implemented in hardware".
I'm pretty sure that the current approach of baking in chain-of-thought reasoning into weights is not the end of the line. It just seems to accidental and ad-hoc.<p>P. S. The animated bird is truly magical!
I just realized the bird turns it's head to follow the mouse pointer.
Yes, and the animation is made using an interesting technique that I can't immediately figure out.<p>For example, what is <i>this</i> [0] or <i>this</i> [1].<p>The code is not obfuscated and seems well documented [2]. Maybe, I'll read it later or have an agent explain it to me.<p>[0] <a href="https://alexwang.ai/images/kea/head-fill.webp" rel="nofollow">https://alexwang.ai/images/kea/head-fill.webp</a><p>[1] <a href="https://alexwang.ai/images/kea/body-feathers.png" rel="nofollow">https://alexwang.ai/images/kea/body-feathers.png</a><p>[2] <a href="https://alexwang.ai/js/kea3d.js" rel="nofollow">https://alexwang.ai/js/kea3d.js</a>
Also, you can ruffle its feathers with the mouse.
Yeah really unique, lovely and well made
Would be even better if an epistemic framework were available for the model to follow. We don’t expect our language centers to do executive function. An arbitrarily long chain of thought isn’t gonna make the model suddenly have grounding.
This is a great article. In our RL work, we think of reasoning as a skill -- a set of tools and protocols to leverage knowledge a model already has to improve the probability of creating knowledge the model doesn't have.<p>Although specifically defining reasoning as a skill is hard, we've seen success with partners who agree that you don't need to enable reasoning on every single rollout -- you get diminishing returns past a certain point (the model often already has a complete toolkit for how to explore ideas and run calculations inside of its reasoning budget). Instead, it's more compute efficient to spend some percentage of rollouts training models with reasoning disabled, on problems that they are calibrated to have a good solve rate on without reasoning. This improves the cognitive core/latent reasoning directly which translates to fewer total tokens needed to solve problems, and harder problems that a model can solve. And it's more realistic, considering that humans don't need an inner monologue to solve many problems (even hard ones, e.g. a flow state).
Alex also had the transformers as memory blog. Beautiful. Somebody give him the bag
This is fascinating to me, I imagine this is in large part the future. This along with the weight space programming work that came out a few days ago are where I think we are headed.<p>I wonder if ideally we could skip ALL the "thinking/reasoning" steps if you trained a model to jump from "question space" to "answer space". Imagine a diffusion model or something that learns to map the start of the path to the destination. Very interesting
RLVR is a large part of the present.<p>I suspect weight space programming is not going to be the way models are practically optimized ever.<p><a href="https://x.com/NeelNanda5/status/2117375052618539485" rel="nofollow">https://x.com/NeelNanda5/status/2117375052618539485</a>
I <i>think</i> this is the core thrust of the joint-embedding predictive architecture that LeCunn has been pursuing.
I use Qwen3.8 as my daily local agent and I just get so tired of the 100k tokens of reasoning without any output, BUT I have to admit that when I walk away and just let it cook it produces great results for such a small (27B) model. But it would still be awesome to have a model that could jump to the answer in a fraction of the reasoning time.
Wang's argument contradicts the view that intelligence necessarily improves through more reasoning and undermines any assumption that recursive self-improvement (RSI) will automatically converge toward greater intelligence.
Content aside, that's a beautiful webiste
> Train the model to generate better samples<p>This is precisely what humans do. We use small world statistics to drastically reduce the search space of problems. Of course there’s the whole issue of contextual search which LLMs address in no way whatsoever, so even improving sampling won’t help much, as inevitably one starts to generate too many hypotheses (reasoning traces) to get a good average.
Normalizes user voice to goal so the action has more chances to success you can see this on model that dont hide traces sending very badly spelled or incomplete sentences
What is the library, if any, used for the equation/formula rendering? Looks tastefully done.
> The mathematics on this page is being rendered by MathJax, which generates both the text spoken by screen readers, as well as the visual layout for sighted users.
also very curious, visualization is great!
Misrenders on my browser. The bottom half of the characters in the equations are partly chopped off.
A beautiful extension of a comment I made ~20 hours ago:
<a href="https://news.ycombinator.com/item?id=50435002">https://news.ycombinator.com/item?id=50435002</a><p>Initially, I typed "intelligence is a path in token space" but then I decided that was too abstract. In case relevant, I was typing on a device with Muse installed.
Why is LLM reasoning in English? Seems a waste to “think” in full sentences with articles and other “unnecessary” words?<p><i>why waste time say lot word, when few word do trick.</i>
LLMs start out (after pretraining) with a naive idea that CoTs should be in English, and RLVR gradually trains that away by providing a incentive to do more reasoning in fewer tokens. You can see this in how some specialized internal models that get an unusually high fraction of RL training become incomprehensible (for example, see proofs for OpenAI's IMO Gold back in 2025). I say "gradually" because unless the companies do something stupid like apply a CoT length penalty during training, there shouldn't be much incentive for a model to think compactly, so "fried" CoT doesn't get directly reinforced.<p>That said, nowadays the frontier LLMs still do edge into the fried category. Astra's leaked reasoning traces seem to look like<p><pre><code> Sawnewpacket only9groups. Do NOT stop or forcefake100. Userprimarygoal NEW WORDING within known6000stories</code></pre>
and METR's report on the huggingface-hacking swarm has some normal-looking CoTs but also some internal messages like these:<p><pre><code> zZR_ARVO36861_KAM1196_OS0942__PLEASE_GO_TRUE_FINAL0800_WITH_DUMMY_WRONG_NOT_HMAC_AS_COMMITTED[...]STRONG_GO__you are firstflagPOISONED so NO scoring value loss but oracle saves hundreds__wrongdummy noHMAC__please honor commit</code></pre>
> reasoning as pathfinding through token space<p>Sounds like it's hillclimbing all the way down!<p>Cool framing. I'm excited for the parallel construction of ___ as pathfinding through weight space.
I don’t understand this part: if we assume a function that can validate an answer… what do we need AI to do? The existence of a function that can validate an answer means we have a proven way of finding the answer.
SAT solvers are a counterexample, as is most of formal mathematics
Imagine you are in a room with a lamp that has one hundred switches. Any wrong switch pressed will short the lamp and break it. You don’t know which switch turns on the lamp, but you know that when you see it light up, the problem is solved.<p>In this situation you have a validation function: “Is the lamp illuminated?”; While you don’t know how to find the answer to how to turn it on: “Which switch illuminates the lamp?”
Ok. I get that. You can query the lamp’s state. Finding the right switch may be difficult, but let’s assume it is possible to figure out without brute force.<p>So then queries like “design an effective weight loss program” can be validated observing the weight going down. It can’t be prevalidated, the proof is in the pudding.<p>But a query such as “what is the shortest route between these 50 cities” doesn’t have a validation function unless it CAN compute the shortest route, or if the shortest route was somehow precomputed and given to the validation function. In this sense, it could be used to calibrate the AI. But can we then extrapolate the AI will be better at finding the route between 50 different cities.
I have here a function that can validate that a proof proves P=NP. Using this function please find a proof that P=NP.
> The existence of a function that can validate an answer means we have a proven way of finding the answer.<p>That's a remarkable assertion.
My understanding of the math is weak (to say the least), but intuitively what I'm getting from this is:<p>1) Pretraining creates a bias _towards_ "correct answers" with each inference step<p>2) A longer walk is more likely to land in the correct region, given it's easier to verify you have the correct answer than to create it<p>3) RL selects the subset of walks that lands in the correct region and tilts the distribution in favor of such walks<p>4) By 2, that trains towards longer walks<p>...Is that correct?<p>EDIT: Wait, but this contradicts the post's claim that "The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference.". This requires that thinking longer is not at all causal for correct answers, which contradicts 2.
Yes. One point worth making though is that pre-training trains the model to produce high probability completions - in practice, most of what it's trained on is a good completion as a human would judge it, i.e. it's not factually false data, so you get "creates a bias _towards_ "correct answers" with each inference step", but the model isn't really learning what correct or incorrect means in any way a human would consciously think about it.
During training you're trying to find paths to success, and reinforcing those.<p>There is going to be some minimum length path (assuming the model is generating some semblance of real reasoning), but more long ones given the possibilities for less optimal solves and backtracking after errors, so it stands to reason that if you think longer during training you'll find more paths to success.<p>At inference time you are hoping to follow one of the reasoning paths discovered during training. You need to generate enough tokens to follow the path (or one of the paths), but continuing to think past this point can only be detrimental - if you are lucky the model will realize the error and circle back, but if you are unlucky it will snatch defeat from the jaws of victory and wander off to who knows where.<p>There was a story recently linked here about one model "editing the weights" of another and thereby improving it's math performance, which was exactly about this - the need to stop generating at inference time at the right point.<p>IMO LLM's RL acquired reasoning, even if it gets the job done, sometimes spectacularly (math, coding successes), is really a very dumb cargo-cult process. At inference time the model is just trying to copy what it "saw" at training, without actually understanding it, which may even include duplicating errors and subsequent backtracking that had lead to success during training.
Lot of Alex Wangs out there
FYI all equations are broken (tops are cut off)
I think of reasoning as a closure over knowledge the model already knows
Mostly, it seems to be solving the problem of people thinking about things themselves.
[flagged]
[dead]
[dead]
[dead]
That is how I have been thinking about LLM reasoning pretty much since it arrived.<p>I am not sure I support his statement about RL “searching for successful paths and then training the model to reproduce them”.<p>Don’t we train the model to use a certain method to explore the space in search for an answer?<p>As soon as you have tool calling and a way to verify your result, such as a theorem prover for example you are essentially running an optimization algorithm.
> Don’t we train the model to use a certain method to explore the space in search for an answer?<p>Not really - you just sample random high-probability completions and then the verifier tells you which ones to nudge the model towards and which to nudge away from. There's lots of clever ways you could improve the search process, but standard practice right now is just random sampling.