This is a seriously beautiful guide. I really appreciate you putting this together! I especially love the tab-through animations on the various pages, and this is one of the best explanations that I've seen. I generally feel I understand grammar-constrained generation pretty well (I've merged a handful of contributions to the llama.cpp grammar implementation), and yet I still learned some insights from your illustrations -- thank you!<p>I'm also really glad that you're helping more people understand this feature, how it works, and how to use it effectively. I strongly believe that structured outputs are one of the most underrated features in LLM engines, and people should be using this feature more.<p>Constrained non-determinism means that we can reliably use LLMs as part of a larger pipeline or process (such as an agent with tool-calling) and we won't have failures due to syntax errors or erroneous "Sure! Here's your output formatted as JSON with no other text or preamble" messages thrown in.<p>Your LLM output might not be correct. But grammars ensure that your LLM output is at least _syntactically_ correct. It's not everything, but it's not nothing.<p>And especially if we want to get away from cloud deployments and run effective local models, grammars are an incredibly valuable piece of this. For practical examples, I often think of Jart's example in her simple LLM-based spam-filter running on a Raspberry Pi [0]:<p>> llamafile -m TinyLlama-1.1B-Chat-v1.0.f16.gguf \
> --grammar 'root ::= "yes" | "no"' --temp 0 -c 0 \
> --no-display-prompt --log-disable -p "<|user|>
> Can you say for certain that the following email is spam? ...<p>Even though it's a super-tiny piece of hardware, by including a grammar that constrains the output to only ever be "yes" or "no" (it's impossible for the system to produce a different result), then she can use a super-small model on super-limited hardware, and it is still useful. It might not correctly identify spam, but it's never going to break for syntactic reasons, which gives a great boost to the usefulness of small, local models.<p>* [0]: <a href="https://justine.lol/matmul/" rel="nofollow">https://justine.lol/matmul/</a>
This is a fantastic guide! I did a lot of work on structured generation for my PhD. Here are a few other pointers for people who might be interested:<p>Some libraries:<p>- Outlines, a nice library for structured generation<p><pre><code> - https://github.com/dottxt-ai/outlines
</code></pre>
- Guidance (already covered by FlyingLawnmower in this thread), another nice library<p><pre><code> - https://github.com/guidance-ai/guidance
</code></pre>
- XGrammar, a less-featureful but really well optimized constrained generation library<p><pre><code> - https://github.com/mlc-ai/xgrammar
- This one has a lot of cool technical aspects that make it an interesting project
</code></pre>
Some papers:<p>- Efficient Guided Generation for Large Language Models<p><pre><code> - By the outlines authors, probably the first real LLM constrained generation paper
- https://arxiv.org/abs/2307.09702
</code></pre>
- Automata-based constraints for language model decoding<p><pre><code> - A much more technical paper about constrained generation and implementation
- https://arxiv.org/abs/2407.08103
</code></pre>
- Pitfalls, Subtleties, and Techniques in Automata-Based Subword-Level Constrained Generation<p><pre><code> - A bit of self-promotion. We show where constrained generation can go wrong and discuss some techniques for the practitioner
- https://openreview.net/pdf?id=DFybOGeGDS
</code></pre>
Some blog posts:<p>- Fast, High-Fidelity LLM Decoding with Regex Constraints<p><pre><code> - Discusses adhering to the canonical tokenization (i.e., not just the constraint, but also what would be produced by the tokenizer)
- https://vivien000.github.io/blog/journal/llm-decoding-with-regex-constraints.html
</code></pre>
- Coalescence: making LLM inference 5x faster<p><pre><code> - Also from the outlines team
- This is about skipping inference during constrained generation if you know there is only one valid token (common in the canonical tokenization setting)
- https://blog.dottxt.ai/coalescence.html</code></pre>
Hello, the part about canonical filtering in <a href="https://openreview.net/pdf?id=DFybOGeGDS" rel="nofollow">https://openreview.net/pdf?id=DFybOGeGDS</a> doesn't seem to try to account for pretokenization. For example, if you receive " 天天中彩票APP" in o200k, it means there has to be a lowercase letter within the span of letters, and while tokens like (4 spaces) may be pairwise compatible with tokens like "123" according to the BPE merge rules, the pretokenizer would split the span of spaces to give (3 spaces), " ", "123" instead. Are you aware of any work that does actual canonical generation for models with this kind of pretokenization regex?
> Here are a few other pointers<p>Proceeds to list all the libraries already listed in the guide.
What a gold mine!<p>Automata-based constraints is fun.
Question for the well-informed people reading this thread: do SoTA models like Opus, Gemini and friends actually need output schema enforcement still, or has all the the RLVR training they do on generating code and json etc. made schema errors vanishingly unlikely? Because as a user of those models, they almost never make syntax mistakes in generating json and code; perhaps they still do output schema enforcement for "internal" things like tool call schemas though? I would just be surprised if it was actually catching that many errors. Maybe once in a while; LLMs are probabilistic after all.<p>(I get why you need structured generation for smaller LLMs, that makes sense.)
Schemas can get pretty complex (and LLMs might not be the best at counting). Also schemas are sometimes the first way to guard against the stochasticity of LLMs.<p>With that said, the model is pretty good at it.
Yes. Most common failure mode for sota models is to put ```json\n first, but they often do just fail often enough to be worth calling api with json response schema.
This is good. It covers the two easiest dominant methods people use. It even touches on my main complaint for the one they seem to recommend.<p>That said:<p>- Constrained generation yields a different distribution from what a raw LLM would provide. This can be pathologically bad. My go-to example is LLMs having a preference for including ellipses in long, structured objects. Constrained generation forces closing quotes or whatever it takes to recover from that error according to a schema, nevertheless yielding an invalid result. Resampling tends to repeat till the LLM fully generates the data in question, always yielding a valid result which also adheres to the schema. It can get much worse than that.<p>- The unconstrained "method" has a few possible implementations. Increasing context length by complaining about schema errors is almost always worse from an end quality perspective than just retrying till the schema passes. Effective context windows are precious, and current models bias heavily toward earlier data which has been fed into them. In a low-error regime you might get away with a "try it again" response in a single chat, but in a high-error regime you'll get better results at a lower cost by literally re-sending the same prompt till the model doesn't cause errors.
> Increasing context length by complaining about schema errors is almost always worse from an end quality perspective than just retrying till the schema passes.<p>Another way to do this is to use a hybrid approach. You perform unconstrained generation first, and then constrained generation on the failures.
Very nicely written guide!<p>If the authors or readers are interested in some of the more technical details of how we optimized guidance & llguidance, we wrote up a little paper about it here: <a href="https://guidance-ai.github.io/llguidance/llg-go-brrr" rel="nofollow">https://guidance-ai.github.io/llguidance/llg-go-brrr</a>
Incredible guide, wow. Will definitely share with people. I wish I had something like this a year ago.
This is a nice guide. I especially like the masked decoding diagrams on this page <a href="https://nanonets.com/cookbooks/structured-llm-outputs/basic-concepts/constrained-method/" rel="nofollow">https://nanonets.com/cookbooks/structured-llm-outputs/basic-...</a>.<p>edit: Somehow that link doesn't work... It's the diagram on the "constrained method" page
Are there output formats that are more reliable (better adherence to the schema, easier to get parse-able output) or cheaper (fewer tokens) than JSON? YAML has its own problems and TOML isn't widely adopted, but they both seem like they would be easier to generate.<p>What have folks tried?
Yes, that's the purpose of TOON.<p><a href="https://github.com/toon-format/toon" rel="nofollow">https://github.com/toon-format/toon</a>
Is there evidence that LLMs adhere to this format better than to JSON? I doubt that.
Their benchmarks compare it against other formats as input, not as output.
It is 100% guaranteed that they DON'T. Toon is 3 months old, it's not used by anyone, and it's therefore not in the training set of any model.
Nice, it would be good idea to develop CFG for this as well so can embed it into all these constrained decoding libraries
We're working on an agentic content transformation pipeline based on markdown with YAML metadata in the front matter. I'm a bit worried about the lack of tooling with respect to JSON payloads but then again it's not that hard to parse and then convert to JSON to validate against a schema.
Just brainstorming. Human beings have trouble writing json, cause it is too annoying. Too strict. In my experience, for humans writing typescript is a lot better than writing json directly, even when the file is just a json object. It allows comments, it allows things like trailing commas which are better for readability.<p>So maybe an interesting file to have the LLM generate is instead of the final file, a program that creates the final file?
Now there is the problem of security of course, the program the LLM generates would need to be sandboxed properly, and time constrained to prevent DOS attacks or explosive output sizes, not to mention the cpu usage of the final result, but quality wise, would it be better?
I use regex to force an XML schema and then use a normal XML parser to decode.<p>XML is better for code, and for code parts in particular I enforce a cdata[[ part so there LLM is pretty free to do anything without escaping.<p>OpenAI API lets you do regex structured output and it's much better than JSON for code.
You should do your own evals specific to your case. In my evals XML outperforms JSON on every model for out of distribution tasks (i.e. not for JSON that was in the data).
I agree that building agents is basically impossible if you cannot trust the model to output valid json every time. This seems like a decent collection of the current techniques we have to force deterministic structure for production systems.
> We use a lenient parser like ast.literal_eval instead of the standard json.loads(). It will handle outputs that deviate from strict JSON format. (single quotes, trailing commas, etc.)<p>A nitpick: that's probably a good idea and I've used it before, but that's not <i>really</i> a lenient json parser, it's a Python literal parser and they happen to be close enough that it's useful.
These are cool tricks but this seems like an impedence mismatch: <i>why</i> would you use an LLM (a probabilistic source of plausible text) in a situation where you want a deterministic source of text where plausibility is not enough?
You... don't. That's exactly what structured outputs are for! You're offloading any formally defined generation to a tool that better serves the case, leaving the ambiguous part of the task to the model.<p>Code is an example of a mixed case. Getting any mechanistically parsable output from a model is another. Sure, you can format it after the generation, but you <i>still</i> need the generation to be parsable for that. In many cases, using the required format right away will also provide the context for better replies.
[dead]
This information is really presented well. I subscribed to your newsletter. Thanks!
The first thing I've seen is that the article uses <a href="https://xkcd.com/2347/" rel="nofollow">https://xkcd.com/2347/</a> without a reference..
Is it famous enough to be sure everybody knows the origin?
I like structured outputs as much as the next guy but be careful not to try to structure natural language.
What would be the point of outputting unconstrained json if the output is consumed by a human?
Stupid question but isn't this useless for 99% of users? By that I mean that either your API provider supports Structured Outputs (OpenAI and Google) or it doesn't and you're SOL.<p>Sure the guide presents some alternatives but they're incomparably useless VS real enforced structured output.<p>I get that some people will run their own models or whatever and will be able to use some of the other techniques, but that's the remaining 1%.
Huge fan of BAML , nice coverage
BAML