Makes sense, I have had the biggest wins with AI by attacking nondeterminism whenever possible.<p>BTW, you should probably fix the Beagle link on your homepage: <a href="https://replicated.live/beagle/" rel="nofollow">https://replicated.live/beagle/</a>
I find some of the most interesting, and catastrophic failures in my agent fine-tuning come from the clamping down of non-determinism. It is totally the correct approach, but must be handled delicately. The non-deterministic core remains, but now under bimodal pressure.
I think this is less about clamping down on non determinism and more remembering that a script is much more reliable than having the agent do some things. Think making a number of API requests to get info for context or running a sequence of testing steps to generate a report. Remove easy places where that non determinism rears its head and there is really no need. I talk about what I'm doing with PR review in a other comment, as an example.<p>In other words, are there places where a one liner for the agent would be more reliable than markdown instructions and crossing fingers?
Thanks, fixed. The runtime[1] and the scripts[2] are the practical ones. I am separating the old repo[3] into submodules since submodule recursion became smooth in Beagle.<p>[1]: <a href="https://github.com/gritzko/jab" rel="nofollow">https://github.com/gritzko/jab</a><p>[2]: <a href="https://github.com/gritzko/beagle-ext" rel="nofollow">https://github.com/gritzko/beagle-ext</a><p>[3]: <a href="https://github.com/gritzko/beagle" rel="nofollow">https://github.com/gritzko/beagle</a>
This makes sense, although it's not well described here.<p>Formal methods, as in proof of correctness, have been around for decades (I was doing that stuff in the 1980s) but pushing the proofs through was too laborious. The seL4 verification effort reportedly used over a decade of people time.<p>The idea is that if you have a formal specification of what you want to happen, you can get a LLM to do the struggling with the proof system to get it right. It's a good task for an LLM, because there's feedback from the prover.<p>I'd like to see more non-trivial examples of this. People keep republishing verifications of greatest common divisor or stack algorithms, which was done decades ago.
might be relevant: <a href="https://martin.kleppmann.com/2025/12/08/ai-formal-verification.html" rel="nofollow">https://martin.kleppmann.com/2025/12/08/ai-formal-verificati...</a>
Problem is, usually describing the problem you want to solve *correctly* using formal tool is a task as hard (and often, equivalent to) the implementation. That said, having a formal description is useful
For some problems, yes. Formal specification is particularly useful in two cases. 1) The problem is simple but an efficient implementation is hard or bug-prone. Examples are garbage collection, file systems, sorts, databases, and tree updating.
2) The inverse of the problem is simpler than the forward operation. Examples include matrix inversion and parsing.
This is a very interesting introduction to a blog post, but... I'm somehow missing the actual blog post. How does this stuff work in practice? What are some concrete examples? How does one get from JavaScript tokenizing things in a commit hook to validating that the LLM didn't disable tests it didn't agree with, or any other helpful property?
I'm seeing tons of blog posts which seemingly amount to having AI write code. It would have never occurred to me to repeatedly invoke an LLM to do what a simple script could, but I guess I shouldn't be too surprised. 20 line bash scripts replacing entire enterprise software stacks was a meme even in the 90s.
I’ve got a test that checks to see if “Logger” has been imported anywhere in my Elixir project, and if it finds one it prints out an explanation of why this project shouldn’t use Logger and what it should do instead. (Which is— emit OpenTelemetry events.)
Basically what I’ve been saying since OldJob forced LLMs down our throats and pegging performance to usage metrics: why the <i>fuck</i> are we handing deterministic processes to probabilistic systems when it should be the other way around (using probabilistic systems to design deterministic ones)?<p>LLMS should be abstracted out of a process as soon as practicable, replaced with deterministic processes or procedures. Otherwise you’ve built the world’s most fragile process at the mercy of token cost, vendor hostility, geopolitics, and model deprecation.
Actually... yes. I was bracing to be very annoyed with your comment starting with "why is everyone using AI so stupid?!" (I know those weren't your words, but it felt like that kind of post)<p>And then... yeah. You got it exactly right. Once a problem or process is deterministic, that's the <i>wrong</i> application of an LLM.<p>But I had never <i>quite</i> thought of it in these exact terms. The way I've been thinking about it up until now is that the very best way to use LLMs is to have them produce tools. The tools get to stay reliable and predictable. They boost your performance. But I think you found the more general abstraction of the same idea. Tool-making is not deterministic. But the tools themselves can be. That's why it fits. Trying to stuff LLMs into what's otherwise a deterministic process is an absurd waste and error-prone.<p>Smart. I like it.
This is exactly right. Abstracted out of the process, or to a point of most optimal application.
Thats the best description I have heard of the problem so far. I ran into this recently where I automated a ton of stuff and got essentially threatened by leadership for not using AI. My system produces the same output 100% of the time, is free, and scales plus is reliable. Doing what they wanted with an LLM was fragile, didn't always produce the same output and was subject to costs. I don't think they could wrap their brains around it.
I love the way you put this. Are there any sites or forums or places where people discuss/hash this out?<p>I've genuinely never considered it from this angle before.
Second this, following Cloudflare's post on how they do agentic PR review, I'm working on a script that renders the conext and diff to disk before passing it off to the agent, which generates a jsonl file of comment add/update, which another script will process. Way better than handing it bash and clis so it can fumble about non deterministically