I wish they'd do a follow-on post drilling into the impact of the programming language on cost-per-task, specifically looking at cost to complete tasks in mainstream strongly typed languages (eg. C#, TypeScript) vs dynamic languages (eg. Python, JavaScript). Does the additional verbosity of the language help or hurt cost per task?
<i>This was mostly because Sonnet 5 worked longer and read more to get there, consuming 1.9x more tokens.</i><p>I have experienced similar behavior between opus and haiku when benchmarking Dara engineering tasks. The “cheaper” model takes many more turns to figure out the task and this is without taking into account other important factors.<p>Another interesting behavior that I observed is that Haiku tended to cheat more maybe because it was having a harder time to find the root cause of the problem.<p>Benchmarking and evaluation of agentic systems is very interesting and if there’s one thing that someone should keep from the Databricks post is how important is for everyone to build and run their own.
1) Many models are now competitive at the top tier, including open source.
2) GLM 5.2 in particular was a major step forward in open source coding agent performance,
3) Harnesses make a huge difference in cost-performance.
4) Cheaper per-token does not imply cheaper per-task.
k;l