I bet I can do better by allowing this: the llm can pull documentation of the language from the web to understand how it works.<p>If the llm has “skills” for that language, it will definitely increase accuracy.
I'm shocked to see how poorly these models, which I find useful day to day, do in solving virtually <i>any</i> of the problems in Unlambda.<p>Before looking at the results my guess was that scores would be higher for Unlambda than any of the others, because humans that learn Scheme don't find it all that hard to learn about the lambda calculus and combinatory logic.<p>But the model that did the best, Qwen-235B, got virtually every problem wrong.
I had hope we might finally be ushering in a bold new era of programming in Malbolge but apparently that was too optimistic.
Mhh... my hunch is that part of this is that all python keywords are 1 token, I assume. And for those very weird languages, tokenizing might make it harder to reason over those tokens.<p>Would love to see how the benchmarks results change if the esoteric languages are changed a bit to make them have 1-token keywords only.
[dead]
[dead]