Benchmarks are great, but I feel like there’s a better way this seems quite subjective.<p>What you really need is an objective benchmark
I actually really like subjective benchmarks, so long as it's a human (ideally me) grading the results. LLM as judge never made much sense.
> What you really need is an objective benchmark<p>"When are all the software engineers unemployed?"
Top solve rate is currently 24% with Opus 4.8... What's a competent human supposed to score?
> You are a senior SWE-Bench reviewer, make no mistakes.<p>I don't know what a better approach would look like while still remaining feasible, however this approach of telling a LLM to make a subjective judgement seems fundamentally flawed.
next round of trust me bro benchmarks
Why didn't they just make it "Staff SWE-Bench", would be much better smh. /s<p>But seriously, as an industry we're terrible at assessing engineering levels, I've worked with "senior engineers" who can't code and I've worked with "junior engineers" who could run rings around them.<p>Benchmarks like this should be much more precise about what they're actually testing, and what axes they're hard on. We also need to rise above prompts like "you are a senior engineer", it's woo, and it's far better to ask for precise outcomes.
As someone who's trying to get better assessments, I'm struggling to come up with objective coding tasks that evaluates all aspects of real life like planning, design choices, problem solving and context usage. From your experience with humans, Do you have any recommendations on what could be effective in measuring it?
[flagged]