Hi, most memory tools get evaluated on benchmarks like LoCoMo, LongMemEval and BEAM, plus how many tokens they use<p>We compared a pre-release build of Graphicle's memory with 7 other memory tools (Mem0, Hindsight and others) on three public question sets. We wrote down the test rules before scoring anything, and used one open model to read and judge the answers for every tool, so it's the same setup for all of them. Details and caveats are in docs/benchmarks.md<p>What we haven't done yet is the thing you're really asking: does a coding agent actually finish more tasks with these tools? We could use the same agent and model on the same tasks, with and without the tool, and compare how many tasks pass, tokens used, tool calls and time.<p>Curious what you're using today, and what result would convince you?
I'm not looking to be convinced about this tool as much as I am genuinely looking for a way to eval these kinds of tools in general<p>I have only just begun playing around with evals for a harness I'm working on and so far I have just been doing runs of converting <a href="https://github.com/realworld-apps/realworld" rel="nofollow">https://github.com/realworld-apps/realworld</a> from one language to another<p>not super pleased with it<p>the unsatisfying answer I think is that crafting good eval environments for long runs takes a lot of time, and there are many things you'd want to verify that you cannot simply lint or test for
Yeah learn how to do even the basics of science, please. Control groups!! Confounders!!<p>Letting another LLM decide quality probably isn't the only meterstick you want, either.