Could you say more about how caching works? One major advantage of sticking with a single model is saving money on cached input tokens. I'd imagine if you swap between a bunch of models, you may improve performance but cost would would balloon out of control
Generally you should only have two models in the pool per domain. I wrote some of my learnings building a router here: <a href="https://try.works/first-principles-of-model-routing" rel="nofollow">https://try.works/first-principles-of-model-routing</a>
The trick is to rarely switch, or switch at task boundaries. Often the conclusion of routing is actually "this one model is actually at the pareto front for this task, just use it always".
and caching is related to performance too ofc
Open source and no markup is the right default for a gateway. The caching question above is the one I would want answered before swapping models though.
>The gateway adds under 1 ms for BYOK requests<p>Amazing! Really brilliant idea, thank you for sharing this project. There is so much ground to cover in the LLM gateway / routing / reporting world, and this is a great start. The Tinker implementation is my favorite part, fine tuning is much better than a sea of context files.
What online signal recalibrates simulated rankings against actual task success? Also do you have a plan to support semantic caching at the router level?
For the online signal, we use a LLM judge with a rubric calibrated offline by the user via TUI. UX of the calibration is a major focus area. Semantic caching is interesting, open to supporting it but not currently planned.
Very cool. Does your gateway decide effort levels as well? Or just models?
what's the business model here. How does experiential labs make money
They make money on enterprise plans: <a href="https://www.experientiallabs.ai/pricing#enterprise">https://www.experientiallabs.ai/pricing#enterprise</a><p>Look at the Intelligence features in the Enterprise plan:<p>* Per-prompt model optimization<p>* Caching<p>* A model you own, trained on your traffic
You started it a week ago? I look forward to checking back in 3 weeks when you've exited for $1B
I have not tried it yet. Is it similar to LiteLLM? If so, what sets it apart?
Super interesting and congrats on the release. Curious if you initially had this in Python and then rewrote in Rust?
Finally an open source tool doing this!
[dead]