It's a reasonable summary, although I feel there's some fundamentals missing. I wonder how much it will make sense to a platform engineer who doesn't already have a very solid understanding of transformer architecture internals such as the Attention mechanism and its more recent variants. I don't know if you can really <i>get</i> this stuff without having some underlying model internals picture in your head of what the KV cache actually is and how it falls out of repeated attention computations on a continuously growing prefix, to say nothing of understanding KV cache optimizations like PagedAttention which depend very much on the details of how you compute the attention weights with matrix multiplications, softmax, etc. I certainly couldn't make sense of something like "Rather than recompute that history, the engine keeps the attention keys and values for every token in flight" without a pretty solid understanding of how Attention works, which I had to gain from other sources.<p>This is also a shortcoming of Philip Kiely’s otherwise fantastic book <i>Inference Engineering</i>; it kind of glosses over the transformer architecture details, which you need to get elsewhere. Sebastian Raschka's books are very good for that but you have to then link it all back to how to do efficient inference, which he doesn't talk about so much.
That's fair. We tried to balance level of detail to make it approachable for newcomers to the space as well as practical tips for experienced practitioners. That being said, we'd love to incorporate your feedback in the guide - feel free to contribute to the repo and we'll also add those topics to our list to cover
This guide is for platform engineers new to inference. If you need to understand how models and GPUs work together and how to run inference in production, this guide will help. You’ll also learn about the tools to use and how to grow into an inference engineering role. This guide was written by the maintainers of the open source CNCF sandbox project Modelplane
Gives me the impression of being agent-written:<p>- one page with reasonably looking/appealing structure, tables, bullet points, etc.<p>- fairly long<p>- looking more closely, things start to be quite generic, lacking nuance ...<p>The issue I have with this is not that it is agent-written or -drafted, but rather that the signal-to-noise ratio seems to be off.
We certainly used agents to help us put a draft together based on a wealth of our notes, issues and data and then we (humans) spent time curating, editing and revising the document. You can find a lot more nuance in our articles in the series where we talk about our operational experience running coding models for our engineering team - <a href="https://www.upbound.io/blog/drinking-our-own-champagne-part-6-the-model-was-rarely-the-problem" rel="nofollow">https://www.upbound.io/blog/drinking-our-own-champagne-part-...</a><p>Not all of the nuance has translated into the guide as we tried to uplevel the specific learnings into transferable lessons and may have ended up sounding overlying generic in some parts.<p>We will improve it based on all the feedback we got (thank you!) and would really love to get the community involved to help keep this a valuable resource.
It looks rather generic. And to be honest I was able to use an agent to hand roll cluster serving stack using vllm, completed with spec decode, kernel cache reuse, and fixing or forward porting engine bugs.
[flagged]
[flagged]
[flagged]
[dead]
[flagged]