Author here. Fair critique, thanks.<p>On the 512 tokens: that's the window of the model we benchmarked with (all-MiniLM-L6-v2), not a claim about embedding models in general. The article does mention text-embedding-3-small's 8,192 window, and the same setup works with 32K models like Qwen3-Embedding. If your documents fit the window, the advice stands: keep truncate.<p>A bigger window makes chunking easier, not irrelevant. max_tokens defaults to the model's own limit, so with an 8K model you get fewer, larger chunks and overlap matters a lot less.<p>Two reasons we still chunk even when the document would fit:<p>1. One vector per document is a summary of the whole thing, so a short, highly relevant section gets averaged away by everything around it. One vector per chunk turns the question into "does this document <i>contain</i> something close to the query?", with the doc scored by its best chunk. Your #include example is exactly that case: the first N tokens of every file look alike, and what distinguishes them is further down. That's the "deep content" split in the benchmark — truncate got 55% recall@5, recursive got 83%.<p>2. Cost. Transformer embedding time grows superlinearly with input length, so pushing a whole 8K or 32K document through a local model on CPU costs far more than embedding it as 512-token chunks. Remote APIs bill per token either way.<p>That said, you're right that our numbers only show the effect against a 512 window. We should rerun the same benchmark with an 8K and a 32K model. I'd expect the gap to shrink but not disappear, and that's worth measuring rather than assuming.<p>On "nothing anywhere told you": agreed, silent truncation is bad behavior. That line describes what Manticore used to do (and what most embedding pipelines still do by default), not a defense of it. truncate is still the default because multi-vector output needs a different column type, so it has to be opt-in.<p>On "what you already have": fair, that sentence reads badly. It means "the old default, unchanged", not "good enough for you". We'll reword it.
"If your documents fit the window, the advice stands: keep truncate."
"Two reasons we still chunk even when the document would fit:"<p>Which advice do you stand by? Obviously, very short content doesn't need chunking, so let's consider a document that fills 75% of the input context.<p>When chunking, your cost overhead (per token) goes up as the number of new tokens per chunk goes down. That's an argument for longer chunks, although the averaging/smearing point argues for not going too long.<p>Embedding calculations are effectively prefill: on my cheapo local inference system (32 GB AMD R9700 + 8 GB AMD RX 7600), the older 8 GB card goes about 80% as fast as the bigger card for Qwen3-Embedding-4B (a bit over 19 chunks/second on my usual corpus, blog posts+comments that are mostly well under 32K tokens). So I would suggest that anyone who is limited by CPU embedding models could benefit from even a small local GPU.<p>For your blog post, I would suggest an explanation of the chunking modes, either in the blog post or as a hyperlink to the docs about them. "truncate" and "sentence" are fairly clear, whereas the others are not. (If "mean" just computes the mean of the embeddings, that seems like a poor choice. The arithmetic at <a href="https://www.johndcook.com/blog/2026/09/16/coffee-milk-latte/" rel="nofollow">https://www.johndcook.com/blog/2026/09/16/coffee-milk-latte/</a> might work for single words, but seems likely to break down at the document level. "recursive" and "fixed" are opaque, at least to me.)<p>If/when I index my team's documents, I will consider a content-aware chunking that fits as many sentences, paragraphs or sections as possible into each chunk, with overlap determined by the level at which the chunk finishes. Content-agnostic chunking is easier to code and more generic, but indexing should respect a document's internal structure.
Just so you know, your comment was automatically hidden (“dead”) until I vouched for it now. Same for most of your recent submissions. Actually it’s probably because of your (exclusively self-promotional) submissions that your comments and submissions get hidden.