This is getting tiring. Watermarking has no effect on model output quality when implemented correctly. It's somewhat like swapping a random RNG seed to the seed 42, and detecting what the seed was from a random sequence. The sequence generated from the seed 42 is just as random as any other seed. There couldn't be a quality difference. And yes, the output from an LLM is a conditional random sequence of tokens from a distribution determined by a model.
The article has a pretty decent summary of the watermarking algo though. This reads as a pretty dogmatic statement in comparison.<p>In your analogy: What if seed 42 specifically causes poor quality behaviour (in some contexts specifically). Normally, these quality differences will be washed out because the seed is random, now it is no longer random, so shouldnt we check into specific behaviour under this specific seed?
Refusal behaviour specifically is interesting, because if you can point out specific cases where refusal behaviour significantly deteriorates due to the watermarking, it may create a token "route" that may be possible to exploit by adverserial prompters. Static hazardous prompt refusal belies the fact that actual adversarial prompters will adapt their techniques iteratively and gain way higher compliance rates.<p>My idea would be that the ngram size over which the watermarking works is necessarily limited in order to resist edits better. It might be possible to lead the model to trigger the refusal in the form of these specific ngrams, the completion of which is then more likely flipped to compliance (due to the logit bias introduced by the watermarking), making hazardous requests systematically more likely to be accepted?
“When implemented correctly” is probably what people are complaining about.<p>Opus 5 started adding a bunch of comments to code, even when instructed not to, and for very simple changes where the comment itself was longer than the code change. Was that so that there are enough tokens outputted for watermarking? Many people suspected so.
I have seen Fable’s reasoning talk itself into ignoring an unequivocal prompt directive not to write comments, and then be startled by the precommit hook that rejects it. It is almost desperately predisposed to emit prose. And horribly turgid, waffling prose, to boot. Claude has been like this since Opus 4.7 though, i.e. (probably) predating the introduction of watermarking.
And that’s not even implemented incorrectly
So that's where that nonsense comes from...
Model companies are doing this for themselves anyways, it’s so they don’t feed generated content back into the slopper and collapse the model. From that angle it over time contributes to better model quality.
Also - it forms a cartel.<p>Detection of watermarking requires access to the watermarking key, a secret in the current suggested scheme (leaking it would amount to being able to strip the watermark).<p>So, there will need to be a watermark checking service. The checking service will of course be rate-limited for common folk (and model distillers). OpenAI/Anthropic/Google/other privileged model builders need to filter out AI slop at scale, so need access to others' service without rate-limits (or the watermarking keys need to be shared).<p>This creates an in-group with pristine datasets, and an outgroup whose models will collapse on the slop outputs with no good ability to filter.
>This creates an in-group with pristine datasets, and an outgroup whose models will collapse on the slop outputs with no good ability to filter.<p>But all the chinese labs who are hot on the heels of american labs thanks to "distillation" seems to be able to work without "pristine datasets"?
Haha, calling them pristine is maybe too much indeed. Right now they seem to manage, but what % of the scrapable web is now AI slop? What if it becomes 99% AI slop, 99.9%, 99.99%, etc. Surely the signal to slop at some point starts to become too low and you need to do some kind of at-scale filtering.
> Model companies are doing this for themselves anyways<p>No it's EU law.
What? It has absolutely nothing to do with "model collapse".
Accepting your analogy at face value, it's still not obvious to me that fixing a specific seed doesn't change things.<p>Take a recurrent PRNG for example. A randomly seeded recurrent function usually has degenerate cycles in its state space. For some functions, this might even describe the majority of the state space. This is why so many non-cryptographic PRNGs are max-cycle, so a different starting point is just further along the same trajectory.<p>I don't think LLMs have quite the same failure mode here, but recurrence + high dimensional spaces triggers my "here be dragons" sense.
> This is getting tiring. Watermarking has no effect on model output quality when implemented correctly.<p>Over a certain token threshold, yes, there are 0 negative effects. Something like 300-400 words. At the boundary and below it, it does effect response quality, so they don’t (shouldnt) do it. It also incentivizes increasing tokens in low token responses so that it can be watermarked which is it’s own quality issue.
That's not true. Watermarks are messing with the next token generation probabilities based on some random seed. The quality is neccesarily lower, the difference is simply too small to notice, typically.
You have a misunderstanding of how LLM generation works. Before any watermarking gets involved with these models there is ALWAYS a random seed used for generation. For any prompt, some seeds will give better answers, and some will give worse ones.<p>Let's say there are four billion possible seeds. There are four billion possible ways we could watermark the generation. We could say "we will choose seed 1, that way we will know exactly what output it produced", we could say "we will choose seed 2, that way we will know exactly what output it produced"... etc etc. Now, if we decide "not to watermark", we STILL must choose a seed. So we are actually still applying one of the watermarks, the only difference is we are not careful to remember which one. Could some seeds give a better or worse answer to some specific prompt? Yes. Could choosing a random "watermark" to apply be better or worse on average than choosing a random seed to apply? No. It's mathematically impossible.<p>This is like an open source project changing their seed from "12321" to "43", and saying that because we changed the seed, the quality is "necessarily lower".
No, the probability distribution is the same. Watermarking changes the rng sequence used to pick from that distribution.
Theoretically, that can be true.<p>In reality, Gemini and Anthropic use SynthID watermarking which affects token probability distribution, i.e. their tournament sampling can pick lower-probability tokens which the LLM's distribution would otherwise not have. They likely use this over unbiased watermarking because SynthID is resistant against text edits.
The conditional distribution is the same. The joint distribution of the full output does change with watermarking. Just look at the diagram in the article: there is a loop back to the seed once a token has been generated.<p>Changing the distribution is the whole point: it introduces statistical regularities that can be detected.
It introduces statistical regularities, but all RNGs introduce statistical regularities. So the question is on average are these statistical regularities better or worse than those introduced by the alternative, and the answer is no, if implemented properly.
> It introduces statistical regularities, but all RNGs introduce statistical regularities.<p>Agreed.<p>> So the question is on average are these statistical regularities better or worse than those introduced by the alternative,<p>Indeed. This is an empirical question. That is what the article is about, for a specific setting.<p>> ... and the answer is no, if implemented properly.<p>I don't agree with that. This is not about implementation. It's about how the statistical regularities that are imposed on the full output distribution affect that distribution. There is a change - by construction. That change can be good in some situations and bad in others. The authors claim that it is mostly bad in the setting that they investigated. This looks like a fair statement.
Benchmarked output quality versus actual output quality are very different things. Some usecases are at the very fringe of model intelligence and depth of intelligence and logic suffers.
You’re using the subjective definition of “quality”, as in the shade of blue it chooses for “Build a website”, or the character names for “Tell me a story.” In those cases it’s likely still subjectively “high quality”, depending on who you ask.<p>What the article is discussing, and what many people are concerned about, is something that you might be missing in your understanding: they’re not actually random. In fact, they would be entirely useless for real work if every token was randomly selected based on all possible outputs. It’s not, even at temperature 1.0. It’s based on the training corpus and once you have your tool names, syntax, and prompting style aligned with the training data then they become incredibly deterministic <i>in the areas that matter</i>, such as tool calling and parameters. I build my toolset by testing thousands of names, syntax, return format, and other aspects until I find a convention that produces the exact correct call, 100% the exact same every time, regardless of context length. Those decisions are <i>per model</i> and what works with Opus 4.7 won’t necessarily work on 4.8, and neither version will work with a local model or GPT.<p>That’s only possible because the massive training corpus is the guiding principle behind the choices. Providing a file reading function called “Read_The_File” will fail, either on the first call or somewhere down the line, because that name is not associated with the concept. Your instructions are trying to override 500 trillion tokens from training and it will cause perplexity to manifest as wrong tool calls, wrong syntax, “oops deleted prod”, “Claude lost the plot again”, “WTF?!”, and probably nearly every frustration you’ve encountered and determined to be “they nerfed Claude” or “it’s a dumbass.”<p>For those that are aware of it, that knowledge lets people tweak and tune the prompts/tools accordingly.<p>You may not put that effort into your system, perhaps because you’re unaware of it, don’t use it in a way that requires it, or you’ve just taken the failures caused by perplexity as something that’s inherent in the framework, but for people that build precision infrastructure around them it’s potentially devastating news. Watermarking, which is based on whatever tokens, threshold, cutoff, and triggers some guy at a desk decided, will necessarily alter that entire system.
The caveat is they have to be a little more sophisticated than just changing rng on the sampling, because they have to detect for snippets in the middle of a long chat and with removed thinking. I don't know that any of the stuff they need for that actually has impact though.
[dead]