Scaling Laws, Carefully

(lilianweng.github.io)

33 points by tehnub4 days ago

2 comments

gdiamos1 hour ago
When I first saw scaling laws in that deep speech experiment notebook, I didn’t believe it could be real. I was worried for months that we made a mistake, or that it only worked for that one dataset.I started to believe it after we (Joel Hestness in particular) reproduced it in so many experiments in “scaling is predictable empirically”.The OpenAI work replicated it in a completely different environment, and at that point I was sure it was real.Sometimes people ask me why I was so surprised by it. Prior work like Banko and Brill and the unreasonable effectiveness of data argued for more data. ML theory had similar models for toy problems, eg coin flips.At the time I thought deep learning was supposed to be complex. Speech and language datasets seemed much more complex than toy problems. Optimization of deep transformers was complex.The idea that it was possible for the whole thing to be governed by a 3 term equation seemed too simple. The implication was that it was simple to manufacture intelligence.Ten years later, I still think it is still the most interesting observation I have seen. We are still learning what it looks like to live in a world where it is possible to manufacture intelligence.
aspenmartin4 days ago
I really wish more people skeptical of AI capabilities would read about scaling laws -- Lilian is always so marvelous at giving a deep overview of the technical side but the whole point of this is: there are scaling laws, and they hold and continue to hold. This is such a huge basis for the predictions about AI capabilities for the past like 5 years.
- an0malous1 hour ago
 Why should the skeptics be reading it? The scaling laws show diminishing returns on more training data and larger models.From the Kaplan scaling laws paper:> We have observed consistent scalings of language model log-likelihood loss with non-embedding parameter count N, dataset size D, and optimized training computation Cmin, as encapsulated in Equations (1.5) and (1.6). Conversely, we find very weak dependence on many architectural and optimization hyperparameters. Since scalings with N,D,Cmin are power-laws, there are diminishing returns with increasing scale.So the skeptics are right to be skeptical of LLMs being all you need for continued advancement in this space. It seems like the believers are the ones who need to learn about the scaling laws.
- FromTheFirstIn4 days ago
 And sitting right next to the data and compute factors in every cross entropy loss equation is the entropy of the language, which is just a fixed constant. There’s such a hard cap on cross entropy loss training and I never hear it come up!
 - aspenmartin4 days ago
 Right but that is context dependent; it drops with context length, depends on tokenizer, etc. It doesn't end up being super relevant, despite the fact that if you look at the loss for real models it's relatively large in absolute terms. But that doesn't really matter -- all of the interesting stuff happens once you start getting closer and closer to it. You've gotten past all of the easy tokens that dominate the entropy and now you get to the really challenging ones that we care about (like e.g. very difficult reasoning about a next step).
 - FromTheFirstIn4 days ago
 My understanding is that the true entropy floor of a language is intractable- regardless of context length there will be “unpredictable” tokens where cross entropy loss is bound to happen. Even with infinite parameters and data you’ll still have a chance at failing to predict the next token correctly a decent chunk of the time.Also, linear gains in context length scale quadratically with compute because of attention, so depending on context growth means taking a bath on GPUs for as long as you can, right?
 - graboy3 days ago
 Yeah I mean, if you and I were to play the word-guessing game where you needed to guess what next word I'm thinking of, there's always uncertainty in your guess because it's a game of partial information - you can't fully observe my inner state. But that doesn't mean you couldn't evolve a strategy that spends a really long time thinking and analyzing to get asymptotically close to the best guess. There's no limit on that intelligence.
 FromTheFirstIn3 days ago
 Isn’t the limit exactly what you’re describing? There’s always uncertainty, and your asymptote can approach its limit but it does have a limit. That’s the limit to the intelligence. And this is just for cross entropy loss- even if you could get loss to 0, I’m still not convinced at all that an enormous semantic map and its convoluted geometries amounts to intelligence.
 aspenmartin3 days ago
 If you get to E you have generated a Bayes-optimal model of the conditional distribution (as in, next token conditional on context). This is something I thought too, but even if you're a fraction of a nat above the floor, you could have enormous headroom in performance left because there are still rare tokens amongst the irreducible noise that require so much capability to predict. It's not to suggest there truly is no cap on capability, but just that this constant isn't really saying what that is.
 FromTheFirstIn3 days ago
 Yeah, it not a linear cap (x% entropy doesn’t mean x% wrong) but it does seem like a hard cap. To be honest, the more I’ve understood scaling laws the more I think that the elephant in the LLM room is the entropy of the language. It explains why coding languages are so much more tractable (they’ve got WAY less entropy) and it explains why we haven’t seen a step function in capabilities for LLMs since GPT-4 outside of making specific toolings for particular contexts. I think E is coming to dominate and there isn’t a workaround for it.