This is true in every large company right now.<p>The transformer based approach is seen as a highway to race down in terms of training and building better models with minimal organizational distractions allowed while every company puts its foot down on this highway. Exploratory research into other AI methods is highly discouraged for now and organizations are encouraging transformer based scale out and iteration as a top priority.<p>As an example Meta literally fired it's previously head of AI who held views that transformers won't scale. He wanted to veer off the highway and heavily fund exploratory new types of models. They got someone willing to scale the current method out. Essentially the messaging it "Foot down, there's a straight highway ahead don't divert".<p>I have the view this makes sense too. Current top models are mere (mere in big datacenter terms) trillions of parameters right now. We definitely haven't hit the ceiling and everyone's noticing "Hey this Opus 5.5 with the increased parameters seems much better than the last one". Well that's a trillion parameter model. We'll soon start hitting 10's of trillions and 100's of trillions of parameter models. Unless there's a ceiling which nobody expects they should keep scaling just as they have been. I really do agree that it'd be insane to veer off this highway with exploratory research at this particular point in time. Foot down, get to 10 trillion parameters, see if the scaling holds, if so off to 100 trillion parameters. If the scaling continues to hold we'll be asking the AI to do the exploratory research for us soon enough.
> As an example Meta literally fired it's previously head of AI who held views that transformers won't scale. He wanted to veer off the highway and heavily fund exploratory new types of models.<p>Almost all statements Yann LeCun made back then (LLMs won't be able to do this or that) turned out to be wrong. They were right to fire him.
Historically, that happens now and then. Examples include color television cameras and fluorescent lamp ballasts.<p>RCA developed the first good color TV cameras. They were monsters. Originally, electronic TV (as opposed to mechanical scanning) used Zworykin's iconoscope. This needed huge amounts of light. It didn't have a photo-multiplier stage to boost sensitivity, but it did accumulate charge from light over the whole frame time. There was an alternative approach, Farnsworth's image dissector, which also needed huge amounts of light. It had a photo-multiplier stage but only the light from the dot currently being scanned by the scanning beam mattered.<p>So RCA threw money at the problem. They developed the image orthicon camera tube. It combined the advantages of both approaches, integration over a full frame and a photo-multiplier stage. It was insanely expensive, costing thousands of dollars in the 1950s. It needed about twenty different voltages and multiple independent power supplies. It worked reasonably well.<p>Then came color. That took three image orthicons (RGB) inside a box the size of a small refrigerator, along with optics to split the incoming image into three colors. Plus
a rack of analog electronics per camera, connected through a large cable. Plus an alignment problem from hell, with a huge number of adjustment knobs.[1]<p>Scaling worked! From the 1950s to the 1970s, color TV broadcasting was powered by those monsters.<p>Now, independent of this, other people were working on other approaches. Vidicons, silicon detectors inside camera tubes, and finally fully solid state image sensors. Eventually, in
1983, after thirty years of image orthicon monsters, RCA produced a usable solid state camera. Today, a cell phone camera costs about US$10 in bulk, and it's far better than the monsters.<p>It took thirty years of low-priority development to get past the giant image orthicon cameras.<p>Electronic fluorescent lamp ballasts had a different problem. That was a suppressed invention.
The magnetic ballast industry was dominated by Magnetek. Someone invented a solid state replacement, which was smaller and cheaper, and licensed their patent to Magnetek for royalties. Magnetek just sat on it, not wanting to gut their core market. The inventor hired Townsend, Townsend and Crew in Palo Alto and sued. (Read this in the law firm's brag book in their office.) Eventually, the inventor got a hundred million or so, solid state ballasts went into production, and Magnetek seems to be gone. (The company called Magnetek today, which makes crane controls, is an offshoot of a Litton unit, and seems to be unrelated.)<p>There are also parallels from the steam locomotive industry, the Baldwin Locomotive Works being the best known. Baldwin actually built the first Diesel locomotive. They made a few half-hearted tries at Diesels over several decades, then gave up and went back to steam. Baldwin ceased all locomotive production in 1956. The corporation lingered on for a while, not doing much.<p>[1] <a href="https://www.tvcameramuseum.org/rca/tk40a/pix/rcatk40aart.pdf" rel="nofollow">https://www.tvcameramuseum.org/rca/tk40a/pix/rcatk40aart.pdf</a>
(So it must be those Orthicon cameras which cause the distinctive weirdnesses of early TV images like the reflections from highly-polished surfaces (such as the bells of brass instruments and saxophones) which gleam <i>black</i>? <a href="https://www.youtube.com/watch?v=2uTQQKU6ge0&t=111s" rel="nofollow">https://www.youtube.com/watch?v=2uTQQKU6ge0&t=111s</a> )
Its anywhere you have finite resources, time, people, knowledge etc and have to take a call on how to split them - Explore-Exploit Tradeoff.
The LA Times has an article on the Magnetek verdict: <a href="https://www.latimes.com/archives/la-xpm-1990-01-10-fi-282-story.html" rel="nofollow">https://www.latimes.com/archives/la-xpm-1990-01-10-fi-282-st...</a>
Nice, it got a press mention, with a quote from Crew, one of the law firm's partners.<p>Suppressed inventions are very rare, but that's one of them.<p>One of the few others is Airadar, from 1973.[1] Not "AI", "air radar". This
was a phased-array radar for small aircraft. Way ahead of its time. So far
ahead that the USAF was upset about it being available commercial before it
was in use by the military.<p>[1] <a href="https://books.google.com/books?id=NWzlTqj0gQ4C&pg=PA64&lr=&rview=1&source=gbs_toc_r&cad=2#v=onepage&q&f=false" rel="nofollow">https://books.google.com/books?id=NWzlTqj0gQ4C&pg=PA64&lr=&r...</a>
Suppressed inventions aren't "very rare" per se. You simply don't hear about those that aren't being rescinded while the inventor is still alive.<p>According to official data from the U.S. Patent and Trademark Office (USPTO), there were 6,653 active secrecy orders in effect at the end of fiscal year 2026. Roughly 100 to 350 new secrecy orders are added every year. Many remain locked away for decades or more.<p>If the U.S. government puts a secrecy order on a patent application against the inventor's will, they are legally forbidden from talking about it. The inventor cannot talk about how it works, publish research on it, sell it, or market it. They cannot even discuss the exact classified details with a regular lawyer unless that attorney has the proper security clearance.
Why are you Claude
For as long as the big companies are committed to focusing on scaling LLMs it does seem reasonable that they don't want their overall efforts led by people like LeCun or Hassabis who don't particularly want to scale LLMs. But it does feel as if there's something a bit extra about many LLM maxis' active hostility to deviationists.
Even if he was wrong about those statements, exploring other types of models isn't necessarily a bad thing. Especially if we can find ones that perform well with less demanding hardware requirements.
Its not a bad thing, but exploratory research as opposed to reaearch that finetunes along a path already paved by exploratory research is something that doesn’t usually happen much in for-profit product-oriented firms when there is such a path available; it happens in academia and corporate labs pick it up once it reaches the point where the basics show commercial promise. CS is a little bit of an exception because the capital cost of basic research isn’t, proportionally, as high, and because the promise-to-product path is often short enough that having done the basic work in house can be a meaningful advantage in getting to market first (though, see Google on much of modern AI for a counterexample).<p>But even in computing, when there is a promising way forward that the basic reaearch has been done on, its hard to justify side quests.
Those exploratory "side quests" in industry are how we got the transistor, the liquid crystal display, Shannon's information theory, the laser and practical photovoltaic cells, among many other foundational technologies upon which we rely today.
Blue LEDs too, which have transformed lighting and displays.
Laser printers were a side project that almost got the inventor fired.
Jet engines, too.
> and corporate labs pick it up once it reaches the point where the basics show commercial promise.<p>How does that work when the research team moved elsewhere? Re-hire talent? That takes time - the 1 thing you don't have in a fast-moving field like AI. Post sums it up nicely:<p><i>"Do the math—how much is it worth to halve training and inference cost given that the industry has investments approaching a trillion dollars?"</i><p>How much is these researchers' salaries, compared to what MSFT spends on compute? Chances are it's peanuts in comparison.
> CS is a little bit of an exception because the capital cost of basic research isn’t, proportionally, as high, and because the promise-to-product path is often short enough that having done the basic work in house can be a meaningful advantage in getting to market first (though, see Google on much of modern AI for a counterexample)<p>...And because they have monopoly money. Basically, research labs are usually vanity spending for corporations with too much money. Xerox parc was funded by a monopoly. Bell labs was funded by a monopoly. Google funds its moonshots with its monopoly.
Remind me again where the transformer comes from?
> it happens in academia and corporate labs pick it up once it reaches the point where the basics show commercial promise.<p>Ah, well, good thing we've almost entirely defunded academic science.
It’s a simple exploration/exploitation trade off. Always explore, never exploit is probably not the optimal strategy every time.
The right idea usually comes from a wrong one.
The issue isn't the exploration itself, it's the opportunity cost. When a figure of Yann LeCun's influence pushes a specific direction, it pulls key talent and resources away from mainstream LLM work. That kind of organizational distraction has a massive cost, and would've stopped Meta from delivering a polished product like Muse sooner.
The mainstream is wrong and in a lot of ways hazardous. He had every right to say he wasn't willing to sign is name on the wrong technology when he new better. When you hire smart people you listen to them. Elsewise you get what you deserve.
>when he new better<p>He was against LLMs, reinforcement learning, etc. Claiming that those are a dead-end. He was wrong. His own "JEPA" ideas have basically gone nowhere. Meta was right in moving on without him.
It's not a distraction if the research pans out and then they become the market leader for the next several years or more.
The job of the CEO is to assess the research risk correctly. Zuck made the right bet here to replace him.
Ok yeah but it’s not a certain bet. Resources are limited and you gotta allocate them efficiently.<p>10% chance to be market leader doesn’t sound as good as 60% second place to them for example
> It's not a distraction if the research pans out<p>It is because even if it would pan out, it would mean temporarily losing AI leadership, reducing stock price, losing the ability to hire talent, and by virtue of that losing the opportunity permanently.
LeCun wasn't wrong. He simply had higher standards for results and accountability that LLMs are, by design, incapable of reaching.<p>Zuckerberg wanted to placate his shareholders with a shiny new thing now.<p><i>would've stopped Meta from delivering a polished product like Muse sooner.</i><p>Muse is not polished, unless usability and programming standards have fallen so far in the LLM vibe coding era that a pre-alpha state is now considered polished.
You will find brilliant scientists in every discipline who are not the least bit interested in a particular branch of research because they/their lab didn’t invent it. That’s fine for public research but not helpful private research.
Compared to the user buying a Mac mini and following a tutorial off Twitter to install some random shit they found online?<p>Muse is imperfect, as is all software, but calling it pre-alpha is a bit much.
Odd. As I see it he was one of the few right about everything. This is also taking his comments about transformers scaling out of context. He specifically said they won’t lead to human level ai and they haven’t, despite exhausting the training data.<p>AGI will require architecture changes not simply more data.
> Almost all statements Yann LeCun made back then (LLMs won't be able to do this or that) turned out to be wrong<p>You're being uncharitable. He would have been right if Chain of Thought hadn't been pulled out of a hat, and no one saw it coming, or how u reasonably effective it would be.<p>Today, no one can know if there will be a future step-change improvement to LLMs at the same level as CoT, and conversely, no one can be absolutely certein there <i>won't</i> be - so it shouldn't be a firable offense because neither position obvious; even though it will look like it in hindsight.
All statements he made about world models did not already turn out to be wrong. He may be right in the long term.<p>LLMs work well for search problems with the help of a strong verifier.<p>As it turns out, every problem is ultimately a search problem. But not every domain has an easily available verifier.
Absolutely agree fwiw. Scale out over LeCunns desired exploratory research was absolutely the right call. Called out because it seems like MS is making a similar call.<p>The author states "I consider the scaling-only focus to be both misdirected and not fun.". I get the point but also totally get the focus on scale out for now.
I empathize with LeCunn but he is a prime example of why scientists often make poor leaders. It’s too tempting to always do the cool and interesting thing instead of drilling down on the boring but reliable thing that actually creates value. There is of course a place for this kind of eternal curiosity and I hope he’s found it, but hyper-competitive industry may not be it.
Ironically, Yann is kind of known in academic circles for being the guy working on boring, unsexy stuff (deep learning) when all the other academics were doing sexy cutting edge SVMs and stuff. That’s probably why he got the Meta job in the first place.
They have different jobs.<p>For a corporation it is to make money.<p>For a researcher it is to explore and illuminate. That this exploration and illumination often leads to a landfall of new wealth generation is a happy accident. This ensures that the monied class would bet some of their capital on them, at least the wise ones.
I'm becoming more sympathetic to LeCun's views, but being able to train something that acts the tiniest bit like those who made the training data (authors, professors, researchers, etc) is an absolute juggernaut and will be for some time.
>LLMs won't be able to do this or that) turned out to be wrong<p>Where can I read more about this? I have seen people make this claim for a while, and when I looked deep, it used to turn out to be wrong.
They surprised almost everyone, but he remained skeptical well after it was clear we could not put boundaries on the capabilities gained from scaling, and also from new RL techniques. Still, I do not think he is wrong about world models and continuous learning being fundamental to AGI, though the latter issue he was himself dismissive of until relatively recently.
It's not so much they _turned_ wrong, they were obviously wrong at the time he made statements, or shortly after. He is just extremely biased against transformers, and it's easy to see why - transformers are already invented, and as a scientist he wants to invent something new.<p>More specifically:<p>* he said that autoregressive transformer is fundamentally flawed: if it makes one error, all further generated tokens make no sense; that's, obviously, bullshit: there are many ways to implement backtracking, and there are many papers demonstrating backtracking at a time he did that lecture. And, of course, it just doesn't matter - reasoning model can just write "Oh, that's wrong. Let's try a different thing". - it's just not derailed by one mistake.<p>* he said that GPT-1000 can't possible have understanding of physical world, when GPT-4 already existing at that time could solve a problem he described.<p>So it's not like he was bad at predicting things, he's intellectually dishonest.
<i>Almost all statements Yann LeCun made back then (LLMs won't be able to do this or that) turned out to be wrong. They were right to fire him.</i><p>The idea that someone should be fired just because they were wrong seems to be something that would cause issues if one wanted to think long term.<p>This approach will make people afraid of being wrong. It will lead them to avoid challenging dogmatic assumptions in large organizations and inevitably lead to group think.<p>People really shouldn’t be punished for saying wrong things. Long term it’s better for a company if they aren’t punished for that. Helps to let people freely criticize dogmas or propose new ones.
Exactly.<p>Yann LeCun has the syndrome where someone who sees the value of B (non-text modalities, i.e. physical reality) cannot stop themselves from constantly critiquing A (LLMs), even though clearly A has its own advantages.<p>I think he is a smart guy. But he constantly deploys the crackpot's PR playbook.
In the short term. In the long term, his research on learning will be the next revolution.
It's Meta, lol. They wasted 80 billion $ on Zuck's fever dream called Metaverse.
It's amusing that their grand dream was basically a clone of Roblox, just with better VR tech and far more advanced graphics. But thing is, kids playing Roblox absolutely do not care about graphics.
Musk initially started his rocket company by going to Russia and trying to buy Russian rockets. The Russians spat on his shoe.<p>The rest is history.
That at least was a reasonable idea. The Russians had a lot of old Soviet era ICBMs they had no need for anymore (plus, IIRC an agreement with the US to disassemble them), so using their engines as a cheap powerplant until SpaceX nailed reusability made sense.<p>That failed and SpaceX was then forced to develop its own brand new engines.
Umm no. There’s the old factory way to build things like Ford model T; and then there’s the f*king Toyota method!<p>Which is superior, the big f*king monstrosity or the elegant on-demand, small batches, reduce waste?<p>Meta will go in the shitter and Deepseek will make the small batches.
TPS (and Lean, which it played a role in inspiring) is largely about finding what works empirically and ruthlesslessly optimizing around it.<p>It does (Lean at least) incorporate controlled experimentation, but tightly controlled with a both a laser focus on what the goal is and tight resource constraints. And Lean (at least several writers on Lean) favors taking multiple paths with a a quick and low risk and a longer-payoff, higher-risk, higher reward path for some efforts—one source used the US Navy’s two-path approach to SLBMs with Polaris and Poseidon as a model for this—but that's different than “maybe there is something down this path” basic research, which is outside of the concern a production model like TPS/Lean and often completely outside the model of a product-oriented firm; basic research is something firms consume and incorporate more than generate, because until the basics have been proven out, the rationally perceived chance of any particular avenue paying off is near zero for any firm.
Ruthlessly optimizing isn't the best goal, because people "optimize" all the quality out, and lack of quality leads to lack of sales and increased cost. Ford/GM may have "optimized" their line for efficiency, but the increase in defects caused more losses. Companies that practice Lean tend to forget this and focus only on speed and cost, leading to enshittification. TPS focuses more on quality and its effects on lowering cost and increasing sales.<p>Toyota is also a good example of what <i>not</i> to do. Their lack of earlier research investment in EVs has led to being severely behind the market, in addition to their lack of focus on quality causing poorer sales (their quality used to be one of the only reasons to buy them). KIA basically stole their playbook by creating loads of good-quality cheap hybrids and EVs. Toyota could've had that market if the invested more ahead of time in research.<p>And that's what's happening with AI. While there are plenty of players now, it's the ones that aren't looking ahead that will be left behind. Nothing scales forever; when you hit the ceiling, you're trapped in a box of your own making.
[dead]
More parameters without significantly more data fails; it loses the ability to generalize and simply memorizes.<p>And we’ve consumed pretty much all the data, and that part is getting harder as data is now expensive, Balkanized by owners, litigated and charged for. Couple that with model collapse as models eat their own output (not solved for any models, despite more than a decade of the problem), and add that lots of previously human curated datasets (stack overflow style), and it very well may be the models are not getting much better.
That's the historical wisdom, but there are reasons to believe the opposite: that past a certain point more parameters are always better. Basically, that overfitting don't matter, because a large enough model will memorize and build the general solution simultaneously. Then weight decay and SGD noise encourage forgetting of the memorized solution so the general solution takes over.<p><a href="https://youtu.be/lKoJJxjUfdw" rel="nofollow">https://youtu.be/lKoJJxjUfdw</a> is a really excellent lecture that changed how I think about it.
Very large models trained with sgd variants don't tend to over fit, especially when using few epochs. See double descent.<p>The coding/math breakthroughs were caused by training models on the subset of their own outputs which pass a verifier, see reinforcement learning from verified rewards.
This is what I thought a year ago, too. But how does it explain what has happened since?
This is a dumb question since I know very little about LLMs and transformers but I keep hearing similar things such as "more GPUs/parameters/compute" leads to a predictable scaling in intelligence.<p>Is that observed scaling part of some equation or law? Is there a limit to how far that scaling goes before a ceiling or diminishing returns hit? Or is it simply based on more GPUs/parameters/compute plus how much human knowledge is in existence and will scale predictably forever?
There are some neural scaling laws:
<a href="https://en.wikipedia.org/wiki/Neural_scaling_law" rel="nofollow">https://en.wikipedia.org/wiki/Neural_scaling_law</a><p>And you can in practice often see a large network follow an expected training trajectory for loss/entropy.<p>In practice data is getting more sparse than compute, so you are usually overtraining using more tokens and being suboptimal.
Another dumb question: can the model be smarter than the data?<p>I wonder about this a lot.<p>There are all these publications from OpenAI and Anthropic that are solving complex math & science problems.<p>Are they just "brute forcing" it? Able to try a lot more "known" or reasonable approaches given existing knowledge?<p>Or are they showing "emergent intelligence" and able to try genuinely new approaches that can't be expected from the knowledge in their dataset?
"Smarter" is ill-defined.<p>Are you smart if you see that cognitive scientists had something figured out, connected it to something in your field of chemistry and made a breakthrough treatment for some disease?<p>Does any of our existing intelligence build on completely unknown information?
You could argue that any new solution/emergent intelligence is an association of existing information in new ways.<p>It also brings us to what is brute forcing? Humans are brute forcing stuff most of the time. You will have 10 stupid thoughts for every good thought. 10 stupid ideas for every good idea, you will discard most of them so fast that you won't think of that you did that. It's very rare, if not near impossible, that a non-mathematician with no prior knowledge, goes into a room on day 1 of deciding to think about a conjecture they don't know anything about, and solve it on the blackboard. They need to use their collect associations, formalize their axioms, think of first principles (known information), and associate it.<p>So to answer your question you need to answer:<p>1) What does "smarter" refer to? Is it accumulation of information about the universe?(if yes, a database is smart) Is it an efficient, abstracted representation of accumulated information? Is it prediction of the future state of the universe given a learned abstraction?<p>Or going further, is it solving tasks useful for the human in the world? Is it acting as an agent with it's own goals and interests in the universe? What if the goals are hard for humans to understand?<p>Does it count as smart if you make an evolutionary approach that spawn 1000 instances and the most fit one survives and spawns 1000 mutations of itself?<p>2) What does "genuinely new approaches" refer to? Is it fine to use math axioms to prove things? Does interpolation between two existing solutions count as new? At what granularity of concept does novelty happen (Does using two higher abstraction concepts, like combining some new, modern neural network architecture with a database count as a new thing?), even if the underlying lower abstraction of the same information can be expressed as storage, latency and compute, and is not novel?<p>---<p>As you can imagine, this is all not really defined or agreed upon. Here are some examples of projects that are set up to drive these ideas:<p>The Hutter Prize poses that intelligence is defined by compression, so if you manage to compress the English Wikipedia efficiently, you are closer to better intelligence (<a href="http://prize.hutter1.net/" rel="nofollow">http://prize.hutter1.net/</a>)<p>Compute: Intel CPU<p>The ARC-AGI foundation poses that being able to be goal-oriented and understand higher level abstractions is intelligence: <a href="https://arcprize.org/leaderboard">https://arcprize.org/leaderboard</a><p>Compute: NVIDIA T40s<p>Modded NanoGPT competition poses that intelligence is minimizing entropy/loss over a dataset in the least amount of steps: <a href="https://github.com/kellerjordan/modded-nanogpt" rel="nofollow">https://github.com/kellerjordan/modded-nanogpt</a><p>Compute: 8x NVIDIA H100<p>---<p>All of this also exists in the context of the compute medium. It could be, that you need a certain capacity of memory/computation to perform whatever we have defined as intelligent tasks. Hutter Prize takes the bet of that to make intelligence, you don't need a lot of computation, an Intel CPU laptop and 50GB RAM should be enough.<p>A lot of the big labs are instead taking the bet on that you just scale everything, make trillion-sized models and things will fall out of that automatically, for this you will need a very different compute medium (data centers) compared to (a laptop).
> Another dumb question: can the model be smarter than the data?<p>There's a mathematical sense in which it cannot be: the data-processing inequality (<a href="https://en.wikipedia.org/wiki/Data_processing_inequality" rel="nofollow">https://en.wikipedia.org/wiki/Data_processing_inequality</a>). The extent to which that sense is totally counterintuitive to how things normally work in ML nowadays has led to investigating different foundations for information theory: <a href="https://arxiv.org/abs/2601.03220" rel="nofollow">https://arxiv.org/abs/2601.03220</a>.
[dead]
It should be noted that these laws aren't, well, laws; <i>"a neural scaling law is an</i> empirical <i>scaling law that describes how neural network performance changes as key factors are scaled up or down."</i>. Notice the word empirical here.<p>These empirical observations on quantities which aren't "intelligence" since no one has yet defined intelligence.
'Scaling law' is the prediction that increasing the training data and model size will cause the result to be better. It's not a law, but it was the prediction that OpenAI staked their fledgling startup on to get to the ChatGPT moment.
Your phrasing (_observed_ scaling _part of_ some equation) might indicate you assume / think there’s some theoretical equation that relate intelligence to those parameters and the we observe the data to confirm it. As far as I know, there isn’t - it’s purely empirical curve fitting. Train a model with different amount of compute / data / parameters, measure performance, fit curve.
Yes, the Chinchilla scaling laws from 2022: <a href="https://arxiv.org/abs/2203.15556" rel="nofollow">https://arxiv.org/abs/2203.15556</a> More parameters directly and mechanically means less prediction error for each token.<p>(This moves the argument to "is being really good at predicting the next token the same as being intelligent?" which has happily diverted many minds for years now)
From your link:<p>> for every doubling of model size the number of training tokens should also be doubled<p>Well, I don't think many claim that increasing the model size while also increasing the training tokens doesn't make sense.<p>The problem is how to get the training tokens...
The Chinchilla paper is probably the poster child of “paper that aged like milk” in LLM field though. Everyone over the past 4 years have trained with several orders of magnitude more data than the Chinchilla optimal number.
If you don’t know which data has the highest value, though more at it. Chinchilla is probably still true, but with that caveat. And it could be like advertising… you know a lot of it is worthless, but you don’t know which part.
That depends on your definition of scale. Yes! We were to literally convert the whole world and its resources into data centers.<p>But the whole problem is that these things aren't acting like other technological revolutions like the internet or Google where lots of profit comes from relatively little spend. They scale linearly maybe if you are super smart and good about your design.<p>Meanwhile, citizens all over the world are fighting data centers tooth and nail because we are hitting a scaling limit in terms of the resources people will allow to be put into this highway project and its equally unwise and untethered from reality to pretend this doesn't matter in a strategic sense.
What makes sense is to do both.<p>Race down the transformer highway as fast as possible <i>and</i> explore other AI methods.<p>One need not preclude the other.
as fast as possible implies spending the resources that way.<p>Going close to as fast as possible while saving a stream for the other approaches makes sense as an approach, but let's not ignore that there's a compromise/tradeoff here. Might not matter in orgs with near infinite resources
Also in an exponential competition environment, the difference between 100% commitment and 98% commitment is very possibly not even the difference between a win and second place, but outright lapping by the winner and humiliation at the scale of the gap for second place.<p>For an example of what this looks like at a more pronounced scale just look at the discourse around European models and how far behind Minstral are perceived to be.
My understanding is that the resources are limited and even AI labs with all the funding in the world don't have enough hardware.
Except doing both..is..so..hard when the future payoff of one is more unclear and you are faced with quarterly earnings reports.<p>Look at what happened to Meta where they squandered a lead, had some mixed results, and reorged to more product certainty and away from fundamental research.
Theoretically one need doesn't preclude the other. The OP gives a detailed argument and his own experience of one need precluding the other in practice.<p>Both of these approaches require quite a bit of institutional support notably.
it literally does when gpu's are scarce.
is scaling working? It is also my understanding that Opus 5.5 is probably a well under 10T parameter model. Probably a similar size class to Fable 5 earlier. And supposedly even GPT-4 was a 1T model. Clearly most of the gains are coming from something other than scaling up the model size. Yes, scaling up RL counts as scaling too, but the fact that labs have pivoted to a different axis of scaling is evidence that it's not just a "straight highway" to improve LLMs.
From the outside it seemed like GPT 4.5 was the apogee of parameter size scaling and since then a huge investment in RLVR became the new scaling axis.<p>So yeah, LLMs keep getting more powerful/useful but because of an industry wide pivot not a "straight highway" as imagined in the GPT-4 era, when simply going bigger and bigger seemed like a reasonable roadmap for the foreseeable future.
I'm seeing the same pattern at other multinationals with huge R&D departments. Mostly colleagues with academic pedigree in traditional ai research who have been for years in denial that transformers are actually - in combination with recent advanced patterns in software development - the key to ai. it's really simple as that! except they weren't part of the revolution and still don't want to accept the reality that their research has become obsolete.
Is it the key to AI? It probably depends on the squishy definition of AI and what different people want from it. If your definition includes self awareness and sentience I’m not sure it is the key (I’m also not entirely convinced it isn’t, either), but it’s not clear either way in my opinion.<p>The one thing that is clear is if it can become more cost effective it’s world changing. Even at its subsidized standards now it’s world changing. And it’s reasonably classified as AI IMO, but it’s not self aware.<p>I think a lot of your general traditional researchers are fundamentally looking to understand ourselves through machines as proxies in a sense. Something self aware that senses, learns, adapts, so on. The current path isn’t there or necessarily clearly the key to achieving that. That may be the disconnect. I think LLMs may give us some hints at that but not the full picture.<p>Still incredibly powerful for things we once considered nearly impossible.
> still don't want to accept the reality that their research has become obsolete.<p>I see it that most of legacy research is not obsolete but can be significantly amplified by AI.
Im at a company with its own set of machine learning engineers, a few separate organizations internally have them so probably a few hundred MLEs in total.<p>More and more, the talk of using generative models (e.g. transformers) is coming up to replace existing models in all concepts. Anything from understanding data, image recognition, labeling, ranking, etc.<p>As someone who is not an MLE and really doesnt understand these models well, it seems like the transformer approach is a "one size fits all" solution that may not be optimal for all problem categories but is good enough that its not worth investing in specialized models to get incremental improvements.
It's worth it to keep exploiting transformers while doing just a bit of exploration of alternatives because <i>they work so well</i>. Heck, just exploiting transformers more will accelerate exploration of alternatives...
100 trillion parameters doesn't make any sense if your dataset is ~30 trillion tokens. My uneducated guess is 1/10 compression ratio is good and that point to ~300 to ~1000 trillion tokens, which again, doesn't make any practical sense.
Depending on how you count, the human brain is between 100 trillion and 1 quadrillion parameters, which is one guess as to why we're so much more sample efficient at learning than LLMs are.
Synapses and LLM parameters are not equivalent.
One key difference between humans and (current) LLMs is that we can actually judge the material we learn from at the time we're learning it, contextualize it and discard garbage when we spot it.<p>LLMs need an awful lot of handholding all the time, meanwhile humans get pretty good at it on their own.
Unless quantum effects do infact play a role in consciousness. It’s not clear they do by any means, but if it did that could shift the understanding of our own minds and consciousness drastically. Otherwise we are just a dense complex network we can reproduce with enough money.<p>I mean we literally do this every few seconds already with natural birth processes.
The extra parameters are useful for optimization. Deep-learning relies on highly singular solutions.
Yup, we havent seen any limits yet. We have discovered a general learning algorithm. We are in for big things
I would say I used to believe how the models would scale, now I'm a bit more skeptical. I would be incredibly surprised if oai, anthropic, etc hadn't at least TRIED this yet. They have the compute to do it. They're either sitting on it and saving it for a rainy day, or there is something wrong with it.
We know nothing about their internal models that, for instance, produced revolutionary results in mathematics. The public no longer gets to see what’s at the frontier, and we may never see much of it. Also, the cost to train these is immense. China literally does not have enough chips in circulation to compete. I’m not sure where US lab capabilities tap out but I suspect they are at least partly bound by GPU production as well at this point.
We know that if this immense cost had in fact been spent, and the internal models really were that revolutionary, these companies would be selling access to it for cash instead of taking on mountains of debt.<p>We definitely know that.
> there's a straight highway ahead<p>Which many credible, AI researchers believe is already starting up an increasingly steep scaling wall of diminishing returns that eventually ends at a brick wall.
I've been hearing a lot of "credible AI researchers" talk about these diminishing returns of transformer-based architectures since 2023 (at the least), and I'm still waiting to see this promised plateau materialize, because I definitely can't see it in <a href="https://metr.org/time-horizons/" rel="nofollow">https://metr.org/time-horizons/</a> or in my personal experience.
Assuming super-linear growth will continue based on the first 3 years of a new technology? Almost every physical, computational and economic system in history, from Dennard scaling in silicon to transonic aviation speed looked super-linear until it hit the inflection point in its eventual S-curve.<p>A Copernican/Lindy baseline suggests the growth rate in LLM intelligence will peak in around another 2-4 years and I'm leaning on the low side because while the growth of LLM intelligence per dollar and per month are still growing in absolute terms, the growth <i>rate</i> of both seems to already be falling as power-law constraints in compute and data bottlenecks take hold.
This doesn't engage with the article in the precise way that <i>transformer-based improvements</i> and fundamental research that way are not being incorporated into models, at Microsoft AI, which is concerning.
What if scaling fails to hold at a certain point? Whether that is a steep or gradual dropoff in returns. What happens to the empires of accumulated compute? How much value do they immediately lose once the returns to scaling ceiling is reached?
If/when this happens the value generation will be focused on application and the training compute will be shifted towards inference.
At this point I think it'll be a lot easier to find out what happens then and what to do next if you're using the frontier AI
I'm still convinced the <i>model advancements</i> are largely US firms jingling a set of keys in front of investors/markets/the public to distract while they scramble to build serious patent moats around the harness/orchestration/infrastructure technologies.<p>I think that we're well into diminishing returns already. The models have improved, but to me the far bigger advances have been in feeding them contexts, building them into loops, etc.<p>I don't even think much more construction is really necessary in data centers. We can do a lot more with the power/HVAC we have, especially once the ASICs start coming online.<p>It's like we invented the wheel in 2022, the crowds are fighting over capital to build wheel factories, and savvier tech firms are quietly patenting automobile designs while everyone's focused on making slightly rounder wheels.
When a company pays billions to OpenAI, any activity that does not do a glorious dance in praise and support of that investment is taken to be an act of disloyalty.
Aren't there scaling laws with corpus size and parameter count?
> Essentially the messaging it "Foot down, there's a straight highway ahead don't divert".<p>Feels like this is leaving major breakthroughs on the table for the competition. Sure, transformers might get us much farther. But if someone figures out an approach that unlocks a performance boost or better resolution then they are going to kill the competition.<p>Feels like the American industry is high on all the capitol being shoveled into the furnace without caring how efficient the furnace is. At some point someone is going to make a breakthrough and burst that bubble. This could crater the value as we are currently betting on the low power efficiency of the models as the barrier of entry and point of profit. Of course that would lead to cheaper AI but is that profitable vs local models once (or if) it becomes affordable to do so?
> If the scaling continues to hold we'll be asking the AI to do the exploratory research for us soon enough.<p>Already happening: <a href="https://news.ycombinator.com/item?id=50397994">https://news.ycombinator.com/item?id=50397994</a><p>Obviously we'll see how it pans out, but other approaches feel a bit like trying to ignore the bitter lesson.
> Current top models are mere (mere in big datacenter terms) trillions of parameters right now. ... We'll soon start hitting 10's of trillions and 100's of trillions of parameter models.<p>Chinchilla Law says no, we wont. We're likely well past the point of diminishing returns but MOE hides a fair bit of it.<p>We already know that power laws apply to scaling so 10x the size is going to result in minor gains - where costs become a real factor (as if they weren't already).<p>FTA "The core philosophy there is scaling, and aggressively not research."<p>This is where the real gains for AI come from - good enough and CHEAP is going to win the day here (brute force, mixture of models). It's the engineers displacing the academics - and that will mean a bloodbath for revenue (see DeepInfer for what the end game looks like).
I'm not an AI booster (in a "should" not "can" sense), but in terms of "can" we are going to see next generation hardware (i.e. Vera Rubin GPU architecture) and improvements in model density (i.e. loosely improvements in intelligence per millions of weights), so the models will get bigger, the models will get more dense, and we may see more "expert models" i.e. in the sense that for a while there were human generalists that knew all of science and mathematics until the 1800s, after a certain point we had to specialize since there was too much to know. The models will need to partially specialize after hitting the hardware ceiling.<p>Bioinspired algorithms may eventually lead to vastly improved intelligence per a weight too.<p>I don't think we <i>should</i> do this, but I feel doomed in that I suspect that we "can".
> The models will need to partially specialize after hitting the hardware ceiling.<p>I'm incredibly skeptical of anything that goes "oh, we need to split AI into specialist systems".<p>I suspect that a lot of the delta between "our best models" and "base models they're trained on" lies in relatively low amounts of parameter changes driven by SFT and RL.<p>Model capacity is not the limit - the limit is our ability to drive those changes efficiently. Even things like "good research taste" could be represented as tiny in relation to total model capacity, if they could be trained into a model efficiently.
At least for now, those keys haven't been completely unlocked, so parameter scaling is the primary way corporations are achieving increases in ability. If they hit the hardware ceiling, maybe the research direction changes then.<p>I forget who said it, but a trenchant insight is that current neural networks are generalized function approximators and essentially they are doing through brute force what biology accomplishes elegantly. Our hardware (both in the sense of our brains in terms of elegance and our computer chips in terms of raw throughput) is amazing.
> Foot down, get to 10 trillion parameters, see if the scaling holds, if so off to 100 trillion parameters. If the scaling continues to hold we'll be asking the AI to do the exploratory research for us soon enough.<p>Where's the money coming for those 10X and 100X ventures?
Get the transformer to help accelerate research. That's how you get a new architecture.<p>Anthropic, which is more open about this has shown the numbers, it works.
> "Hey this Opus 5.5 with the increased parameters seems much better than the last one". Well that's a trillion parameter model.<p>Opus is 5T, not 1T.<p>> We'll soon start hitting 10's of trillions<p>Fable is already 10T.<p>> "Foot down, there's a straight highway ahead don't divert". I have the view this makes sense too.<p>The roadblock they ignored is politics.<p>Three days ago the states where most of the new datacenters are being built (Texas, Ohio, Georgia and others) elected governors who campaigned on halting datacenter construction. The highway metaphor implies money in => intelligence out. Turns out there's at least one other ingredient, and we just ran out of it.
These corporations have an obscene amount of money<p>This is a false dichotomy if I ever saw one<p>Why not hedge and just keep an ai skunkworks team working? These companies are big enough that they can have these people far away from the main work without distraction or without starving those teams of bandwidth
> Hey this Opus 5.5 with the increased parameters seems much better than the last one"<p>Is this true?<p>I barely notice any improvements ever since... 4.6? At least for software development.<p>Opus makes the same sort of mistakes and has the same sort of strengths ever since that point in my perception. My impressions is that increasing the number of parameters clearly has diminishing returns.<p>In fact, the best improvement I got from Claude was Haiku 5.5. Good cheap workhorse to implement plans.
Well 4.6 was released Feb this year.<p>As in your statement to me indicates that you clearly see the scaling working over longer timespans. You chose a relatively recent 2026 model as your baseline after all, not a 2025 model or earlier. You clearly see that the 2026 model is better than ones in the past.<p>Might i suggest that you perhaps just don't see the improvement as clearly when comparing iterations that are fairly close together? If you're not anecdotally seeing improvement between an early 2026 model and one from ~6 months later perhaps just give it another iteration? Personally i do see 5.5 as much better but i can understand not seeing a lot of difference to 4.6 since they literally aren't that far apart in releases.
> Might i suggest that you perhaps just don't see the improvement as clearly when comparing iterations that are fairly close together?<p>No, you can't. Because you invalidate your argument by adding later:<p>> Personally i do see 5.5 as much better<p>It has to be one or the other.<p>Either each minor release Anthropic farts is a major improvement from what came before, or I don't see much difference because it has been very little time in between releases. Holding both arguments as true would require me to be excessively apologetic for a company that desires a 2T IPO.<p>And to be clear, I do think that there is an argument that not much time has passed since Opus 4.6 and Chat GPT 5.3 Codex days.<p>Part of this conundrum may be because the models reached a good enough point for software development. Maybe walking 80% of the way was easier than the final 20%. I am nowadays more excited about how cheap models can be useful for it (hence my positive remark about Haiku 5.5 and Chinese models in general).
> Foot down, get to 10 trillion parameters, see if the scaling holds, if so off to 100 trillion parameters<p>Moore's law: Computing power is multiplied by 10 every 6 months? Can't wait for the optimization and rationalization phase, where we'll recover a trillionth of that power to run powerful databases.
It seems extremely likely to me that scaling produces some benefits, it will get better at particular things things continue but scaling by itself isn't heading to general intelligence.<p>I should also say that I use ChatGpt daily and enjoy it for researching algorithms. I tried to set up Claude last week and it/he was so overwhelming as to be unusable. So I think frontier models can be "unwieldy" or something for average use.
Ignoramus question: does the 100s of trillions of parameters world still exist with a constrained domestic data center build out? Democrats are signaling they are about to go psycho mode on data center contracts.
If not in the US then in other countries.
Like the sister comment, if not in US then in other countries.<p>US rightfully placed a bunch of labor/environmental regulations around manufacturing which moved to China but stupidly didn't place restrictions on goods manufactured with severe labor and environmental violations.<p>We are repeating the mistake again.
> US rightfully placed a bunch of labor/environmental regulations around manufacturing which moved to China but stupidly didn't place restrictions on goods manufactured with severe labor and environmental violations.<p>The 'US' isn't a monolithic thing. US <i>market leaders</i> convinced US <i>politicians</i> to open US markets to China, and the lack of labor and environmental regulations was a <i>feature</i>, not a bug in the plan.
This refers specifically to "AI Frontiers", a research group within Microsoft Research that focuses on fundamental advances in AI. There is no end in sight to real AI frontiers.<p>Perhaps "The End of Microsoft Research's AI Frontiers Group"
would be a less ambiguous title.
The real news is "The End of Microsoft Research". Langford's group is just one of many that have been parted out over the last year or so. The division that was T&R under Kevin Scott was quietly dissolved about a year ago and MSR was reorged under Azure Security with the AzSec leader as hatchetman for the department. MSR itself doesn't have a future beyond a few years at most I reckon. I don't suspect Kevin Scott will be around much longer either since his department keeps getting reduced and is fully eligible for VEO.<p>Langford has been at MSR a very long time and has done amazing research doing that time. He's bounced around a bit at MSR during his tenure and it's a shame to lose him. AI Frontiers wasn't around that long.<p>Much of this is Microsoft performing a very large and uncommunicated strategy update. All updates of MSR reorgs are just that "MSR is becoming more lean and focused" when it's obvious this isn't the case. This gets into the conversation that many Microsoft employees tend to have these days which is "Why does culture no longer matter at Microsoft and why did Satya turn into another heartless CEO". I won't speculate there, but the place isn't nearly as fun as it used to be.
Yeah, I went down the rabbit hole scratching my head the whole time. This is about the group at Microsoft Research that was disbanded or reorged into what is now Microsoft Research.<p>To the author: 100% need this kind of thinking. It’s a balancing act. Historically though, dedicated labs haven’t had the best track record at commercializing their tech. The skillset differs vastly from running a product-focused business.<p>That said, even within this narrow scope - there are areas ripe for research.<p>Some of my particular interests include:<p>(1) Latent thinking/reasoning, which could arguably be more efficient but also opaque is one<p>(2) Separation of “intelligence” from knowledge - ie. keep the ability to reason like a frontier model and make tool calls (calling the ‘oracle’ or searching the web) but remove all parameters associated with niche knowledge/facts.<p>These are a couple off the top of my head that are very much aligned to scaling.
Ok we'll use that above. Thanks!
> There is no end in sight to real AI frontiers.<p>That statement is broad enough to be unfalsifiable, but if you narrow it down just a bit to scaling LLMs, maybe there is an end in sight. At least financial markets are skeptical of there being no end in sight.<p>If scaling LLMs really was the road to God in a box, then compute demand would be infinite, and nobody building to meet that demand would have a junk debt rating. Did those money people just not get it?
I see this playing the same way in the other big companies that have a pressure to deliver (cough *value for shareholders*)
It makes me sad because there are so many unexplored areas of AI research, even on text generation. Billions are invested in almost one idea, which very realistically isn't the most optimal way to do text generation, but a lot of companies are afraid of taking risks. I still wouldn't put $1b on JEPA but I prefer a world where we can try things like that.
LLMs have very favourable exploration<>consolidation dynamics. An LLM alternative would need to display fundamental improvements at relatively large scales which is extremely challenging for new research.
It's a lot easier to explore these other areas with the help of LLMs.
> there’s a breakdown of the basic deal in MSR: do good work and good things happen<p>This really hit home for me. This is the contract that held for a *long* time at Microsoft and has been broken and rebroken with current leadership. It’s culture-destroying.
He didn't say they were not doing transformers research. In fact, he specifically mentioned a type of transformer they were working on. And labs like Qwen and DeepSeek (at least) have shown how incredible results can be when you innovate in the area of LLMs.<p>I think the real point of the post is that office politics and poor management got in the way of innovation due to infighting and poor execution in transfer of technology from research to product.
To me, all signs point to US over investment in hardware capacity to support the "frontier" when in reality the next major breakthroughs are likely radical optimizations that end in increased consumer training and inference efficiency.<p>I'm not sure how that, combined with open models inching up on frontier intelligence, is compatible with these massive AI IPO valuations?<p>It seems like closed model AI demand has a lot of contingencies that are ultimately fragile and possibly short sighted.
The article is about a guy leaving Microsoft and the disbanding of an internal team called "AI Frontiers".
Yeah, I'm speaking to "scaling vs research" and where investment is going. Which is relevant to the article. Accelerating scale requires demand, closed frontier demand is what I'm questioning.
And gp's comment is relevant to the particular problems the guy (John Langford) cites in his comment about leaving. IE, that research first winds being only about scaling and then winds-up ignored because "just scale" can be an easier answer.
Do you think humanity is just going to run out of problems that need the smartest intelligence?<p>If so, then yeah, increasing efficiency 100x means we hit that point much faster.<p>If not, then it just means we make improvements 100x faster.
Everyone wants radically more efficient AI. That doesn’t mean it’s coming any time soon. And when models are twice as efficient, guess what happens next. We try making one twice as big for the same cost.
Companies with billions of $, stopping R&D on the most revolutionizing technology/ breakthrough is something I can’t understand. I do understand there’s an AI Race to win, and that is with transformer and current LLMs architectures, but if research could find a better, more efficient approach they would re-shape this race, lead it, or even dominate it.
The problem is the "if". There is no guarantee that there's a "better, more efficient approach" to be found.<p>So far, the attempts to replace the transformer LLM with something else have yielded either "the same transformer LLM but with this one tweak that makes it 10% better" or "a radically different architecture that's a bitch to work with, and performs, best case, about the same as the transformer LLM".<p>By now, I'm firmly in the camp of "the architecture is good enough". Rifle is fine. Transformer is fine. Get better at the transformer before you look at anything else.
I think it’s more likely that some future LLM model finds a more efficient architecture than a human does, at this point. So yeah I’d go all in on LLMs too.
They are losing tons of money on it. Simple as.
But is it a race to zero? Maybe AI will make very expensive things (eg software engineering) very cheap, but there will be enough competition that the margins are tiny and don't actually justify putting crazy dollar amounts into research
Almost as fascinating is that the article references Digg as the source - genuinely hadn't heard of Digg for maybe a decade? I thought the DVD encryption fiasco + reddit had ended it?
I think the dichotomy of open-research vs. commercially-driven innovation might be easier to coordinate, with three stages, not two:<p>1. Open research: Great creative freedom, with great resource constraints. Lots of ideas, lots of things tried, at insignificant cost relative to commercial work.<p>2. Results filtering, vetting, regularization: Opportunistic selection of apparently successful work from stage 1, for tentative testing at increasing scale. The best ideas end up with moderate budgets that verify benefits, which are reworked for compatibility before any handoff to stage 3.<p>3. Commercial efforts: aimed strictly at improving quality and economics of current revenue sources.<p>Direct negotiation between #1 and #3 involves too much dissonance over globally complementary but locally incompatible values.<p>#2 can be a relatively small, high payoff group, keeping #1 and #3 from butting heads, while facilitating them both.
Relevant: <a href="https://www.quantamagazine.org/when-chatgpt-broke-an-entire-field-an-oral-history-20250430/" rel="nofollow">https://www.quantamagazine.org/when-chatgpt-broke-an-entire-...</a><p>On the flip side, frontier models are so good and relatively cheap now that it's possible to pursue your pet research project on your own as long as resources are not a constraint.
The headline makes good bait for AI skeptics. But this is a team of 50 people with a grandiose name.<p>The more interesting idea is that MS has abandoned AI research entirely, presumably because their zillions invested in other companies are a better bet in that area.
If MS abandoned a promising idea, pursuing it outside makes perfect sense. I am sure there are investors willing to fund this. This could even be an opening to attack the moat of the current AI frontier model companies.
Curse of money strikes again. $4tn market cap company thinks team of 50 researchers too expensive.
John was the creator of Vowpal Wabbit, which used to be a popular ML library: <a href="https://en.wikipedia.org/wiki/Vowpal_Wabbit" rel="nofollow">https://en.wikipedia.org/wiki/Vowpal_Wabbit</a><p>It was my introduction to feature hashing.
It is notable that with the advancements of 24/7 coding agents with basic research capabilities the premise:<p>> a clever idea pursued by one person it was likely to arrive late and unconvincing because one person could not move fast enough and the baselines which one person could assemble just were not strong enough to really show that an idea was worth taking flight<p>may no longer be true. The limiting factors are: frontier model token and compute budgets, researcher skill, drive, foresight and imagination, not the ability to work in teams. Of course, working in teams still helps, and benefits from internal use, peer review and promotion of the research line are certainly there, but compared to some time prior to 2026, it's pretty much not a factor.
Hmm... So strange. It seems obvious(?) that both research and scale are needed, and that one should never have one starve the other. In fact research can influence the scale needed, and vice versa.<p>I guess when money is concerned people will always seek out the the most reliable and predictable path - at the risk of being potentially left behind if it turned out to the wrong (or at least suboptimal) path.
John Langford was the creator of "VW" ("Vee Dub", or "Vowpal Wabbit"), an ML framework that was quite efficient back in the day ( ~ 20 years ago ).<p><a href="https://vowpalwabbit.org/" rel="nofollow">https://vowpalwabbit.org/</a>
This is the canary in the coal mine. The canary is dead. The AI Winter is coming.<p>This scaling approach is unsustainable, it is generating diminishing returns and ballooning costs. It is already blowing up.
Lack of focus and long term planning at MS
Ironically now by collaborating with AI one person can absolutely move fast enough. I used to work in orgs like that and am now vastly more productive than I ever was back then. It could take me months to convince a peer of an idea that now only takes minutes to explain to AI.
This seems to track with a recent blog post from MAI regarding their approach to pre-training their first Frontier level model which seems promising but does seem to have a focus on scaling.<p><a href="https://microsoft.ai/news/frontier-pre-training-at-microsoft-ai/" rel="nofollow">https://microsoft.ai/news/frontier-pre-training-at-microsoft...</a>
If you value the freedom to pursue your own research direction and share your work freely, then choose academia. All the perks of working in an industry research role are provided in exchange for this freedom.<p>If your research aligns with the incentives of industry, you can get the best of both worlds in industry. This was the case for machine learning researchers in the post-AlexNet, pre-ChatGPT era. The good times can't last forever, though.
>If you value the freedom to pursue your own research direction and share your work freely, then choose academia<p>Perhaps it is more free, but my understanding has always been that professional academia still restricts freedom of research. You have to apply for funding, convince the board that your direction will be fruitul.<p>An archaeologist can't just dig wherever. They compete with their peers for a limitted funding pool. If the school's board is more interested in the bronze age than the paleolithic, you're probably not going to get your dino dig approved.<p>The same is true for CS, I assume. You can't just "i would like spend university resources developing brainfuck 2.0" because the other teams are more likely to get their funding over you getting yours.<p>Both corporations and university's have finite budgets, both with bottom lines and margins. The uni wants their research done in a popular field that will attract students because students generate income that funds research that attracts students that generate income that funds research that...
Academic research is free in the same sense as running your own business. You can try whatever you want if you can find the funding, but you still have to deal with people and regulations.<p>Universities are mostly just administrative organizations. They have little control over your research, they rarely fund it, and they are usually not even interested in it.
That’s not the issue here, it’s not like AIF was doing purely academic work.<p>They just think it makes sense to explore things, rather than just rinse and repeat with more cash.<p>The good news for John is that when 99.99% of the competitors are doing all the same, it’s when the rewards for looking around can be the biggest. Those are the real moments for disruption, and companies disregard them at their peril.
>
If you value the freedom to pursue your own research direction and share your work freely, then choose academia.<p>The working environment in academia is often precarious :-( , so such people often have no alternatives to going into industry, even though they still deeply appreciate and are influenced by the values of academia.
I largely agree. For US though, academia has not been a good place lately.
Wait until you hear where academia gets its funding (hint: it is not no strings attached)
The Microsoft AI (MAI) part stood out to me. I bet the politics of being in Mustafa's group were not fun.
but Mircosoft AI model is growing greatly and recently beat the nanobanana in generate video domain. I can't image the situation of them are bad as you explained.
Is this the same group that did ResNet? Bad news
I'll play devil's advocate. Of course John Langford thinks what he and his group were doing held a lot of promise, and the work was fun and challenging and super important. But in the end you also need to live in the present and deliver. In the words of Jerry Maguire, "Show me the money". He had years to show Microsoft the money, and in the same time other people at other firms showed their bosses the money, big time.
Yes, leave it to the Chinese to advance the state of the art. America throws money at it, the rest brains. Because they don't have the HW.<p>Same happened to computer chess in the 80ies. Clever small SW was leading the tournaments, until Ken Thompson could persuade IBM to throw hardware at it, Big Blue. This win was shortsighted. HW didn't scale and nobody profited from it. SW did, esp. reinforcement learning.
Microsoft's now shown researchers their good work isn't safe.
New AI lab incoming?
That's a name I haven't heard in a while
> scaling is straightforward<p>I remember IllyaS saying scaling doesnt work anymore so we are just RLcrazy. Is that not true?
Well I never thought I’d see Digg used as a primary source<p>This is certainly an interesting timeline in this section of the light cone
Some more discussion on source: <a href="https://news.ycombinator.com/item?id=50406589">https://news.ycombinator.com/item?id=50406589</a>
(In the context of Microsoft Research) Man, the history of how large corps fumble major wins should be studied.<p>I remember during my thesis how Microsoft Academic (a Google scholar/Scopus alternative) was <i>ridiculously</i> good at its job.<p>I'm talking, "input the key words and filter out the wrong fields and you've got 20 highly relevant papers that no other search term has found" level.<p>...Of course, they proceeded to close it down.<p>I can't even express in words how mad and disappointed I was and still am at Microsoft for this alone. It felt the closest to a genie or an oracle in being amazing. (The old Swype vs modern Gboard is probably a close example for those who'd used both.) It's just... crazy, frustrating, and crazy frustrating.
Not what i expected, but good read nonetheless
Executives chasing short-term returns. Google did 20 years of AI research, even creating custom chips, just so we could get to this point. Without long-term strategy Microsoft will perpetually be the also-ran. (which I guess is fine with shareholders?)
Mr. S Altman just said almost all Startups will die within a couple years. This is sad.
Real research at most big companies is dead. Unfortunately, what has replaced it is mostly venture-backed startups, where the timeline to a return is still expected to be quite short.<p>As a result, you end up with companies doing basic research under the guise of developing a product, announcing timelines which are never going to be met. Fusion energy and quantum computing startups are some of the most egregious examples of this.
Agreed. I used to be an industrial researcher at one of the last companies that didn’t require their researchers to think of short-term productization goals. However, things changed at my former company, and I ended up leaving for a tenure-track position at a community college.<p>I don’t know what industrial research in CS is like in the era of agentic coding. On one hand I suppose building prototypes is faster than ever, but on the other hand I genuinely don’t know how well LLMs do with harder programming tasks. It’s one thing to have an agent work on a CRUD app, but I wonder how effective agents are at handling things like parallelism, distributed computing, load-balancing, and other topics that are harder to write than the average CRUD app. I am also aware of efforts such as Sakana AI’s automated scientific discovery research, and I wonder if many of the incremental, short-term research topics many companies pursue can be automated.<p>I’m a bit pessimistic about the state of research these days, from funding to publish-or-perish to the effects of generative AI (automation, AI slop publications, AI slop reviews). Then again, I’m a bit of a curmudgeon, and maybe things are better than I think.
There’s actually a bizarre irony.<p>Remember Bell labs, XEROX parc? Ask yourself why that existed and why research came out of them.<p>Monopoly.<p>So - many people don’t want to admit this but monopolies can be good from an R&D perspective, but without strong oversight by regulators - money will instead spent on R&D projects that specifically raise entry barriers.
I can show you many more examples of monopolies which did not produce PARC or Bell Labs than those which did. It's possible such research happened in spite of dominating the market, not because of it. Anyway, you cannot have both monopolies and strong regulators. These are incompatible situations. Monopolies occur because regulation doesn't exist or is captured.
> Anyway, you cannot have both monopolies and strong regulators...<p>> Monopolies occur because regulation doesn't exist or is captured.<p>IMO, the stronger the regulator, the stronger the incentive to capture it.<p>Unless by "strong" you mean "competent"?
I meant competent, but regulator strength in the way you're implying at its limit becomes monopolistic itself. So when I speak of regulation strength with respect to monopoly, I am speaking to the ability of a regulator to effectively both understand their duty and carry it out.
I have been shorting most of the Quantum Computing startups for about a year. They have been printing. Missed out on Fusion! (?) These were one of few examples of the market leaving money on the table. Others examples have been Nikola. Usually the scammyiest companies don't IPO.
Microsoft is a really shitty place to work right now. There are a hundred stories like this all over the company like this.
They've offered voluntary redundancy to a quite large proportion of the staff (a quarter?) so they'll also soon be losing anyone with employment prospects elsewhere.
I see a lot of news about waves of layoffs in large tech companies. Is MS the only shitty place now? I think all of them are, maybe except OpenAI & Anthropic, since they're building something new and are kind of on the "good" side right now.
Yep, from what I hear and see, it's like this <i>everywhere</i>. It actually started with the Twitter layoffs and has spread across the industry. At first, it seemed like a reaction to the ZIRP-era overhiring, but it's kept going long after correcting for it, and with AI has been cemented as the modus operandi for many companies.<p>From the outside, Microsoft seems to actually be one of the better places, and is making at least token efforts to reduce headcount in more amicable ways, as exemplified the buyout offer.<p>But in general, the strategy seems to be overload people until they burn out and quit, because hey, AI will replace them anyway.
Something tells me the only “good thing” in those two companies is the pay.<p>They look more like complete organizational shit shows.
Yay China!
It seems as if Microsoft is losing developers now.<p>The whole "let's go into AI" is not really compatible
with oldschool software jobs. AI drains humans dry.<p>I mean we are already seeing the hype cycle coming to
a close, with the stock market showing this, but key
people leaving kind of means that Microsoft is losing
its profile here. Add to this that many don't like Win11
and you see a huge paradigm shift here - and tons of
associated problems.<p>"In practice, there are substantial issues because
the philosophy of the ex-Inflection leadership of
MAI is incompatible."<p>AI is killing people here - not necessarily literally
(though we have to see how much economic damage AI
has caused) but figuratively speaking. The counter-wave
is coming here.
More here<p><a href="https://archive.is/AaY5O" rel="nofollow">https://archive.is/AaY5O</a><p>(Original <a href="https://hunch.net/?p=13764006" rel="nofollow">https://hunch.net/?p=13764006</a> )
This should be the main link
Appreciate this, much needed context.
Ok let's put that link above!
Hugged to death like it’s 2008 and you’ve just lit your LAMP stack.
[flagged]
[flagged]
[flagged]
His complaints sound childish to me. If the research products were truly innovative and useful, he wouldn't need to force Microsoft to adopt them.
> If the research products were truly innovative and useful, he wouldn't need to force Microsoft to adopt them.<p>That's just patently not true. History is littered with products that were very innovative and useful (and often far superior to competing products), but were canned, bought by competetors, or shelved in other way - often because they threatened the profits being made by some entity (though that's not the only reason).
You think incumbent megacorps are all quick at adopting new, useful technologies?<p>Some are, some aren't. Some try, but botch it.
Autogen is open source. I don't know the availability of other projects.
Generally engineers in megacorps aren't shy of picking up the latest AI tools. I suppose Microsoft may have barriers around this though, curious to discover more details.<p>Edit: Fara is open weight as well. Seems like a great match for <a href="https://www.microsoft.com/en-us/windows-365/agents" rel="nofollow">https://www.microsoft.com/en-us/windows-365/agents</a> ?
Yeah, I agree. What he describes reflects the exact challenges I've experienced literally everywhere. He is basically saying work/life is hard so he looking for a better playground. I do not know his contributions to AI so maybe he deserves to be better listened to but I find that never reflects corpo life.
[flagged]