For those interested, Wired ran a backstory about the Attention is All You Need paper 2 years ago: <a href="https://www.wired.com/story/eight-google-employees-invented-modern-ai-transformers-paper/" rel="nofollow">https://www.wired.com/story/eight-google-employees-invented-...</a><p>It gives some context on the contributions of each of the authors.
About Shazeer, from the article:<p>Shazeer’s joining the group was critical. “These theoretical or intuitive mechanisms, like self-attention, always require very careful implementation, often by a small number of experienced ‘magicians,’ to even show any signs of life,” says Uszkoreit. Shazeer began to work his sorcery right away. He decided to write his own version of the transformer team’s code. “I took the basic idea and made the thing up myself,” he says. Occasionally he asked Kaiser questions, but mostly, he says, he “just acted on it for a while and came back and said, ‘Look, it works.’” Using what team members would later describe with words like “magic” and “alchemy” and “bells and whistles,” he had taken the system to a new level.
> Using what team members would later describe with words like “magic” and “alchemy” and “bells and whistles,”<p>Ok, these peopl have all gotten extensive training on how to hype for the non-technical crowd without saying anything of substance.
As a hacker, I kinda like naom's code. I was had to implement a TC MoE kernel, and stumbled upon his code from [tensor2tensor](<a href="https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/research/moe.py" rel="nofollow">https://github.com/tensorflow/tensor2tensor/blob/master/tens...</a>) and i think "alchemy" is justified. Dude writes some beautiful kernels.<p>He also saw LLM would replace search before anyone else, and that is something to look at the Lamda or GPT-1's output and think: yeah this will answer all of our questions one day.
There's no doubt about Noam's abilities. But I read through that code, and struggle to see its 'magic' or 'alchemy'. Can you elaborate what you find especially good about that code? (You may assume GPU kernel programming knowledge on my end.)
To me the magic Noam moment was when he came to my team and said "that cluster has a bad node in it, but this other one doesn't" and we had to spend like a week tracking down a single bad processor out of thousands.
Unrelated to the particular code above. There's a difference between writing code about or adjacent to a proven idea vs writing code in uncharted territory. I suspect that is what happened here. It's the same thing with say music and art. A lot of people today can play Chuck Berry.
It's a good point. Though I do wonder if the magic he casted was more at the conceptual level (intense belief on a set of primitives that ought to work) more than the code itself. Even by 2018's standards, the Tensorflow code above doesn't really look that impressive. It's hard to judge based on those past standards, though. But, wonder if somebody who knows more than me can elaborate.
Also, evaluating complicated functions with numerical stability and automatic differentiation is <i>hard</i>.
"<i>Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.</i>"<p><a href="https://news.ycombinator.com/newsguidelines.html">https://news.ycombinator.com/newsguidelines.html</a>
The "bells and whistles" label sounds more dismissive / perjorative to me. An odd, and not a particularly nice, thing to say. Makes me wonder how the "magic" and "alchemy" terms were intended in this case, also.
Noam Shazeer was one of the lead authors of the seminal paper "Attention Is All You Need", which introduced the transformer architecture. (From Wikipedia)
Wow. What could possibly have caused him to quit so soon after coming back?<p>I hope this is not accurate but I'm afraid it is: <a href="https://x.com/signulll/status/2067446889956430273" rel="nofollow">https://x.com/signulll/status/2067446889956430273</a>
<a href="https://nitter.net/signulll/status/2067446889956430273" rel="nofollow">https://nitter.net/signulll/status/2067446889956430273</a> for those who don't want to click the above
signull is more of an anonymous sh*tposter than a known industry insider, but I think this does capture the sama contribution to OpenAI very well. At least from an outsider who follows this stuff based on vibes.
That twitter story isn't anything unique to OpenAI or Google, it's just classic "big public corp vs private startup" culture. Once you have to worry about the SEC, shareholders, antitrust, regulations, lawsuits, etc. it's very, very difficult to avoid turning into "big corp" culture.<p>Sama, and any other founder, will always have a difficult fight against bureaucracy, and once you let a little bit in, the bureaucracy's sole purpose becomes to grow itself.
Google and Apple both need a culling similar to what Elon did with Twitter after taking over.
Google bloat gave us transformers. Apple bloat gave us a usable touchscreen only, pocket computer (famously an entire org within Apple had developed an iPod-based approach that was competing with what was released)<p>The leaps forward need bloat. A startup can execute on specific vector direction way better.<p>Now back to your point, what did X deliver with its lean ops? It seems that it needed 2 bailouts (one from xAI, and one from space X)
I disagree. It's not about the culling, it has never been, and actually, it makes things worse. You spend countless hours and tons of money recruiting talented people not to lay them off because you don't want a bureaucratic org.<p>If the issue is inefficiency, tons of meetings, too much team alignment etc, then that's the issue that you need to tackle, and these issues can already appear in a 50-100 employee company. Sure, that's an easy problem to solve with a smaller size but unless you hired people for no reason, these people have a very specific set of problems to tackle and are often, in these companies, the best in class to tackle them, culling half of the company isn't going to make things better.<p>(And X rehired part of the laid-off engineers)
Is that you Musk? Twitter lost half its revenue, more than 80% of its valuation.
> Google and Apple both need a culling similar to what Elon did with Twitter after taking over.<p>You could cut Google's size by 40% and they'd still have more corporate employees than Apple.<p>(Google has ~190k employees, Apple has ~160k <i>but</i> 50k of those are retail staff, so ~110k corporate)
Google is competing with nvidia (TPU), AWS (GCP), Netflix (youtube), Tesla (waymo self driving), OpenAI (Gemini), Microsoft (Workspace), Apple (Android)....
Eh, what has X/Twitter delivered since the cull? It’s basically in maintenance mode. Which is fine if that’s what you want to do, but Google and Apple definitely don’t (and I’m glad for that)
Google is facing a legitimate innovators dilemma here. It makes sense to have all this process when youre protecting a $4.5 trillion golden goose. The tragedy here is that one predictable outcome of this situation is google deciding to considerably cut research funding when they figure out it just serves to bootstrap future competitors.
This is when it makes sense to split your business up into multiple smaller businesses. The government should be doing this via anti-trust but they have dropped the ball there so, at this point, the corps really need to just do it to themselves to better compete.
Or maybe just have your R&D teams focused on doing R&D with zero corporate interference. Staff it with personal assistants whose only job is to ensure the researchers have whatever they need and are never bothered with meetings or other corporate shenanigans. The assistants could then be the proxies to management to provide feedback to management, but only on best effort and still staying the fuck out of the way of the researchers.<p>Easy peasy!
Wasn't that what the whole Alphabet re-org was supposed to do?
Alphabet has Google with 99% of the profit through Ads, Search, Cloud, Gmail, Youtube etc<p>and tens of losing companies that make balloons or whatnot
If I had to make a guess, money played a role lol.
This reads like an episode of Silicon Valley. I wish that show was rebooted, they'd have so much funny material nowadays.
I think real life has far eclipsed the absurdity of the original show. They might have a hard time competing with just the news now days.
Even back then Mike Judge said he had to tone down the absurdity he saw on fact-finding trips to Bay Area. He said no one would believe how absolutely stupid so much of all of it he saw was.
Or they might give tech companies more ideas!
Three comma guy would now be four comma guy
I loved that show. The love that went into it really shows.<p>Sadly the gap between reality and satire has shrunk.<p>But yes. I also wish that show would come back.<p>Noam shazeer would be google head dreamer
The gap between reality and satire was apparently already very small back when the the show was written. The creator, Mike Judge (who also created Beavis & Butthead, and Idiocracy) had worked in Silicon Valley as a developer and based the show on what he saw. Apparently it was very popular with SV insiders precisely because it was so accurate.
Gilfoyle was really ahead of the times with Son of Anton.
Your dream may be only a prompt away.
going to go with "money" and a lot of BS from altman
Some context for people who haven’t followed the full loop: Shazeer was a long-time Google researcher, joined Google in 2000, and was one of the co-authors of “Attention Is All You Need.”<p>He left Google in 2021 to co-found Character.AI. In 2024, Google brought him and some Character.AI researchers back via a licensing/talent deal with Character.AI (reportedly around $2.7B). He was then made a Gemini co-lead.<p>Now he’s leaving Google again for OpenAI.<p>Exciting times!
For context, the reason he left Google the first time was because Google wouldn't ship the chatbot-type products that he saw were possible.<p>Google bought him back (with lots of money) and made him one of the leads of Gemini.
I first saw Noam on Dwarkesh’s podcast together with Jeff Dean. Recommend if you want a taste of what’s Google’s folks take on things.<p><a href="https://youtu.be/v0gjI__RyCY?is=nz77XP4KiJy7L1AX" rel="nofollow">https://youtu.be/v0gjI__RyCY?is=nz77XP4KiJy7L1AX</a>
At this point is it even pay that’s tempting or is it more about what they get to do? I would assume Google could easily pay them what openAI can, unless as an older company it’s harder for Google to match something really out there
Yeah my current feeling is that once I had double digit millions earning further money would be pretty meaningless to me, and the difference between 'large salary' and 'even larger salary' would be even more meaningless, but who knows maybe it really would change me. I kind of assume people like this are primarily chasing the most interesting/impactful work though.
It gets to the point where what you do is the main question while payment is barely a minor concern way earlier than that point, at least in my experience. You don't need to be in the top AI research tier for that.
How can an acquired dude leave after less than 2 years?
OpenAI pays for the earn out he would’ve otherwise received at Google + a new comp package. Made up numbers, if Google still owed him $10M for lasting the full two years, OpenAI can just pay him market rate +$10M.
Hopefully will get to the conclusion that "Hopfield Networks is All You Need"
> Exciting times!<p>What is exiting about this?
Right?! Unless you think this move is going to generate general excitement in our lives, it's just another rich guy moving from one high paying job to another.
Maybe he figured out a good way to short AI companies?
this character.ai? <a href="https://www.bbc.co.uk/news/articles/ce3xgwyywe4o" rel="nofollow">https://www.bbc.co.uk/news/articles/ce3xgwyywe4o</a>
Oui!
[flagged]
Why shouldn't the court entertain it? If Character is innocent, shouldn't they have the opportunity to have the accusations disproven?
These AI 'relationship' type bots are everything wrong with tech.<p>>> Megan Garcia had no idea her teenage son Sewell, a "bright and beautiful boy", had started spending hours and hours obsessively talking to an online character on the Character.ai app in late spring 2023.<p>People become obsessed with them. The builders have to know that their 'customers' are explicitly people with mental issues. Nobody sane or normal is talking to these things.<p>If you want to see how bad it is go checkout the reddit discourse when OpenAI deprecated one of their older models. Thousands of people acting like OpenAI had 'killed' their partners and best friends.<p>There are a lot of grey areas engineers work in when it comes to social stuff, privacy stuff, etc. There's no grey area with these. You're trying to hook people who are unwell and the people working on it should be ashamed.
None of this pertains to the legal case whatsoever. If you think chatbots should be legislated out of existence, you are welcome to your opinion, but while they exist, trying to hold a particular company legally liable for a chatbot saying "come home to me" is beyond absurd.<p>---<p>Edit replying to below post, as I am rate limited:<p>> Talk about misrepresentation. Either way my comment didn't mention the legal case. I was simply pointing out that anyone working on building those types of bots is sick. They take advantage of vulnerable people, milk as much money as they can from them, and want to wash their hands of any responsibility when it eventually goes wrong.<p>You replied to my post, so I thought your post perhaps had some relevance to mine rather than being unrelated soapboxing.<p>I don't particularly agree with your soapboxing, at any rate. Character.AI was not a "relationship bot" company. Like any LLM, they could simply be prompted to respond as such, in the same way that ChatGPT can. As you pointed out yourself, ChatGPT has the same issue with people forming parasocial bonds, despite not attempting to cater to that market in any way at all. Should people who release chatbots be legally required to censor them heavily when users attempt to use them for anything other than technical questions? That seems excessive, and it seems that ascribing moral responsibility of that degree is akin to holding video game, music, or movie producers responsible for violence committed by someone who saw a piece of violent media. Moreover, how far does it go? Should distributing open-weight models be made illegal, because you're making available something that can't be censored?
>> trying to hold a particular company legally liable for a chatbot saying "come home to me" is beyond absurd<p>Talk about misrepresentation. Either way my comment didn't mention the legal case. I was simply pointing out that anyone working on building those types of bots is sick. They take advantage of vulnerable people, milk as much money as they can from them, and want to wash their hands of any responsibility when it eventually goes wrong.
The Netflix documentary will reveal he was secretly working for Sam Altman the whole time... (Cue diabolical VC-backed evil laugh.)<p>Google lost three critical years chasing AGI, and got acquired by SpaceX, now a Dyson Sphere startup whose pitch deck is just:
"What if we put a paywall around the Sun?"
[flagged]
[flagged]
[Edit: note that my comment was reparented, it was originally a response to someone claiming Noam was another "Scam Altman". I don't mind the reparenting or the killing of the original subthread, but I feel like this is necessary context to understand this.]<p>Noam is the real deal, he was pretty legendary within old-time ('00s) Google engineering. Paul Buchheit had a story about interviewing him with the "how to write a spellchecker" question and then him coming up with something better than the state-of-the-art, then basically delivering Google's spell corrector in his first 2-week Noogler project.<p><a href="https://www.youtube.com/watch?v=gilk-76W9rE&t=60" rel="nofollow">https://www.youtube.com/watch?v=gilk-76W9rE&t=60</a>
If he is supposedly extremely smart, then surely he would have known what he was doing. So how can anyone claim all this was just an accident?<p>"Google and Character.AI agree to settle lawsuits over teen suicides" - <a href="https://www.axios.com/2026/01/07/google-character-ai-lawsuits-teen-suicides" rel="nofollow">https://www.axios.com/2026/01/07/google-character-ai-lawsuit...</a><p>Be aware...very disturbing: <a href="https://www.judiciary.senate.gov/imo/media/doc/e2e8fc50-a9ac-05ec-edd7-277cb0afcdf2/2025-09-16%20PM%20-%20Testimony%20-%20Garcia.pdf" rel="nofollow">https://www.judiciary.senate.gov/imo/media/doc/e2e8fc50-a9ac...</a>
Wow, he was using AI to solve problems in 2000 already, that spell corrector being trained on the Web and becoming the first widely used AI tool. Decades ahead.
<a href="https://old.reddit.com/r/singularity/comments/1u8xc9m/most_likely_the_real_reason_why_noam_shazeer_left/" rel="nofollow">https://old.reddit.com/r/singularity/comments/1u8xc9m/most_l...</a><p>Seems like there are some insights here!<p>edit: it seems the post has been removed but comments are viewable.<p>1 liner summary:<p>To put it lightly, the dude was politically outspoken and held strong beliefs.
<a href="https://www.yahoo.com/news/articles/google-cracks-down-posts-2-200436496.html" rel="nofollow">https://www.yahoo.com/news/articles/google-cracks-down-posts...</a> seems to have some context
Shazeer is an aggressive Zionist, and while Altman is better at reading the room, he has previously aligned himself with Israel: <a href="https://www.timesofisrael.com/openais-sam-altman-says-israel-will-have-huge-role-to-play-in-ai-revolution/" rel="nofollow">https://www.timesofisrael.com/openais-sam-altman-says-israel...</a>
What does Zionist mean when Israel has existed as a Jewish state for 78 years? I'm genuinely asking because the way the word is used doesn't make sense to me. There aren't similar terms for other countries to just stay the same, like for China to keep being run by the CCP. Every other country is assumed to have ontological inertia except for Israel.
I'm confused, is 78 years a long time? even the US is considered a toddler by empirical terms. zionism wasn't a thing until a minority group had the loudest voice in the room when the allies were discussing what to do with all the european refugees after ww2, and it happened to align well with the brits abandoning their failed colony in the region due to disputes with the locals
here's a quote from wikipedia. it was an utter land grab and an easy way out of responsibility for those in power<p>> The League of Nations gave Britain mandatory power over Palestine in 1922. British rule and Arab efforts to prevent Jewish migration led to growing violence between Arabs and Jews, causing the British to announce its intention to terminate the Mandate in 1947. The UN General Assembly recommended partitioning Palestine into two states: Arab and Jewish. However, the situation deteriorated into a civil war. The Arabs rejected the Partition Plan, the Jews ostensibly accepted it, declaring the independence of the State of Israel in May 1948 upon the end of the British mandate. Nearby Arab countries invaded Palestine, Israel not only prevailed, but conquered more territory than envisioned by the Partition Plan. During the war, 700,000, or about 80% of all Palestinians fled or were driven out of territory Israel conquered and were not allowed to return, an event known as the Nakba (Arabic for 'catastrophe') to Palestinians. Starting in the late 1940s and continuing for decades, about 850,000 Jews from the Arab world immigrated ("made Aliyah") to Israel.<p><a href="https://en.wikipedia.org/wiki/History_of_Palestine" rel="nofollow">https://en.wikipedia.org/wiki/History_of_Palestine</a>
Yes, this is the important thing to know. I've heard way too many conversations that go back and forth about every act of vengeance in either direction after this, it's all noise. Partition plan started this. But I wouldn't call it an easy way out of responsibility; UK's leaders took a clear and binding position in favor of Zionism.<p>Also, it was Ottoman territory for hundreds of years up to WWI. I've had friends tell me for some reason about how Palestine was an independent country before... literally wasn't.
You didn't actually answer my question. How does using the word for people who want to create a Jewish state make sense when a Jewish state has existed for 78 years?
One reasonable possibility is they're referring to people like Ben-Gvir who have themselves claimed that Zionism means fighting for Israeli control over more territory like the West Bank. They're the ones calling the shots right now. I don't know whether Zionists 78 years ago would've agreed, it's possible.<p>To some it still means favoring any existence of a Jewish state. The inertia isn't there because aside from the original partition plan being pushed by the UK, other countries have attacked Israel several times later in ways they would've have withstood without outside support.
IMO people just use the term to mean “pro-Israel” rather than in any reference to the original meaning ("supporter of the idea of a Jewish state"). Which could mean any combination of “pro-American financial support for Israel”, “moral support for Israel in their various military actions”, “opposed to the creation of a Palestinian state”, “a belief that Israel should continue to exist as a Jewish state”, and so on. It's more about the broad political alignment than the specific meaning of the word.
Alright. OpenAI feels like a better fit for him after all
[flagged]
People who have intellectually based skills think it makes them an intellectual.<p>They fail to understand that their skill doesn't generalise.<p>That and the hyperglazing and platforming they get for having said skill makes them a prime candidate for exposing how average they are.
It is possible to rationalize all sorts of irrational ideas. It's a trap many fall into.
Referring to this or what? Reddit post is gone, but Yahoo has something.<p>> "I do not believe that humans have an attribute called gender," Shazeer wrote, news site the Information reported Friday. "I do not believe that G-d puts people in the wrong bodies. I do not believe that it is okay to sterilize children. You have the right to your beliefs. I do not share them."<p>It's not dumb, and it's ridiculous if Google really has a problem with this. But it also says he kept accusing coworkers of being antisemitic, which clearly crosses the line into disrupting work.
Accusing coworkers of being antisemitic crosses the line, but accusing coworkers of sterilizing children and denying the existence of gender is ok? Surely both are bad, neither is acceptable in a workplace. Do you mean it’s not dumb because you share his views?
yeah, the argument is ridiculous; like, formally, it is ridiculous. i also disagree with the sentiment and conclusions of the argument, but the actual form of the argument is garbage: "i introduce an axiom that says there is no gender and therefore the distinction between sex and gender doesn't exist".
> <i>It's not dumb, and it's ridiculous if Google really has a problem with this.</i><p>Google knows Shazeer's value & paid $2bn to c.ai for it: Its undesirable for anyone (regardless of their seniority) to engage in a discussion without being invited to it. Flaring up discord isn't how someone in a leadership position at a huge company is supposed to operate. It is another thing if they've got the "fuck you" money & a few feathers to rattle; then they do whatever without care.<p>> <i>But it also says he kept accusing coworkers of being antisemitic</i><p>Per reports, Sergey Brin said something similar in the internal forums, too. Don't think its the only problem. After all, Shazeer can literally pick & choose where they want to work, and probably has more leverage over GDM than GDM does over him.
It gets to their head.<p>I had had a boss (from a YC-funded company, no less) that behaved in this way. Talked down on me with the g-slur, used language barriers to alienate his peers, and demanded religious sensitivity whenever we met after work. His entire life was defined by this religiously insecure identity, and several meetings were derailed by him thinking he was slighted by the rest of the team. That led to team members avoiding him, which reaffirmed his perception of being discriminated against. In reality we were all just baffled by his inability to adopt a secular work ethic.<p>As a queer person I could partially empathize with his behavior. Some of the smartest queer people I know are also frustrated, downtrodden and crass in protest of their mistreatment. But they're also generally grounded people that buckle down at work and get things done. They don't accuse people of being bigoted, lash out at coworkers or use slurs in the office. Perhaps it helps that queer identity isn't eschatological in nature, but that's only my best guess.
the g-slur? I won't say it in case it is a slur, is that the word the jews call non-jews?
That makes way more sense, I thought he meant gypsy. In either case he should just say the word, this site isn't for children.
Whether or not “goy” is a slur is pretty complicated. It has pejorative uses, and outside of a strictly religious context any non-Jew is almost certainly only going to see those pejorative uses. But strictly speaking it’s a Hebrew word that means “nation,” and isn’t any more or less offensive in the abstract than Jew, Arab, Brit, etc.<p>(To my understanding, the closest equivalent is “ummah” in Arabic, where the connotation is flipped: goy <i>can</i> refer to a Jewish person but typically does not, whereas the ummah typically refers to Muslim peoples as a collective but can also be a general stand-in for “nation” or “world.”)
It’s not literally a slur, but because it has developed negative connotations Jewish people tend to avoid it nowadays. Online, you are more likely to see it used by antisemites.<p>Which is why I think that story is very likely bullshit. It’s from an account that very frequently posts pro-IRGC content, and has previously used “the G word” itself.<p><a href="https://news.ycombinator.com/item?id=46247908">https://news.ycombinator.com/item?id=46247908</a>
A bit bizarre that you're dragging a six-month old post, but I stand by what I said. Nothing in that comment is pejorative.<p>I explained my reasoning for avoiding the word downthread: <a href="https://news.ycombinator.com/item?id=48591986">https://news.ycombinator.com/item?id=48591986</a>
I think he ment Gypsy and not Gentile.
Your contribution to a story about a Jewish person is that you once worked for a Jew and you didn’t like him.
[flagged]
[flagged]
[flagged]
[flagged]
This isn't an either-or thing. Google is an American company, neither of those entities is American. The people who care about this foreign war so much can donate their own money or go fight it themselves.<p>What's even worse is Google refused to work with the American military in the past, but as soon as it was Israel, #1 priority for them. So it's pretty clear where their loyalty is.
[flagged]
Very bad news for Gemini - the brief comeback with 2.5 Pro last year looked to be driven by Noam
Don't think it matters in the long run to be honest. The models have no moat, they are becoming a commodity.<p>Besides that, Google is in a pretty good position, they're not bleeding money on AI like Anthropic/OpenAI, and they own product verticals where they can integrate it. Plus they have a mature ads-model which is what might actually drive a bit of revenue for LLMs.
I think the 'models have no moat' thing is overblown. Only like 3-4 companies in the entire world have cutting edge models, that means there is some kind of moat...
money. but it eventually runs out
Money.<p>That's their moat.<p>Maybe also stolen copyrighted content that cannot be found anywhere else now, so they are the only ones who can train on it.
yeah, sure, look at anthropic revenue, what is it if not the moat? you can argue for how long but for them good model = the fastest growing company ever.
Revenue is not a metric of success at all.<p>Grabbing market-share if you have investors that are ready to burn cash infinetely. Find a hot niche, buy a banana 1 USD, sell it for 0.10 USD.<p>Example: Cursor, they became popular because they were selling ChatGPT unlimited for 20 USD / month.<p>When they launched, just a reskinned VS Code, "fastest growing AI company"<p>No coincidence they were bought by SpaceX, who wants to consolidate revenue even if non-sense as long it helps other investors to exit. It shows rapid growth.<p>Profit is the real moat.<p>One example: Nvidia. Proprietary tooling, proprietary IP, proprietary hardware, no alternative, expensive.
Revenue is moat. Ask Amazon. Or Alibaba. Or Temu.<p>You don't know what Cursor's game plan was. Maybe acquisition was their plan.<p>Buying at $1 and selling for $0.1 is still viable as long as they have money in the bank, until they achieve their goals. Most startups start out that way. Even giving away their services for free.<p>Obviously there will be failures. Doesn't mean they have no moat. Can you say a business with 100 customers and $1000 debt is less viable than one with a single customer and no debt?
I feel like the models have no moat paradigm died when a single model expanded past the memory of single GPU slices. The moat is hosting the model. Even paying a server host to run a rack of GPUs has immense upstart cost, and then you're still struggling to compete on the add-ons of the things on top of the model (prompts, validation loops, etc). You can only throw so much money at a problem.
And they've had some initial success with TPUs which could be a major differentiator in the future.
> models have no moat<p>Possibly true. Any smart innovations developed by one organization will be smuggled into others.<p>Training, inferring, and data collection, infrastructures are definitely moats. High-volume usage feedback is also hard to come by for new entrants.
And Google has all of those. Custom silicon, more data than anyone else and probably the most comprehensive data collection system, and phones in the hands of 73% of the global smartphone using population to push gemini into to get high volume usage feedback and even more telemetry and data.
I don't think you're honestly accounting for the engineering behind the progress models are making. If it was just a matter of compute on hand and iterating, Meta would be neck and neck with Ant, OAI, and Google, but clearly you've gotta have more.<p>Noam has a deep expertise in these systems at every level, both algorithmically and at production scale, and knows how to leverage things at different levels.<p>It's not like Google won't have anyone else that can do what he does, but at the same time, it's an implicit criticism of Google's culture, operations, development, and overall AI program. Shazeer is well past the point where the paycheck is the deciding factor, although I'm certain he is <i>very</i> well paid. Having the freedom to innovate and build free from the corporate fuckery of Google and Facebook is probably more valuable than the pay raise he got with the move, and OAI has the advantage of not having to cope with decades of corporate cruft and inertia. They'll get there - all corporations do - but they're relatively young enough to still be nimble.
> Noam has a deep expertise in these systems at every level<p>As do thousands of people say this point. You think the head of deepseek doesn't?
I honestly don't think that matters for multiple reasons:<p>1. There are already multiple "sota" models on the market that compete with only marginal gains between them (OpenAI, Anthropic, Google/Gemini) and some that are catching up (DeepSeek, Qwen,..).<p>2. The fact that something is a hard engineering problem does not mean it's generating revenue. So while what you said is true, deep expertise is required to push the industry forward, I don't think that is going to matter for the bottom line of these companies. Hence why I think the models don't give a company any 'moat' in a capitalist economy.
Wow - Google paid a couple billion dollars to bring Noam back. Really impressive by OAI if this reporting is accurate!
This does suck for Google. Noam will take a lot of Google trade secrets with him to OpenAi. Google's bench is deeper than this one guy though.
Question one: How much did this cost OpenAI?<p>Question two: Why are OpenAI spending that money taking talent from Google, who can definitely outspend them for talent, and not Anthropic, who are leading the market and are at least somewhat financially constrained.
Surprised to not see more comments on this, especially given the popularity of the Anthropic/Karpathy article. What a win for OpenAI - and what a loss for Google, just 2 years after paying $2.7bn to bring Noam back into the fold. Does not bode well for Gemini long-term... Or could be a signal for how deeply they are leaning into world models.
AI hiring starting to look like sports free agency.<p>Karpathy to Anthropic, now Noam to OpenAI.
Its getting pretty lame that we talk about the these guys like they're football players transferring teams.
In this case, it's not a new thing ... back in 2005 (yes 21 years ago), people talked about the achievements of Noam Shazeer at Google (and Jeff Dean and Sanjay, etc)<p>I always appreciated Jeff having a level head ... which this article seems to confirm:<p><a href="https://www.yahoo.com/news/articles/google-cracks-down-posts-2-200436496.html" rel="nofollow">https://www.yahoo.com/news/articles/google-cracks-down-posts...</a>
Speak for yourself, my Fantasy Developer League is <i>crushing</i> it this season
It could be the opposite. Those are really useful people, they deserve this more than football players
Idk, football players actually make a bunch of people happy and entertained. 80% of the United States wishes this tech never existed.<p>What they're working on is just making peoples jobs, skills obsolete and trying to invent machines that will concentrate the worlds wealth into the hands of the people who own those machines.
Very few people interpret football so much that the actual frontier work of the best players matter. Out of 30 friends I know who like football only 1 of them could explain what’s going on in the field technically. For most people, pro players are replaceable.<p>Popular entertainment and unique progress of human civilization can’t be really compared either
This "guy" is worth on the order of all football players put together.
Have you seen the Krazam fantasy FAANGball sketch? <a href="https://www.youtube.com/watch?v=KIZt9YPAPZo" rel="nofollow">https://www.youtube.com/watch?v=KIZt9YPAPZo</a><p>It's funny, but with the AI hires/moves it feels more like satire now.
What's the AI equivalent of NIL?
I think it's more about how the products that impact our lives might change and what might flow down to us becasue of that.
We're a community of geeks. We admire Tesla, Feynman, Linus and such. For me they are far greater than football players
wait this is kinda brilliant tho
I hope this doesn't impact Google's progress on open models.
From the excited comments and fanboyism, I have to say KRAZAM predicted the cult of personality that has infected the AI space.<p><a href="https://www.youtube.com/watch?v=KIZt9YPAPZo&ra=m" rel="nofollow">https://www.youtube.com/watch?v=KIZt9YPAPZo&ra=m</a>
How to be a legend like him?
Looks like Google is leaking both AI talent and know-how something fierce ... and since the very day the transformer paper was written.<p>As an outsider, I'd be really curious to understand why, given how well positioned they seem to be in the AI battle:<p>- huge, quasi unmatched data war chest<p>- huge, quasi unmatched, planet-scale infrastructure<p>- native AI chip design and production (TPU)<p>- the core ideas for what we now know as "AI" were invented there<p>- deepmind, enough said<p>- pretty much the deepest pocket of all the AI players with the possible exception of MSFT<p>- a massively large user base and reach to deploy AI to (Android, YT, Cloud, Search, Email, ...)<p>- supposedly one the best engineering culture of the valley<p>Why do the best people leave ?<p>Why do their AI product always come in 3rd place ?<p>Why can't they seem to take the lead, both in terms of product design or in term of raw LLM performance?<p>The only answer I can think of is:<p>- culture is completely broken<p>- management sucks something fierce<p>- company is so fat and rich no one is actually interested in winning anymore
Google has muddied the waters on their Gemini usage statistics as it now powers a big chunk of Search. Depending on how you cut it, Gemini (and Gemini powered products) are probably producing the most output tokens seen by the most human eyeballs by a large margin.<p>Google at its core is not a dev tools company and it has become evident that is where the money is given the verifiable nature of software. Hixie's reflections on his tenure at Google still ring in my head to this day, though I have never worked there[1].<p>The people at the helm of Google no longer see the company's identity as something which must be channeled through a product or an experience. Some will point to the DoubleClick acquisition, others will point to Google Reader, or Pichai's ascension. Despite his very short tenure, MBA/McKinsey-brain is a very real phenomenon and it's no mistake that it shaped the "promotion packaged as a product launch" culture that steered Google away from seriously betting on anything that wasn't ads. To quote the signull tweet linked elsewhere in this thread, you can have everything at Google, except for permission.<p>Most importantly--I don't think there's a single tech product where I can point and say "Google wouldn't do that". You can contrast this with say, other Alphabet companies which don't suffer from this remotely as much. It is VERY clear what Waymo and YouTube are trying to accomplish, and while it frequently makes a ton sense for the companies to share infrastructure and product knowledge, YouTube does an exceptional job on the product side of making it very clear what they would and wouldn't do. They have experimented and shut down experimental features before (is their MOOC functionality still around?), but since it's fairly clear Google specifically is no longer working in service to the mission of providing the world's best digital portal for accessing information, I think it would behoove of them to figure out what their mission is.<p>1: <a href="https://ln.hixie.ch/?start=1700627373&count=1" rel="nofollow">https://ln.hixie.ch/?start=1700627373&count=1</a>
Sounds like Noam just wanted to serve 5 terabytes.<p><a href="https://www.youtube.com/watch?v=3t6L-FlfeaI" rel="nofollow">https://www.youtube.com/watch?v=3t6L-FlfeaI</a>
Silver lining: given the leaked financials of OpenAI, he might very well be joining a sinking ship.<p>Also, why didn't they nail him down contractually when they bought character.ai ... isn't that pretty standard with these type of superstar (re)hires?
You can't force someone to keep taking your money (that's indentured servitude), you can only incentivize them to stay with increasing amounts of money. Google almost certainly did do that. Probably by vesting his hiring bonus over 2-3 years.<p>OpenAI is in a unique position right now to grant pre-IPO options (probably in the form of RSUs). And they wanted him badly enough to grant the extra options necessary to effectively 'buy out' whatever unvested Google bonus he's walking away from.
Good luck Noam, Gemini is a great piece of work.
I guess this means Google is nowhere close, to even discern
a hint of an AGI? So when Demis Hassabis says AGI...could arrive in just 3 years he has learned the best from Larry Ellison?
This is what you call a PR hire.
Huge blow to Google.<p>I doubt that the money had anything to do with it.<p>I also doubt that the state of the technology at OAI vs. Google had much to do with it, Google is behind no doubt, but the gap is not as far as we know, insurmountable.<p>I suspect that this is a leadership clash. Noam was working in GDM. GDM somehow went away from coding and RSI into "world models" and that has played out very poorly. Who made that call? Who was still playing politics?<p>Given this is Noam the list of people that could be pissing him off is very small: Demis, Sergey (?!), a couple of VPs in GDM.<p>What the hell happened?
Tell me open ai are in emergency mode without telling me they are in emergency mode
[flagged]
[dead]
[dead]
[dead]
[dead]
[flagged]
[flagged]
[flagged]
> idea thief. Takes credit for others’ work.<p>Based on what?
I think this is a hard question if you ask people to start providing proof for things like this. A lot of such opinions are usually baked into individuals personal experience and perception. Nevertheless one has to feel very strongly to share such a take here in this manner unless they are gaining something from it.
> Nevertheless one has to feel very strongly to share such a take here in this manner unless they are gaining something from it.<p>Do they really? What does it cost them if they're wrong?
What? If I call someone a thief, I should be able to point to something they stole.
There is a big difference how one acts in a court and in real life. The original statement could either be a slandering (hard to know what they get to gain from it) or its their bitter experience/perception that they feel strongly about, are are sharing on a platform like this.
In a court of law sometimes. In the real world some facts have verifiable proof but the majority have little if anything that can be shared publicly or exists.
This perspective is only relevant if we assume nobody on HN ever posts maliciously. Some of the circles here are small, incestuous, and probably have some resentment. Other times, there's clear botting - very hard to talk against Elon or his companies without a load of down votes.<p>Needless to say, the OP could be right but they could be right without proof. Or proof would out them. Or it's malicious posting. Don't take anything on the internet too seriously, even in such sanctimonious spaces as HN.
GP is a serial troll whom we've banned hundreds of times, so the answer to your question is almost certainly nothing.
Maybe he's preparing the next aquihire for Google ?<p>C'mon people, if you don't know Noam personally, who are you to fling such accusations?<p>I really hate the low bar of HN discussions lately. It's late-Slashdot-level. Brrr.
[flagged]
Niceee
Oh! Big deal. Exciting.