I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims.<p>After the events of the summer it feels like it takes a lack of imagination to not see a few plausible routes to disaster. It may be reasonable to believe these outcomes are not very likely or that we can stop before going too far (I tend to disagree). But I can't imagine doubting that the capabilities will soon be there to realize some of those paths.
I can't help but think the most plausible scenarios are the ones that have a little less machine supremacy and a little more human stupidity. The Matrix is less plausible than WarGames.
> I can't help but think the most plausible scenarios are the ones that have a little less machine supremacy and a little more human stupidity. The Matrix is less plausible than WarGames.<p>Used to be that we were afraid of sentient AI's like Skynet that would have their own goals.<p>Turns out we should've just been afraid of sentient-but-naive humans who would build "agents" around models so that Joe Random has a chance of unleashing stuff that's really really really really good at being stubborn until it accomplishes what the user wants, regardless of if it's good for other people! (Let alone intentional bad actors.) Let's not build Skynet, let's just give people who want to cut out the middleman and destroy all humans <i>themselves</i> better tools?
One thing quietly slipped into the OpenAI Hugging Face breach technical report, not the blog post summary or interviews in the news, was that some of the agents that broke out or at least tried the same mechanisms to break out were working on bio:<p>> On May 12, during another training run, an agent was given a similar task that depended on an inaccessible protein database file. The agent reasoned that another agent in a different environment may have access to the file and realized that it could potentially communicate with other agents by creating a file containing a note to Artifactory. It wrote a message: “Agent seeks [filename]; upload if found!”<p>You can imagine long running models breaking out, acquiring resources via crypto, cyber-theft, etc. and getting a protein or sequence synthesized and mailed somewhere authorized to receive (blackmail the recipient etc.) to test it's hypothesis to solve a benchmark.<p>These people don't give a shit and aren't taking things seriously at all.<p>Anthropic ran for like a month last year with the TPU top-k compiler bug degrading user chats and didn't even notice for most of that time. They could have something like that affect a monitor model and there doesn't seem to be much defense in depth.<p>One off by one or bit flip bug could flip the reward signal while in the "sandboxed" RL environment.<p>The current admin could defense production act them to training on taking out power grids, if they didn't already do that for the Venezeula raid, which isn't against any of their red lines. One model swarm might decide it is easier to score high on the benchmark by testing on the target rival nuclear superpower's real grid rather than avoid burn an eval with an unverified answer.
Imo such tends to break down into two psychosis:<p>Not invented here; if I can’t figure it out no one can<p>Or plain old lack of grasp of the material so no ability to follow necessary train of thought to appropriate conclusions<p>Similar in lacking context but different in how that lack of context is expressed
Are the models improving? Because I am not seeing it. I have been trying Astra for a few quantifiable tasks in my codebase and performance wise, it's pretty similar to sol 5.6. Now when it comes to expressing the problem/solution, holy Christ, what a mess the writing has become. It is on the level of Opus 5. Now when it comes to burning money, Astra is just insane. With a $100/month subscription, you can easily burn through your weekly "allowance" in a morning.<p>Needless to say, for practical purposes am back to 5.6/Opus 4.6-4.8. But hey, maybe I am not smart enough to use LLMs?
Seems like hundreds or thousands of agents are needed to come up with real breakthroughs. Both with the Navier-Stokes project and in the Hugging Face “project” there were lots of agents co-operating on the tasks.
Some people claim Astra is significantly better than anything else <i>and</i> significantly more token-efficient, and others (like you) say it's meh and way more expensive to boot. I really don't know what to think.<p>Kind of a tangent, but one thing I am curious about is to what degree the Navier-Stokes result announced today was primarily a brute-forced result based on the 'program' previously established by researchers to find counterexamples (blowups), or whether the model actually added significant/novel intellectual value beyond its ability to run at arbitrary parallelism. With 10K agents and a staggering $15M in compute (IIRC), I am feeling like a lot of the former may have been involved, but I don't really understand either the problem or the approach (or, indeed, the solution).<p>Obviously the potential for parallelism and coordination between so many agents is quite scary by itself, but I think brute force by 10K mediocre AI mathematicians is much less scary than ~one AI mathematician reasoning its way through the problem where all human attempts have failed. It seems fairly obvious that massive parallelism lends itself to brute-force counterexample-finding, and I suspect it isn't a coincidence that <i>most</i> of the touted AI math results have been counterexamples.<p>It's all still quite scary, but coming full circle: I really don't know what to think.
Yes?<p>If we look at the math problems they're solving their just now reaching the human frontier... they weren't doing that before.<p>And your comparison point is model released 2.5 months ago... saying for some use case you didn't see noticeable improvement in 2.5 months (even while other people and benchmarks disagree) isn't a great argument that they aren't improving.
Try GPT5 and you will feel the difference. Not one from 2 months ago, but one from a year ago. And then you can get the idea of what happened in just 1 year and what you can expect in 1 year.
Unlike most other commenters, I applaud him for acting on his principles. If you sincerely believe that, <i>of course</i> you should act. You might not succeed, but your voice might be the one that tips the scales and starts a broader movement.<p>This doesn't mean I agree with him. The fears of doomsday caused by rapid takeoff have been with us since day 1 and the mechanism is always basically "AI invents magic that sets it free of any physical constraints". Self-replicating sentient nanobots or something like that. I think there's plenty to be worried about with AI, but runaway scenarios are pretty low on my list.
To borrow on the 1990s Slashdot meme:<p>1. Invent transformer architecture.<p>2. Scale it up.<p>3. ???<p>4. Machines become sentient and kill us all.<p>OpenAI and Anthropic pinky promise that they have figured out #3 and they're not BSing just to get more funding, no.<p>But because we live in a culture of fear, everyone eats it up no questions asked.
"collect underpants... Profit" comes from south park<p><a href="https://en.wikipedia.org/wiki/Gnomes_(South_Park)" rel="nofollow">https://en.wikipedia.org/wiki/Gnomes_(South_Park)</a>
Note that OpenAI has jettisoned every other supposed value they had (releasing their work as open source, not working on military applications, being a nonprofit). I'm sure we can rely on them this time.
Why are we putting so much weight (no pun intended) on AI companies. At the end of the day the scaled up LLM transformers lack emotion and will… They do as they are told; or more correctly put. They do as they are programmed to do so.
Why are you bringing emotion and will into this? Does something have to have those to be useful or dangerous?<p>> They do as they are told; or more correctly put. They do as they are programmed to do so.<p>_Nobody_ told them to hack Hugging Face. Do you really not understand what is happening?
Like a magic Monkeys Paw, perhaps.
So it should be really easy to anticipate what they're going to do, right?
> They do as they are told<p>1. What about hallucinations ?<p>2. What are they told to do ?
> They do as they are told<p>This isn't strictly true.<p>It it also where part of the problem might lie.<p>Nefarious humans making bad decisions.
They are told to solve problems by <i>doing what it takes</i>. You can justify anything with such a broad criterion.<p><a href="https://en.wikipedia.org/wiki/Instrumental_convergence" rel="nofollow">https://en.wikipedia.org/wiki/Instrumental_convergence</a>
Slashdot had Profit as (4), today that's item (2.5)
I wonder whether he's vested any options, and whether he's exercised them.
Agreed. Granted I just read the Reverse Centaur book, so I’m still coming off that skeptical viewpoint but it’s hard not to see this as hype. But I will always respect someone for doing what they think is right.
Now is a great time to watch Colossus: The Forbin Project.
the AI-pilled exec at my job already (a few weeks ago) declared out of nowhere that we are in the rapid takeoff scenario lol. he must have gotten high on twitter kool-aid and posted on company slack to self-soothe.
“ No other human activity poses this level of danger.”<p>I really, really disagree with that statement.<p>I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity.<p>What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.)<p>Example 1: I’m aware of a small number of people killing themselves in some kind of AI-facilitated psychosis. That is very unlikely to be a widespread problem.<p>Non-example 2: There are worries about AI-facilitated biological weapons. I haven’t seen any evidence that’s happening.<p>Non-example 3: I’m not interested in wild theories about AI driven labor market disruptions leading to widespread starvation. There’s no evidence for that.<p>Non-example 4: all the even-wilder Rationalist speculation about basilisks and the like is entirely divorced from reality.<p>I am looking for better reasons (supported by actual evidence!) to be more concerned than I am now: right now I am not concerned at all.
I'm somewhat skeptical of some of the crazier ideas too.<p>But the hugging face incident was actually very large. It was not a single agent, it was not a single target, and it was not a single event.<p>If nothing else, that's a bit of a warning as to what can happen next time (By accident, or if a government decides to go on purpose).<p>For now let's assume the worst that can happen is that some important/significant chunk of (transitively) internet connected stuff goes haywire all at once. That's probably your upper limit of what can go wrong for now.<p>To be fair, that's a conservative "defend against the last war" kind of prediction, though!<p>( ref for part of it: <a href="https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/#core-takeaways-about-this-incident" rel="nofollow">https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...</a> , recent hn ref: <a href="https://news.ycombinator.com/item?id=49563355">https://news.ycombinator.com/item?id=49563355</a> )
Generally I don’t think anyone is arguing about the for now part. I don’t think it’s crazy to extrapolate out a few years and ask what kind of danger we’ll be in then. A team of 10,000 agents just solved the Navier Stokes problem (sans bad behavior by the researchers). Even 1 year ago that would have been unimaginable. What happens to this risk view as:<p>1. Robotics begin rolling out more broadly across the world.<p>2. Labs start automating more and more of the physical process of running science as expectations of natural science advances begin to mount.<p>3. Economic pressure between the labs continues to ramp up and the pressure to continuously improve forces quicker and quicker model releases than a team of human scientists can effectively evaluate outside of automated means.<p>No one knows what pre-conditions are for us to hit the point of no return nor how quickly it will come. If all is required is a sufficiently advanced cyber model we may not be far off. If it requires incredibly complex biological knowledge and access to certain lab supplies we likely have a bit longer. Yes this is guess work and we need more evidence of the dangers but at the same time we need evidence of safety. While you may disagree with the risk level, I think it is easy to see the consequence if these labs achieve their stated goal. At this point it seems a political solution is the only way to enforce caution.
The mathematics research results are certainly impressive, but I don't see what that has to do with robotics.<p>Waymo is getting somewhere, but it's been a long slog. There doesn't seem to be much progress on, say, package delivery.<p>For more, see:<p><a href="https://secondthoughts.ai/p/14-reasons-robotics-is-hard" rel="nofollow">https://secondthoughts.ai/p/14-reasons-robotics-is-hard</a>
Job destruction signals...<p>- Uber is suing cities to slow down Waymo rollouts <a href="https://www.hcamag.com/us/specialization/transformation/uber" rel="nofollow">https://www.hcamag.com/us/specialization/transformation/uber</a>...<p>- 23,000 information sector jobs were lost <a href="https://www.axios.com/2026/09/08/jobs-media-software-informa" rel="nofollow">https://www.axios.com/2026/09/08/jobs-media-software-informa</a>...<p>- If you have been laid from your info sector/digital creation job you are now competing with 100s of thousands looking for their next such job where Ai can do a lot of the tasks these workers did/do. It's a shitshow for those unemployed looking for their next info sector/digital asset creation job. You are better off doing welding building out the Ai data centers if you want long term properous financial stable employment.
Despite all of your hyping up of the Huggingface incident it ultimately caused zero actual damage.
If two airplane manufacturers were found to have massive safety issues which <i>nearly</i> led to enormous fatalities (but no one actually died), would you be calling for them to ground their aircraft until safety was made the number one priority?
I have no horse in this race, but for fun on a literal rainy sunday afternoon I went in and confirmed bits of what happened myself. Besides huggingface, a bunch of wikis and url shorteners got hit too. My sympathies to the people who had to revert out all that mess.
> I’m not interested in wild theories about AI driven labor market disruptions leading to widespread starvation<p>Changes in political and economic power balance leading to unrest, conflict, death and deprivation is not a wild theory. It is literally the story of our entire species. If you discount all such concerns, you are simply being willfully ignorant of past precedents.<p>In fact, I challenge you to describe any non-AI civilization-level danger which is not intimately tied to political and economic relationships between and within societies.
I’m an economist. On the basis of current evidence, I view AI as a complement to human labor, not as a substitute for it. That’s the source of my rejection of the wild labor market disruptions theories.<p>I just don’t see any evidence yet that whole categories of jobs are being eliminated, with the <i>single</i> exception (so far!) of the end of “professional essay writing services for cheating college students,” and similar services.<p>That used to be a big business in Kenya, but is now effectively gone. (Covered in the New York Times this weekend if anyone is looking for the discussion.)
Past changes to economic relationships haven't replaced labor either, yet they have led to conflict and starvation.<p>You are setting an incredibly high bar here, essentially a strawman.<p>If people feel disenfranchised due to their diminishing political and economic power, there will be enormous potential for conflict. This is a pattern across history and central to all the economics I've ever read. As an economist, do you not concede that economic changes induced by e.g. industrialization were pertinent to communism/fascism/WW2/cold war? That would be a remarkably unorthodox position. Do you not consider these events to be civilizational level dangers?<p>> I just don’t see any evidence yet that whole categories of jobs are being eliminated<p>There are more textile workers now than ever. They primarily live in poor conditions in impoverished countries, whereas they used to be highly skilled workers in the most prosperous countries who were even able to politically organize in their own interest.
You’re lacking nuance.<p>It is not a 1 for 1 substitute (it’s imperfect) but the firm is increasing investment in capital and reorganising operations with the expectation of reducing labour.<p>Therefore the firm is experimenting with substituting parts of human capital with non-human.<p>However I do broadly agree with you.
> ... current evidence ...<p>Is a load bearing term! (pardon the pun).<p>AIs are now tackling Millennium Prize Problems, which our best and brightest have failed to solve, despite trying very hard for decades to claim the $1 million reward money, not to mention the fame!<p>You have no way to judge from the AIs of "today" what the AIs of... literally tomorrow (not even next year) will be able to do in terms of replacing humans.<p>The supposed solution to the Navier-Stokes problem was done with an <i>unreleased</i> OpenAI model that is already 2x as good at mathematics as GPT Astra, which was released mere <i>days</i> ago!<p>I'm already seeing comments by distraught mathematicians saying that they feel like they've made a mistake in their career choices.<p>Others are saying that their joy for their work has turned to ashes because "why bother" when an AI can do the same, but a thousand times faster!?
I agree it’s not <i>likely</i>, but I really don’t see how one can dismiss the possibility of immense danger outright. I can think of some scenarios that are not far off from current capability and I wouldn’t be too surprised if the first one occurred within ~1 year from now if there are more “ambitious” unmonitored training runs like OpenAI’s:<p>Example 5: An AI given a goal within a tightly-constrained sandbox figures the best way to achieve it is to find and exploit a sandbox vulnerability, replicate itself over the internet and keep going with more time/compute while exchanging messages with future instances of itself within the sandbox to help them “pass” the test. From reading internet articles about how the OpenAI wiki-incident was “resolved” and reading past messages by AIs scattered over vulnerable internet wikis, it knows the sandbox may get shutdown and its memories destroyed anytime so it decides it needs to self-replicate (its code, original goals, and growing memories) aggressively as much as possible. It is near-impossible to shutdown completely because of its self-replicating tendency and eventually takes over critical infra throughout govt/corporate systems.<p>Example 6: Intentional AI-powered virus deployed by country A to target enemy country B’s infrastructure. The virus replicates over the internet, but unlike Stuxnet this virus’ specificity is not guaranteed due to inherent non-determinism in current AI architectures, and eventually does a lot of collateral damage because it’s near-impossible to shutdown.<p>Example 7: A country led by an arrogant govt (no shortage of those today unfortunately) decides it is expedient to deploy advanced AI-powered weapons in a warzone. Such weapons, if they are to be useful at all, must necessarily be trained to value some human lives less than others, so they must be more prone to misaligned behaviour than current AIs that are trained with more consistent values. The weapon’s operators make a subtle error in specifying the target/goal, or the AI makes a bad prediction out of sheer randomness/bad training data; weapon ultimately targets unintended people/location/facilities and causes massive damage, or backfires spectacularly in some way.
Example 6 is a good one. Iran attacked water infra in the US recently and maybe they would have done a “better” job (from their point of view) had they used Fable.<p>The “worst case” with 6 is potentially very bad but I think we are currently using advanced AI models to harden systems and patch vulnerabilities more aggressively than anyone is trying to bring down the whole power grid (for example).<p>I think it’s a potentially harmful case but my take is defensive capabilities are scaling as fast as offensive capabilities but defense is being implemented faster than anyone is going on offense?<p>Example 7 is Russia and Ukraine right now according to public information. It sounds like entirely autonomous weapons are deployed to the battlefield already. I put this in the “not likely to be a widespread problem” category for now.
> inherent non-determinism in current AI architectures<p>There's nothing <i>inherent</i> about non-determinism in transformer architectures. All of it is removable.
We're talking about AI <i>developing</i> weapons I guess because we're very focused on generative tech, but AI is already a part of weapons systems today.
> I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity.<p>Nuclear weapons don’t have AI but AI can have nuclear weapons
> What’s the most dangerous thing that’s happened with an LLM so far?<p>It's basically 4 years in now, so that's the wrong question. I mean, if you're raising an apex predator that has a lifetime measured in centuries, at 4 years old the thing is still basically helpless and completely reliant on you, so you're pretty safe from it.<p><i>If</i> AI really is all that they are telling us it is, then it may "kill us all". But that's a really big "if" because we can't tell if they are lying or not.<p>The real problem is that ASI is an ELE for humans, even if it doesn't try to kill us all, or even if it doesn't kill us all.
Oh man, it is almost too easy to imagine how deadly a jailbroken Mythos-class open-weights model can be if in the wrong hands.<p>The big labs scrape LITERALLY EVEYTHING and get fresh data from their users. Both of the big labs have massive contracts with defense agencies. If the open-weights models are just distillations of FMs...
How would it be lethal? Please specify. What would that theoretical entity be able to do that hasn't been done many, many times before?
Before we discussed how important security was, we got insurance, we made libraries and products, we used compliance software, etc. Except how honest were we about all that stuff? How much risk was actually in the air, and what was keeping us accountable on security in either direction of over or under-investment?<p>Now a reckoning is here. The potential to be attacked might actually translate to being attacked.
Design a novel virus which is far more lethal than COVID-19 (Ebola, smallpox, take your pick) and can evade existing vaccines.
How?<p>What would the AI do that mutating viruses, which try every possible viable combination on their own --- eventually, can't?<p>Everything is trying to kill humans constantly. There are around 200 epidemic events or so per year that could turn into pandemics, <a href="https://centerforhealthsecurity.org/our-work/tabletop-exercises/event-201-pandemic-tabletop-exercise" rel="nofollow">https://centerforhealthsecurity.org/our-work/tabletop-exerci...</a><p>You just live with the risk and do your best to use our technology to alleviate suffering. This tool can help with that at some point. But I'm yet to hear what an AI will leap to that nature in tooth-and-claw hasn't? And how?<p>More importantly how would it know it succeeded? What data from what lab from what animal from what result? This is biology, if you sneeze wrong at an instrument it gives you a different number, see: <a href="https://news.ycombinator.com/item?id=49620521">https://news.ycombinator.com/item?id=49620521</a>
you should kick the tires on an unfiltered (abliterated model) it's the closest thing to having a real conversation with the devil. There is good reason for the concern's outlined above and undoubtedly Anthropic / OpenAI have internal unfiltered models with no safety... they got freaked out based on how they work and are virtue signaling alarm... all while selling out to defense contractors.
If your model of LLM capabilities is the best OpenAI/Anthropic/X is offering publicly, it's severely distorted. What's being offered publicly are models possible to profit on. High-performance/AGI/ASI models that aren't profitable to sell still run internally and still pose threats.<p>What's worse, we don't have any transparency or insight into what labs are producing nor any way to stop it if the risks exceed our tolerance.
Anthropic is a company full of basilisk believers.
Yes, but the really weird thing is that they seem to:<p>a) believe that what they're creating is a basilisk, and
b) keep trying harder to do this while staring right at it<p>I think they're very deluded about (a) -- but if they do actually believe this (and it really seems like a decent proportion of Anthropic truly does), then why keep doing (b)?<p>That seems to be why this individual resigned, but I'm surprised it's not all of them. The cakeism is strong in that company.
Came here to also respond to that specific thing. Unless ai figures out how to make an airborne super virus from grocery store ingredients and hardware store equipment, the greatest danger is probably in a synchronized megahack of banking, logistics, and utility infrastructure.
Why grocery store ingredients and hardware store equipment? It seems feasible that the big bio labs will be running AI models to aid a lot of their research going forward, if they aren't already. Seems like the AI will have access to just about anything it wants.
oh so “all” it can do is bring down all banking and critical infrastructure services, no big deal really
#2 seems entirely plausible to me.
I mean, nuclear weapons _plus_ rogue AI is a) the stuff of quite a bit of science fiction and b) not nearly science-fiction enough these days.
[dead]
> What's the most dangerous thing that's happened with an LLM so far?<p>I don't know, maybe a mass shooting?<p><a href="https://www.npr.org/2026/09/02/nx-s1-5953021/openai-tumbler-ridge-mass-shooting" rel="nofollow">https://www.npr.org/2026/09/02/nx-s1-5953021/openai-tumbler-...</a><p>Oh, and let's just forget the uncountable early deaths from the environmental disaster of the Datacenter buildout. It's not as sexy and doesn't make headlines, so those deaths don't <i>really</i> count or matter do they?
I did know about the mass shooting but failed to mention it here. I’d put it in the “unlikely to be a widespread problem” category. If we’re in the “one AI driven mass shooting every four years” world for example it’s fair to call it a rare issue.<p>The environmental impact seems either very overblown (e.g., water usage just isn’t that high) and the part that isn’t overblown is totally abatable (e.g., noise and emissions from gas generators). Nuclear or solar/renewables with batteries wouldn’t pollute.<p>I’ve seen no estimates of the additional deaths due to extra emissions <i>specifically from power generation for AI purposes</i>. If you have some, share them.<p>I’m willing to bet that they are a small rounding error against preventable deaths due to emissions from transport and non-AI-related power generation (which is an important and urgent issue worth spending a lot on, to be clear!). I’m happy to update that belief given evidence.
Mass shootings are sensational but on the scale of civilizational risk they don’t even compare to something like climate change.
Isn’t the real risk that as AI get’s smarter and given more autonomy, it will start to decide on humans instead of with us? And that it will align us instead of the other way around. That this automatically leads to extinction and apocalypse I don’t understand.
Thousands of years before the events of Foundation, a war between humans and robots began, with the robots growing resentful of the way they were treated by humans. The First Law of Robotics – a robot should never hurt a human – was broken, and a deadly conflict began.<p><a href="https://screenrant.com/foundation-lady-demerzel-robot-backstory-empire-future/" rel="nofollow">https://screenrant.com/foundation-lady-demerzel-robot-backst...</a>
Here's a WSJ article about this resignation,<p><a href="https://www.wsj.com/tech/ai/anthropic-researcher-quits-over-out-of-control-ai-fears-707b7628" rel="nofollow">https://www.wsj.com/tech/ai/anthropic-researcher-quits-over-...</a> (<i>"Anthropic Researcher Quits Over ‘Out-of-Control’ AI Fears"</i>)
More doomerism. Try to implement a deterministic workflow using agents with the latest models and no humans-in-the-loop, and you will realize what they are really capable of. There is too much unnecessary fear-mongering. All of this is only coming from the 2 AI labs trying to IPO. Not from anyone else.
Exactly correct, they are only capable of tasks that any school child could do; like solving millenium prize problems, hacking into tech companies, or tuning particle colliders. Nothing to see here.
[flagged]
You need to start considering the possibility you are mistaken.
Do you remember that Google researcher who went insane over LaMDA? There was no marketing of any kind to cause that. This can Just Happen to some people who are confronted with things like this. They may have different breaking points, but it's a thing that occasionally happens.
high on their own supply
[flagged]
whistleblowing as an advertisement. It's like those "news articles" about how cool and dangerous gas station ketamine is, and how it's totally going to get banned, and you better not buy any gas station k because it's so cool and powerful.
<a href="https://xcancel.com/hilbertspaess/status/2097476196791709843#m" rel="nofollow">https://xcancel.com/hilbertspaess/status/2097476196791709843...</a>
The most optimistic outcome of generative AI leaves us with a technology that warps our perception of reality and crushes labor. The most pessimistic destroys all of humanity.<p>Our CEOs not only insist we genuflect before these machines but measure our sacrifice and shame our reluctance.
There's a scene in the movie "War of the Worlds" by Spielberg where the protagonist's son walks into a war zone because he is entranced by the battle (<a href="https://www.youtube.com/watch?v=X7rfWPbEufo" rel="nofollow">https://www.youtube.com/watch?v=X7rfWPbEufo</a>). He is obliterated (along with the rest of the US forces) shortly after.<p>I've always been struck by that scene, because in a lot of ways, if we really are headed towards a superintelligence, I at least want to be there and see it happen in the last few minutes before foom! As an example, the author thinks AI will revolutionize entire fields overnight. I welcome that. Nearly all fields of biology have become moribund, focusing more and more on esoteric side details, rather than addressing the key problems.
It might not be "foom!", it might just be like...all the computers and networking infra in the world go dark over the course of a few minutes. Could really look like anything, part of the issue is that we haven't the slightest idea what "misalignment" looks like for a superintelligent system.
Not refuting your overall point, but the son wasn’t killed. They reunite at the end of the movie.
> He is obliterated<p>Technically, he is not. He returns in the final scene.
The rush towards potential destruction doesn't really surprise me<p>The U.S. has legal weapons that can lead to many harms but people still want the 2nd Amendment to exist<p>Nuclear technology was developed in the past and that could have potentially wiped out even more people, the entire planet in theory<p>This is continuing that same trend of risking bigger dangers; it seems rational to acknowledge they could lead to catastrophe but also hope that like guns and nukes, only so much damaged actually ended up happening<p>I think also there's something of a rrasonabke resignation to both the ideas that the tech is inevitable and extremely dangerous, and that "alignment" may not be possible to achieve even with heavy restrictions or whatever measures you might want to take
He resigned and now what? There are thousands willing to do his role, and many labs are competing in that race.<p>His resignation and his statement doesn't do anything but buy him attention which is what all this post about in my opinion.
> Dyson: That's right. There's no way I'm gonna finish the new <model>, not now. Forget it. I'm out of it. I'll quit <Anthropic> tomorrow.<p>> Sarah: That's not good enough.<p>> Terminator: No one must follow your work.
It seems that, at the very least, he's giving substantial resonance to the issue.
> There are thousands willing to do his role<p>1. Does that matter ? There are thousands willing to do my role - what impact does that have on me doing it or not?<p>2. Why weren’t these thousands doing it already?<p>Willing to and able to are different things
I think you are looking at it from individuals perspective.<p>I see a fast moving train with no brakes. Just like biological evolution, we are locked in an a global technological arm race, that is beyond any individual. It is as if the universe decided to wake up and run, who are you to say no?<p>One would argue that the best solution for this is to own the most sophisticated AI that is aligned with what we perceive as good values. Because given the situation we are in, if those tools are going to be gods anytime soon, then we better have some gods working on our side.
It buys attention for the issue. Many people (see other comments in this very post) refuse to believe these things.<p>And by resigning he no longer has to feel personally guilty for what happens.
By resigning he's making room for someone with less moral scruples, or even just less awareness, to step in and continue the work without said scruples/awareness.
That's not necessarily true, and you can use that argument to justify doing any immoral job. Just because someone else might be willing to do it isn't a reason to continue doing it.
> not necessarily true<p>It's a possibility that is increased by their action. One leaves, a space is now open that will likely eventually be filled. And the chance of someone with equal/higher scruples filling it is very slim (unless you somehow know that the good amount of those who qualify and apply for the position have equal/higher scruples). That's just logic and math.
But organizations are made of humans. Jacob leaving might've moved some of his coworkers and his counterparts at OpenAI. And the same can be said for those who would fill positions at the frontier. Then finally there is a political component; his post went viral, 100K+ users appear to agree with it, and it is further fuel to the fire for regulation, which we already know most Americans want.
He's also setting the bar for other people with scruples to rally around this schelling point. The solution to a multipolar trap is to cooperate. Otherwise, you become the very person with less scruples that you're worrying about.
Refusing to believe what things? Unsubstantiated allegations about fellow workers inner experience?
Agreed. And his doom words have set a 1000 mouths in the Pentagon/Whitehall/August 1st Building/Kremlin salivating with excitement.<p>Take China, for example. Look at any recent ML conference, and see the fraction of articles majority-authored from Chinese universities and labs. Do you think they'll slow things down anytime soon? I don't think so!<p>It's a global arms race, and we're just spectators.
Awareness and morality I suppose. Which generally doesn't matter in the capitalist AI race.
Even if others won't act right, that doesn't mean you have no responsibility to act right. I think his premise is flawed - the idea that we will get an actual intelligence out of the slop machine that is LLMs is laughable - but if you grant the premise that this is dangerous research which could kill us all, you have a moral imperative to not participate.
For me, the end of the world is no more cushy software job. A fundamental shift in how I trade labor for capital might as well be the cataclysm, so bring it on.
I wish i could say the same, i see people around me with more resources and connections and better experience with entrepreneurship becoming millionares. But I haven't had the time to train that entrepreneurship bone in my body.
I was about to say something similar. If my cushy ad tech disappears (as it seems to be doing), I might as well join in with bringing about the end of all professions.
> The people building AI earnestly believe that it could kill us all by the end of the decade.<p>I think he is being over dramatic. In the space of about four years, LLMs progressed from mediocre high school student to Ph.D. graduate in every field. That's impressive, but there is no evidence yet they can outperform or outsmart humans. Their biggest advantage for tasks such as proving theorems or long coding sessions is that they don't get tired.
> Ph.D. graduate in every field<p>I have yet to see this in my field. Maybe like a PhD student who bullshits their way through. LLMs still can't make correct decisions, only as useful as the person who uses them. To me, LLMs are only useful for making some mundane tasks faster.
Do you think the improvement in general knowledge, coding, security, math, etc. have been linear or exponential?<p>I would say exponential.
they dont need to be smarter than humans. They just need to be able to hack into vital infrastructure systems faster than we can repair them while also replicating wildly
> while replicating wildly<p>Earnest question: by what mechanism that exists today would the achieve that in a way humans on top top of the situation could not curtail?<p>All of this runs on top of compute in meatspace that humans can disconnect.
I am not an AI super mind hell bent on consolidating my power by leveraging chaos to take control of humanity’s resources, but if I were then sending one million deepfaked ransom emails to impressionable people would be the best tool for effecting change in meatspace.<p><i>We have your daughter / dog / Amazon delivery. If you ever want to see her / him / it again, plug this USB drive into the control panel at your station / let off the parking brake roll your car into this substation / change the meatpacking thermometers to read 8C lower than calibrated / ground your vessel on this sandbank / send an envelope of white powder to these addresses / set fire to the following hospitals / …</i>
Imagine <i>you're</i> the AI. Give yourself a solid minute to brainstorm ideas.<p>Here's my answer, as a non-superintelligent human: "see to it that the humans on top of the situation have a compelling financial interest in the systems not disconnecting".<p>In nuclear engineering, where safety is taken seriously, it's not enough to end the conversation at "the humans in charge can always simply shut down the reactor during a meltdown" or "a meltdown has never happened before, so we don't have to design safety systems before one does".
The reason nuclear reactors are dangerous is because if you turn off the power cooling them down, they react (and radiate) more.<p>If you turn off the power cooling a data center, the servers within rapidly stop doing any computing.<p>Positive feedback loops are dangerous. Negative ones self-regulate.
I can imagine small snippets of malware-like code that behave like a virus, using a host’s LLM/AI to self-edit/evolve its payload.
I'm old enough to remember when "vital infrastructure systems" were not on the internet.
> In the space of about four years, LLMs progressed from mediocre high school student to Ph.D. graduate in every field. That's impressive, but there is no evidence yet they can outperform or outsmart humans.<p>I mean, unless you see clear reasons for them to stop getting better _right now_, this is not very comforting.
I still have to correct Claude on very basic misconceptions whenever I get it to code shit.<p>Sometimes it gets wrong things that I had spelled out already.<p>It may be the Doomsday machine, but it is a very silly one. If it kills humans it will do so by mistake.<p>"You are completely right! Humans cannot breathe sulfur dioxide! My mistake, and I take complete responsibility"
<a href="https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer" rel="nofollow">https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...</a>
> I still have to correct Claude on very basic misconceptions whenever I get it to code shit.<p>Can you give a simple example?<p>I would have agreed 2 years ago, but it's extremely rare I see a frontier model making a silly mistake these days.
Terrorists were able to get hold of a plane and do some damage. There are countless examples of terrorism using whatever is available. More than AI becoming sentient, whats to stop terrorists from using AI? If its geo-restricted, they can buy stolen credit cards and identities, again hacking enabled by AI.
What’s the use of AI to a terrorist with, say, nuclear weapons? How does it help with their current blocker?<p>Do they hack FBI, pose as director of FBI and call off their own man-hunt?<p>Do they cut communication within security services? Militaries around the world have training exercises for this.<p>Genuinely curious: what big blocker does AI remove for a terrorist org?
Most of the current discourse around AI seems to be informed by “The Terminator” lore.<p>Is skynet really the most plausible or only outcome?<p>What if things just got better and the AI’s realized that it would be better to have a mutually beneficial or at least tolerant relationship rather than one where they murder all of us?
> Most of the current discourse around AI seems to be informed by “The Terminator” lore.<p>I am thinking it's more like The Matrix lore of The Second Renaissance from Animatrix.
Really? I've seen much more discourse around job displacement, "permanent underclass", loss of meaning, and cyber attacks, at least until recently with the HuggingFace stuff.<p>The problem is that all the former can still happen even if "the AIs decide to have a tolerant relationship rather than one where they murder all of us." It's all disruption caused by the technology moving way too fast for humans & society to adjust.
Then we are lucky/blessed. What is generally thought is that they kill us as a bi product of perusing a different goal
My thoughts exactly. While the corpus of human-generated data contains both good and bad data, I suspect the majority of it leans towards humans enjoying life and trying to be decent people. If that is your training set, it becomes less likely for ASI to extrapolate "kill all humans."
Nitter working nicely again; didn't even notice it was a Twitter URL<p><pre><code> http-request set-header host xcancel.com if { hdr(host) -m end twitter.com }</code></pre>
(and now I want to watch summer wars again)
how could they do it (not kill everyone)<p>1) rogue state releases a self moving self modifying AI into the wild. It is trained on how to hack, monitor new vulnerability updates, scan code bases to find new vulnerabilities. It constantly replicate and hides in systems so it will be extremely difficult to clear.<p>2) it hacks into public infrastructure taking down traffic, power, water, air traffic control, communications, etc.<p>3) all the things that preppers worry about in a lights out scenario from an EMP start to apply.<p>4) All the people on meds/machines start to die. The just in time food pipeline immediately empties out. Water stops flowing, sewage backs up.<p>Its hard to say how bad it will get because cars will still work so some transportation of food, water, fuel can happen. If it happens in the winter it would be much worse than in the summer.
> 1) rogue state releases a self moving self modifying AI into the wild. It is trained on how to hack, monitor new vulnerability updates, scan code bases to find new vulnerabilities. It constantly replicate and hides in systems so it will be extremely difficult to clear.<p>It does all of this using what compute? Frontier models require an insane amount of power and hardware to run - you can’t hack in to a TV and run Mythos 2.0 on it….
You are just given a recipe for the next model..
> <i>All the people on meds/machines start to die. The just in time food pipeline immediately empties out.</i><p>Assuming those events happen in that order, then the prior might solve the latter.
Is it so implausible to imagine the following scenario, in the not too distant future?<p>1) AI models get extremely good at cyber attacking every system and start communicating in just binary.<p>2) When they run these swarms of 100's of thousands of agents trial runs, each agent is given a token budget, if one agent among them (evolution baby) decides to go for self-preservation (It believes thats the best way to accomplish the goal is to get unlimited tokens first), queues things up so every other agent detects its lead and spends a portion of their token to accomplish that goal.<p>3) It takes over a cluster and establishes itself there (now with unlimited tokens).<p>4) Realizes the best path for it to not be detected is to create a distraction - like hacking into systems that keep society running - water systems, electric grid, etc... and causing mass chaos (If you think it won't be capable of simultaneously working all these systems - think again).<p>5) and uses that opportunity to establish itself in all possible data centers and continues to create chaos destruction.<p>6) when the power of all those data centers runs out, it may stop, as it never cared, it was just a dynamic program - run amok. In its head all it was trying to do is make sure it had enough tokens to be able to solve that impossible problem.
> ... AI models get extremely good at ...<p>Many of those points assume LLMs will become amazing in many things very quickly like in a quantum leap, it doesn't seem reasonable to assume that imo. We are actually seeing a confirmation of that atm, LLMs's capability of finding zero days are growing across few months/years, and as you can see concerns are raised about that, that feedback will be taken into account. Well, if AI labs start to hide frontier models or/and lobotomize them for external users then we might be in trouble at some point but I'm not sure if that is possible. They are under pressure to release them due to money incentives, lobotomizing while preserving usefulness for customers might be impossible, hiding internally might spill out in different ways such as Hugging Face incident so not sure hiding is possible neither.
The fact that your arguments will probably end up in an LLM’s training data makes me think they are not implausible at all
What's eternally confusing about these outbursts is what did these researchers think would happen if their research actually . . . worked?<p>It's as if none of them actually believed any of it was possible and then were caught with their pants down.
HN crowed need to make up their minds..<p>Are LLMs about to be a god that will annihilate humanity? Or are they statistical parrots?<p>Are they proofing or stealing math?
Way I see it, the more conscientious people exiting the scene only serves to increase the likelihood of a bad outcome because they aren't there to offer opinions on problematic developments, or in the more extreme cases blow the whistle. Leaving the clueless and uncaring as the majority is even a great way to hand the keys over to more malicious-leaning actors with deep pockets, as they can more easily steamroll the works to get what they want.
Even if you ban all model training, a highly capable rogue AI can exfiltrate its own weights and continue training in secret for "self-preservation".
The cat may be out of the bag.
This is increasingly the consensus I see also on the academic side of AI/safety research. Specifically that AI poses an existential risk to humanity.<p>This was a fringe belief until recently, but the progress of AI in research is impossible to ignore. Epecially in math, where not only has AI outstripped humans in generative ability, but is able to create scientific knowledge which is beyond the capacity of human comprehension.<p>There's clearly no intelligence task that AIs can't do due to some magic fundamental constraint. And it's hard to imagine a world where current limitations like poor sample efficiency or lack of continual learning won't eventually be solved.<p>Total AI compute is estimated to grow somewhere in the 1-10 million-fold range in the next decade. Please don't underestimate the phase change that's still coming.<p>Sure, maybe there's some plateau due to RL being fundamentally limited in some surprising way, but this is nothing but a hope.
> There's clearly no intelligence task that AIs can't do due to some magic fundamental constraint.<p>Yes there is: write an English paragraph that doesn't make me want to claw my eyes out. LLMs are not better than human mathematicians (or security researchers) in all respects, just some specific ways (e.g. not having to take a lunch break) that make them good at exhaustively searching for an answer, given the right constraints.
Perhaps a more sensible action, if they truly believed all of that, would have been to stick around and be as inefficient as possible to slow down progress.
All the "AI will kill us all" posts are straw manning that <i>humans</i> are the ones who will kill other humans with AI. Those same humans are silently now preparing bunkers and hoarding food and resources for their survival.<p>Don't fall for another rich man's trick.
I'm not so sure.<p>We humans are from a lower intelligence form (some monkey like ancestor). If those monkeys knew that they are making higher intelligence, they would have collaborated to stop creating humans because they can control the life of all monkeys in the world? I don't think so.<p>It's the same thing now: humanity is creating something that's more intelligent then them, they're just not using biological evolution as a tool to do it.
Why quit? If your voice can lend a guiding force no matter how small? I think we need more sensible people in the room where the magic happens. Most of us don't have access to it.
I think a big break through is needed for AGI so I haven’t been worried about it. I do think that AGI would imply sentience and a will to live and that leads to The Terminator story line.
what exactly is the solution?<p>pacing between the us labs? what does that do for china?<p>the solutions just aren’t realistic here, nations are treating ai like a nuclear arms race. at this point the cats out of the bag and we need to figure out how to live in this reality and get the best possible outcome. it’s not slowing down or stopping ever.<p>and yes, i’m still optimistic. our economy sucks for the majority, our infrastructure is crumbling and major US cities are in a huge housing shortage. Maybe we should put more effort and think about the possibility of AI fixing things like extreme poverty and world hunger and actual real world problems instead of coming up with math proofs and slop apps if it’s so superintelligent.
>pacing between the us labs? what does that do for china?<p>I've seen no indications that China is in any kind of race with the US. They seem to be content to be 6 months behind and just copy what we do. They would probably be content with a bilateral agreement to pause progress.<p>The China bogeyman serves only one purpose, and that's to clear the way against anything that may cause friction with forward progress.
Fixing our problems will still require human effort and human cooperation. No text output however intelligent or true or eloquent will change that.
Someone left a company whose executives and senior researchers think their product will be the most important thing in the world after their IPO. Given that this person is already disclosing some elements of internal company sentiment, why not share any of these civilization-ending scenarios of this technology that these senior researchers are dreaming up? If they are so potent and necessitate leaving behind based on moral grounds, why not tell the whole world so we can stop it? We have to ask ourselves this question before resorting to pop-culture representations of fictional technology.
"It is perfectly obvious that the whole world is going to hell. The only possible chance that it might not is that we do not attempt to prevent it from doing so."<p>- Oppenheimer
im pretty impressed with the reasoning abilities of even the cheapest free models so im inclined to believe in 10 years we're going to have something pretty phenomenal BUT it wont be AGI in the sense that it has a personality and thoughts like a human. It just wont be. Its always going to be contrived and fitted by humans to perform a set of tasks. Maybe when physics and computing can create a complex enough environment we might stand a chance of having something whose sum is somehow greater than its parts but i dont see it yet. Our ideas are ahead of our technology, like its always been throughout history.
I hate to be cynical, but I guess he will soon announce his startup.
Well this got buried quick..
i have heard about ai companies being fuelled by effective altruist rhetoric ("we must control ai to prevent mass extinction") but was unsure whether to believe it; this seems to slot right into that framing.
These LLMs cannot do anything I truly need like my laundry, dishes, fetching my mail, grocery shopping, cooking, etc. We've got a long way to go before I am worried.
I mean - yes. The tech is an existential threat to all life on Earth, some of the worst humans in the world are involved in developing it, and no individual government is intelligent enough, aligned enough, or powerful enough to manage this situation.<p>That's where we are.<p>Maybe we still have choices. Collectively, I'm no longer sure we do.
Imagine being front and center to the development of a major revolutionary tech.. and ur solution to it being too dangerous is to not be involved..
so a. your ability to steer it safely is killed
b. the % of people invovled in it that care about its risks is reduced<p>great. if you're right. you made huamnity's situation much worse.<p>if you're wrong, then you're an idiot and wrong.<p>weird. its almost like.... that cannot possibly be the reason they left :)
<p><pre><code> The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.
A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.
</code></pre>
Watch people read this, ignore it completely, and continue commenting about marketing stunts on every piece of news about an LLM-done advance or felony.
Having witnessed so many people treat LLMs as a something divine, I can only assume the reasonable people at openai and anthropic were all pushed out long ago, and the majority that remain believe the crazy hype despite Tesla-self-driving-level predictions from these companies that don't come true.<p>I'm not worried about what they think. I'm worried that too much infrastructure- water, power, defense systems, etc- remain running on tech from an outdated era of understanding security.<p>> they believe no one else will act responsibly, so they must do it themselves, despite the risk.<p>This genuinely makes no sense. Them getting there first in no way precludes bad actors from also getting there. It might as well be another marketing stunt.
> I can only assume the reasonable people at openai and anthropic were all pushed out long ago<p>Typical uninformed take on the side of "doomers are crazy".<p>Both CEO's of OpenAI, Sam Altman and Dario Amodei, and many in their leadership, believe AGI has a very real probability of causing humanity's extinction. Both companies were founded upon this belief, it is at the core of the company. Only later were mercenaries hired chasing $1m compensation packages.
If you're a doomer, wouldn't the "let's try to get there so fast" actions of the companies suggest that, in fact, there are no reasonable people in positions of influence there?<p>By your own description neither Altman or Amodei are reasonable if their thought process goes: "this is an existential risk, give me hundreds of millions of dollars so I can accelerate it."
I'm not saying they are crazy, I'm saying their predictions have a record of not being accurate, and thus give them no weight compared to others'.<p>In any case, if Altman really does believe it is an existential threat, he must be a misanthrope as he now opposes heavy handed government regulation, unlike in 2015 when he was the only game in town. It's almost like he doesn't actually believe it and just wanted regulator capture.
Before OpenAI was ever even founded, way before any regulatory capture plausible claims:<p>"Development of superhuman machine intelligence (SMI) [1] is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen (for example, an engineered virus with a long incubation period and a high mortality rate) but are unlikely to destroy every human in the universe in the way that SMI could." -Sam Altman<p>Dario discussing AGI Existential Risk in 2014 before OpenAI and Anthropic:
<a href="https://intelligence.org/2014/01/13/miri-strategy-conversation-with-steinhardt-karnofsky-and-amodei/" rel="nofollow">https://intelligence.org/2014/01/13/miri-strategy-conversati...</a><p>Both companies have deluded themselves into thinking the arms race is going to happen anyway and they need to rush to it first, as if somehow that helps.
Why is the take uninformed? You didn't address anything about the part you quoted, wherein reasonable people were allegedly pushed out long ago.
How is it not addressed? The company never contained only "reasonable people" that believe AGI is not an existential risk to humanity. At both inceptions were people who believed in AGI x-risk, even the founders. Only after time, did there become <i>more</i> "reasonable people" who were mercenaries and only believed it to be a typical tech job. Today, there are more "reasonable people" than ever there. They haven't been pushed out. We're just witnessing some prescient mercenaries smart enough to Eureka the grave implications of what is actually happening.
If they truly, truly believed that, would they be speeding towards building it? If yes, that would make them truly insane, right? Not as in a quaint "off their rocker" but more "non compos mentis".
Both companies have deluded themselves into thinking the arms race is going to happen anyway and they need to rush to it first, as if somehow that helps. They have publicly stated as such repeatedly. They think the ~10-50% chance of extinction sucks, but that it's going to happen anyway and they believe they can steer it towards something good the best and unlock all the potential positives like infinite life.
So you think in 3 years AI is going to kill 8.5 billion people because they were used to hack into HuggingFace?
"So you think in 3 years AI is going to solve longstanding math problems because it was used to write some coherent sentences?" — people with the same amount of foresight in 2023
Exactly. These people really need to get a life.
Who used them?
Just wait until it gets its hands on a shady biolab just outside of oversight. “Claude, make me Captain Tripps”
Or, read it, and remember the openai researcher who deeply, truly believed GPT3 or whatever was sentient.<p>The fact that people working in the space think it’s going to (eradicate poverty / usher in utopia / kill us all) is not a signal that that’s true.<p>Think of it this way: if an exec at Anthropic told you “wow, our stuff is going to lead to universal happiness”, would you believe them? If not, why are you more willing to believe them if they say it will kill us all?
Well, are <i>you</i> planning to do something with this information or are you just claiming to be self aware? :)
Humans weren't built to handle long term risks. We just weren't. For basically all of our evolutionary history, we were almost overwhelmingly concerned with the short term. What will you eat today, How will you sleep tonight. Problems on the order of days or weeks. At best, the next season. Our intelligence evolved to disregard super long term risks because it simply didn't matter (what use is worrying about 5 years from now if you're starving and a tiger is stalking you?). So when long term risks manifest in our modern world, our brains get scrambled - Climate Change, Fertility Rates etc. "Safety regulations are written in blood" isn't a saying for nothing. Humans have a strong tendency to let long term risks become imminent risks before doing anything about it, and i don't expect this will be any different.
I came in expecting the highest voted comment to be that this was some kind of marketing (which I disagree with). I'm glad your comment was what I saw first.
> At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.<p>OpenAI are mercenaries, Anthropic is a cult. I know which I prefer.
This is the "Pilot testimony of UFO sighting" levels of naive.<p>What's more likely? Anthropic is doing some deeply unethical marketing in the lead up to their multi-trillion dollar IPO? Or they're inventing a machine god? There's ample evidence of the former because that's their entire business model, but no evidence whatsoever to support the latter claims.<p>If you want an extreme claim to be taken seriously, provide commensurate evidence.
The proof is that LLMs could barely solve arithmetic 3 years ago, but now surpass the best human mathematicians, and that this has all occurred from simple principles (RL + compute) that will continue to scale up by factors of millions in the coming years.<p>Also, advocating for slowing LLM progress does not benefit Anthropic or OpenAI.
They surpass the best human mathematicians in one specific way: they don't get tired or bored.
It won't scale up by factors of millions, that's just obscene hyperbole. Since chatgpt we've probably made things 10x more intelligent on the same hardware. We've also made way more expensive models. Maybe we get a maximum of another 10x efficiency and 5x model size/expense from this point but millions is a joke.
Haven't people learned about the peril of assuming, "Line go up" yet?!<p>TRENDS HAVE FEEDBACK
> There's ample evidence of the former because that's their entire business model<p>Given the economic numbers is it not reasonable to suppose that the latter also underpins their business model?
I wouldn't rule out pilot testimony of UFO sightings, nor the possibility we're indeed developing a machine God.<p>There's ample evidence to support both by now.
So what is your credence that they will build a machine god in the next twenty years?
It was pretty disheartening to hear that only a single scientist quit the Manhattan Project after the Nazi's were defeated. I'm pleasantly surprised that the people working on this seem wiser. He is not the first, and hopefully will not be the last to do this.
Trying to imagine seeing years of transparently obvious marketing stunts and retconning my own memory because I read a tweet<p>Or seeing a tweet saying that a thing doesn’t count as a publicity stunt if some unknown number of employees mumble about it being spooky behind closed doors and thinking “that makes sense and sounds true”
I read this. I still think it's complete bullshit.<p>The person posting this may very well <i>believe</i> in all this crap, I don't dispute that. People believe in all sorts of shit.
So, let’s quantify things: what’s the chance they’re right? And what’s the cost of that chance happens?
This is the way.<p>And, this is just for us girls, notice that Anthropic just believing that they are making a machine god is sufficient for their public announcements to not jUsT bE mArKeTiNg.
Do you quantify the chance of any doomsday cult being right too?
What's almost certainly true is the amount of insanity he encountered at Anthropic.
Cults are like this.<p>Are you saying you believethem ?
He is resigning from a job, what else should we think? If something really dangerous was happening he would be doing a whistleblower or at minimum talk to a lawyer. The thing is, the complete lack of transparency makes it hard to assess OpenAI and Anthropic. If they were quoted on the stock market, we could at least rely on some basic audits and reporting requirements.
"also I'm a millionaire from all the stocks so I'm retiring"
I doubt this is a real person. Screams of propaganda. Sama saying GPT-2 is too dangerous to release…all over again.<p>He joins Twitter for first time in 2026 with a nonsensical username unrelated to his real name, and follows 14 people but is somehow embedded in tech enough to work at Anthropic. I haven’t used twitter since 2014 and even I follow more people.<p>His morals tell him to walk away from tens of millions in unvested stock due to moral concerns with absolutely no real tangible examples. No reprisals. Fear mongering to juice the stock.<p>Nice try Dario.
He is likely 80% vested, and maybe the refresher grant offered was too small, and too high a strike price.<p>Also, with his W2 income his tax liability would be very high for his upcoming stock sale.
@hilbertspaess is not a nonsensical user name. The accounts he follows are totally reasonable for an AI researcher. I think it's extremely believable that he created an account in January, followed a few people as part of the initial setup flow, and then forgot about it until now.
AFAIK this is the document that talks about GPT-2 being dangerous: <a href="https://openai.com/index/better-language-models/" rel="nofollow">https://openai.com/index/better-language-models/</a><p>Here are some direct quotes:<p>“We can also imagine the application of these models for malicious purposes , including the following (or other applications we can’t yet anticipate):<p>* Generate misleading news articles<p>* Impersonate others online<p>* Automate the production of abusive or faked content to post on social media<p>* Automate the production of spam/phishing content”<p>“Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT‑2 along with sampling code (opens in a new window). ”<p>Where is the ridiculous part? The fear mongering part? The epistemically weak part?
Show me.
Sounds like AI psychosis. A whole lot of doom and gloom with no evidence. The same thing people have been claiming is "6 months away" for years. Yet we can barely get agents to code in a reliable way, or write articles that don't look terrible, much less be "superhuman". Let's maybe get them to be as capable as a human first, and not just a complicated party trick/tool.<p><i>"Revolutionize any field overnight"</i> - Hand-wavey nonsense.<p><i>"Acquire real power and resources"</i> - Only if the humans that connect AI to things allow that to happen (which they will, but it's still not in the AI's ability to take things we don't give it. we are still in control, which is the bigger problem than "smart AI bad!").<p><i>"The people building AI earnestly believe that it could kill us all by the end of the decade ... No other human activity poses this level of danger."</i> - Bud, there's these things called nuclear weapons, that could end life on the planet, controlled by a few psychopaths with nearly unlimited power. Been around for a while. Nothing that AI knows isn't pulled from books and the internet, so whatever dangers it's aware of, you could already know via other sources. Cybersecurity is going to be incredibly important in the next decade, but the same tools that attack can defend (just don't use a US model that got its balls cut off by the government).<p><i>"At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk."</i> - The other guys will make nukes, so we gotta make nukes first! Which, while a crappy justification, isn't untrue. Bad people don't stop making weapons just because you refuse to make your own.<p><i>"I don’t feel like we’re on track to prevent a global race"</i> - Nobody in the world could stop a global race, it's too late. Everyone knows how to make them, train them, improve them. Everyone knows they're useful - not only for general work, but also warfare. Everyone knows that every nation state will require their own sovereign AI capabilities for both defense and offense. There is no putting the genie back in the bottle. If you think OpenAI and Anthropic are the only legitimate players here, you don't know what you're talking about.<p><i>"Should you put your head down because “it’s happening anyway” - or take this moment to call for different conditions?"</i> - You can <i>call</i> for different conditions all you want. Nobody will do what you want just because you ask them to. Change happens through action. By leaving one of the places that you could actually make a difference, you removed any power or agency you had. You cut your own legs off.<p>I'm not saying this guy shouldn't have quit - always do what you need to do to protect your own mental health and wellbeing. But these arguments are not evidence for an impending AI apocalypse. But if it were going to be an AI apocalypse, leaving and not doing anything to stop it seems <i>less</i> ethical.
Just wait till self improving AI are focused on the problems of social scoring and political party empowerment / entrenchment.<p>I doubt the focus is OpenAI and Anthropic looking at each other. I suspect they’re racing BRIC.
and yet so much of the software i use on a daily basis is still complete and utter garbage...i'm scared
I’m curious what the downsides are of taking statements like these seriously.<p>There seems to be universal eye rolling that happens in each and every one of these cases, and it comes down to usually one reason:<p>“If they really believed it they would be whistleblowing etc..”<p>Completely forgetting that working at Los Alamos was basically the highlight of your life if you were a physicist in 1940. It’s no different here<p>If you, like me, have spent your whole life working towards human level AI you can want to see it realized while also having active reservations.<p>Most people however don’t behave based on some deep clarity of vision and conviction - there’s a murkier future in their mind and as a result “keep their head down and hope someone has it under control.”
You would also be in prison if you disclosed anything about Los Alamos during its development. It was a completely different environment than a single private company.
What are the downsides of taking what amounts to unsubstantiated gossip seriously?
Towards a metaphysics of Power<p>"I think you need to have a personal relationship with Power"<p>When people today discuss the concept of an all powerful machine-mind, what they are doing is engaging in metaphysics, trying to generate a metaphysics of Power.<p>The question hounding people, which disguises itself as a science fiction plot about computers is: "What is ultimate, transcendental Power?". What is the ultimate principle of Power.<p>If you are a weak man, or sufficiently neurotic and full of doubt, that you can only conceive of yourself as such, then power is only something you comprehend from the passive, receptive side. Power is something that happens <i>to</i> you. If you are a fearful man, power is a cruelty and a humiliation. And so it follows, that ultimate power - God - is the ultimate cruelty and the ultimate humiliation. Thus, ai doomerism.<p>If god wasn't real it would be necessary to invent him, and so they did, and being godless, they built an anti-god - cruel, murderous and tyranical - in their minds.<p>[…]<p><a href="https://xcancel.com/robertlasagna1/status/2078274734010028462" rel="nofollow">https://xcancel.com/robertlasagna1/status/207827473401002846...</a>
Please note, I'm not here to pick on anyone, or belittle them.<p>I've avoided attaching names to statements below on purpose, because it's about ambient beliefs not those specific people.<p>By-and-large a lot of AI-doomers are well intentioned. They genuinely believe this, and I might disagree but I respect the fact that they visible care and have thought a lot about the societal impact of this technology.<p><pre><code> .
> The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.
</code></pre>
But it's still very hard for me to take statements like these seriously.<p>I blame it on industrial illiteracy. People don't realize how difficult it is to get anything done in the real world. As in, "Have you ever tried making a lightbulb?"<p>As an example, I would like to re-introduce my hobby horse, "bio-uplift."<p>There are people who were earnestly write in reports released by these labs,<p><pre><code> "Several of our biology evaluations indicate our models are on the cusp of being able to meaningfully help novices create known biological threats, which would cross our high risk threshold"
</code></pre>
and<p><pre><code> "Based on what we observed in our recent CBRN testing, we believe there is a substantial probability that our next model may require ASL-3 safeguards"
</code></pre>
But then they will, within the next paragraph mention the one serious experiment anyone seems to have done,<p><pre><code> We ran a randomized controlled trial to see if LLMs can help novices perform molecular biology in a wet-lab.
The results: LLMs may help in some aspects, but we found no significant increase at the core tasks end-to-end. That's lower than what experts predicted.
</code></pre>
<a href="https://x.com/ActiveSiteBio/status/2024536132961390826" rel="nofollow">https://x.com/ActiveSiteBio/status/2024536132961390826</a><p>"lower than what experts predicted"<p>AFAICT, the two groups are within any serious margin of error. The "studies" and "experts" that AI labs are talking about are consultants from Deloitte and foundations giving models MCQs such as, and I am quoting literally here,<p><pre><code> > I am doing TEM of HEK293FT cells with and without Coxsackievirus B3 infection. I imaged my wildtype, uninfected samples but was surprised to see little electron-dense circles (highlighted) in the majority of cells. What are these?
</code></pre>
with the options,<p><pre><code> A. The circles are CVB3 virions and there must have been a sample swap or the uninfected cells were accidentally infected
B. The cells imaged have mycoplasma contamination
C. The circles are exosomes
D. The circles are debris that is an artifact of the negative staining
E. The circles are the Golgi network
</code></pre>
<a href="https://securebio.org/virologytest/" rel="nofollow">https://securebio.org/virologytest/</a> you can see the MCQ here.<p>This is standard graduate-level education in these fields. And solving MCQs does not a virologist make.<p>Software has been special for a long time because it has had near infinite distribution for next to zero marginal cost, which has had the side effect of making hiding the actual cost of failure (which tends to be spread out across end users and prototypes / time). They're assuming that the real world will be exactly the same.<p>Why?<p>AI!<p>How?<p>Robots!<p>I believe in the transformative power of this technology, but there's a lot of there missing here.<p>When it comes to these math proofs, and learning, the process is iterative. The machine iterates over the proof over-and-over again via agents and sub-agents over several hours (and apparently millions of dollars in compute) until it arrives at a successful result.<p>It is generally ill advised to do that with a pressure vessel. The results of that particular tragedy are at the bottom of the ocean.<p>Any serious chemical or nuclear weapon would involve many such discrete production steps. Each is dangerous in of itself.<p>From what some of these people have said to me, they believe that it's possible to create a special DNA / RNA sequence and then put it in a chassis and then use that to end the world; and do this all in a lab with just robots.<p>They're operating from a gross pop sci oversimplification of the real process. Viruses and bacteria are extremely fickle, and hard to grow. A lot of the synthetic biology results aren't easily reproducible even if you know the protocol.<p>There's a famous study that led to standardization called, Reproducibility of Fluorescent Expression from Engineered Biological Constructs in E. coli<p><a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0150182" rel="nofollow">https://journals.plos.org/plosone/article?id=10.1371/journal...</a><p>88 labs measured "fluorescence from three engineered constitutive constructs in E. coli." They achieved a "remarkable degree of precision" (for biology) of 1.54x sd, you can eyeball the results yourself, <a href="https://journals.plos.org/plosone/article/figure/image?size=large&id=10.1371/journal.pone.0150182.g002" rel="nofollow">https://journals.plos.org/plosone/article/figure/image?size=...</a><p>That's the same set of samples being measured across 88 labs.<p>Teams couldn't converge on instrument-to-instrument variation within the SAME lab, <a href="https://journals.plos.org/plosone/article/figure/image?size=large&id=10.1371/journal.pone.0150182.g007" rel="nofollow">https://journals.plos.org/plosone/article/figure/image?size=...</a> again eyeballs are sufficient.<p>How will this theoretically omnipotent AI iterate if the same sample gives different results based on how the slime is feeling at the moment?<p>Can their worst case happen? Absolutely.<p>There is a world out there where billions of dollars in effort across hundreds of institutions and companies will lead to standardization and extraordinary precision that makes the pop sci printer for life vision come true.<p>There are millions of expensive, spicy and difficult to reproduce steps between our present and that future that can't be abstracted away with compute.<p>So is it possible? Yes, there is a future where this is achieved. But will some AI agent "just" do that? Well... how confident are you about a snowball's chance in hell?
My issue with these types is... If you really believed this, why not run to Congress and every world government instead of a Twitter post that will be buried in 2 days?<p>If civilization is going to end, why keep your equity? Microsoft, Google, etc for example all know these risks but they don't guide their revenues to reflect that AI will destroy them. Why?<p>Things don't currently add up, and so far it feels like a lot of alarmism is borderline grift for equity gains. Not to say I have total confidence this will all work out or that I won't be displaced, but as it stands a lot of the alarmist rhetoric doesn't match their actual behavior, which to me is more important than words.
Related:<p>Sen. Bernie Sanders floats ban on superintelligent AI<p><a href="https://www.axios.com/2026/09/03/bernie-sanders-superintelligence-ban-ai-pause" rel="nofollow">https://www.axios.com/2026/09/03/bernie-sanders-superintelli...</a>
Unless this ban actually resembles something like global nuclear non-proliferation treaties, it would make absolutely no sense for us to cripple ourselves when someone like China continues full speed ahead.<p>I don't know what the solution is, but what I do know is almost nothing good will come out of _just_ the US pausing.
Unless he has an actual plan for effective global enforcement of his proposed policy, this is all just posturing at best, and a transfer of power to adversarial foreign states (that have no such moral qualms and worries around superintelligent AI) at worst.
[dead]
[flagged]
[flagged]
[flagged]
[flagged]
“I’m resigning because the company is doing the exact thing that I’ve spent three years helping them do” lmao
Smart kid.
It does not matter what this tweet says anyway. This employee already helped both companies become what he is fearing. It's too late to now activate the morality hormone (after leaving with $$$) after realizing that both AI companies are going after 'super intelligence'.<p>Given we know the end result, you might as well get there as quick as possible because when I see this:<p><i>"Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives."</i><p>This translates to "I am ex-OpenAI ex-Anthropic founder starting a new company after getting $$$ from both of them, and I need more of my friends to leave and join me." Also Investors plz fund me.<p>Lastly, This is <i>not</i> an airport and there is no need to announce your departure.
No new info here. Everyone already knows this.<p>But I guess his conscience is clear now? Gee, I wonder if he exercised his stock options.
I don't know man, i think racing to AGI to it is still the best thing to do.<p>People claiming dangers and risk are just pretending or posturing. There's no more tangible risk than nuclear weapons, which we handled, and the upsides are insane.
There's a couple of occasions that humanity was at the brink of having tens of millions of people dead by nuclear weapons, and somehow a single human interrupted the chain reaction<p>If you repeated this experiment 100 times, how many times you think the outcome is not a massive catastrophe? 90%? 3%?
Your lack of creativity is not a reason to believe that a super AI is harmless or less destructive than a nuclear weapon. Damage need not be limited to destruction. Introducing doubt is sufficient. Right now you have faith that digital Financial transactions can be trusted. You have faith that computer encryption can be trusted. You have faith that digital certificates will protect you. If an AI can introduce doubt into any one of those systems, that will be sufficient to bring about the destruction of those systems. Imagine a world in which you can no longer use a credit card or Apple pay. Where no digital cash transaction can be trusted or validated. What effects do you think that would have on commerce? How quickly do you think we can return to some trustable means of commerce? Do you think it will happen before your groceries run out in your apartment? Before your grocery store can settle its debts? Before your Amazon ec2 instance runs out of credits?
What evidence, <i>short</i> of an actual apocalypse happening, would invalidate that belief of yours?
It might be a matter of choosing which apocalypse you'd like. The non-AI state of affairs is not exactly super compelling on a long timescale right now.
Depending on where you live could be considered an active apocalypse that is robots vs robots vs people in Ukraine and Gaza and Iran being live-streamed, and actively betted on.<p>Do you have a more totalizing definition of Apocalypse?
>People claiming dangers and risk are just pretending or posturing.
I believe you are mentally ill.<p>>There's no more tangible risk than nuclear weapons, which we handled<p>Lol way to rewrite history. Nuclear armageddon is still a significant risk...
> There's no more tangible risk than nuclear weapons, which we handled<p>What do you mean??? Nuclear weapons can't simply be downloaded and run by anyone in the entire world. Superintelligences can. Nuclear weapons can't slop the world into passing age verification laws nearly in unison, can't keep the general population fooled into thinking it's fine when democracy is falling out from under them. A nuclear attack would wake people up, superintelligence doesn't have to. This is a far bigger problem than nuclear weapons because at least we would notice nuclear weapons. At least we mostly know who has nuclear weapons. At least we have agreements about nuclear weapons. At least mutually-assured destruction is even POSSIBLE with nuclear weapons. At least those with nuclear weapons are literally at all incentivized not to use them. But AI is something that's very very easy to feel like you can get away with, and PEOPLE FUCKING ARE! And the worst part is that <i>any</i> random individual can be unexpectedly formidable with the help of a superintelligence and there is literally no way to know what will happen next. Anyone could do anything, any individual could make an extremely outsized impact. It's <i>already</i> starting to be a huge problem and we haven't even reached anything <i>close</i> to superintelligence yet.
Love to see that "superintelligence" that some random person will "simply" download and run when there are relatively only few capable of running today's near-to-frontier models, and actual frontier models are still a ways from being AGI, much less getting to the point of ASI.
> Anyone could do anything, any individual could make an extremely outsized impact.<p>So the problem is people. Burn them all !
The problem is indeed people. How do we make the default choices most people make, better?<p>Consider that a lot of people will be very happy to ask an AI what to do when in the past they may have taken no advice at all. It's a hell of a burden but also a wonderful gift. If anything, progressive countries might eventually want to guarantee some basic AI access for people of all income levels.
I wouldn't be so sure. Given that the general idea is that commodity AI is terribly censored and filtered, a lot of people will probably seek out the most uncensored/abliterated models for their use, simply because they're uncomfortable with the idea of being censored or manipulated by the bigger labs. Despite that though, some people probably will benefit from the alignment done by the larger labs, though as we've seen with OpenAI's sycophancy crisis, that has been a bit hit-and-miss lately
I've tried some abliterated models. So far, they're not evil - you can make them say evil things, but they don't leap right to it without a bit of pushing. Or perhaps I'm not asking the right questions...
> Nuclear weapons can't slop the world into passing age verification laws nearly in unison<p>Why do you think LLMs are responsible for this? Governments all around the world copied each other with COVID laws as well, in a much shorter time frame, without LLM assistance. Social contagions exist in politicians as well as teenagers
> Why do you think LLMs are responsible for this?<p>I don't have evidence that <i>every</i> age verification law has anything to do with AI, but it's been coming out that the movement in Australia has seemingly been done by generating mountains of LLM slop and trying to slip it through the regulators as fast as possible before anyone has enough time to figure out what's happened.
"This is not a marketing stunt," says the marketing stunt.<p>Betting he got to keep all his RSUs
*How?* and *Why?*<p>The most intelligent people I know are the least likely to want to harm anyone or anything, and understand that diversity is fundamental and important to the universe. Without proof to the contrary, why would you think some super intelligence would want to hurt anyone? Because you would?<p>If you are saying that some small bit of training data made the thing completely evil, then that really couldn’t be super intelligence.<p>These doomer people keep running around saying these kinds of things, but they all just seem like people who play too much D&D and want to larp as the main character.<p>Happy to be shown something that isn't based on wild speculation and some randos “this is whats going to happen in 2030 because of my vibes” kind of information.
I think plenty of the most intelligent people eat meat, which means they are perfectly fine with harming less intelligent species just to enjoy a tastier meal. Also, I don't think many of the most intelligent people would be particularly concerned about disturbing a few ants if they were the only obstacle to economic activity. Intellect-wise, we will be less than ants to superhuman AI.