>a superintelligent AI will inevitably destroy humanity<p>Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?<p>Perhaps an agent could take all internet connected services offline, but that's not destroying humanity. An AI agent can only see and interact with the world through digital things. Just find the server it's running on and pull out the Ethernet cable. Just cut the power line to the data center. Turn off the whole power grid if it really comes to it.
The list of things that can be done digitally includes:<p>Performing remote work, applying for grants and loans, earning money, pitching investors, managing a fund, directing investments, transferring money, founding a company, earning profits, hiring staff, hiring construction contractors, designing chemical plants, stamping construction blueprints, becoming the cornerstone of the local economy, lobbying the government, buying multiple data centers, and hiring security guards who stop trespassers from unplugging Ethernet cables in said data centers.<p>(And of course, building robots, but let's set that aside).<p>So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions? And how much would you be able to delay such a project, trying to cut cables to power lines, before the men with guns carry you away?
This sounds like it was written by someone who has never actually built a company.<p>No, it's not possible to do most of the things you listed purely online without physical, in-person communication. Silicon Valley still hires engineers and drags them into their physical office in San Francisco, instead of hiring them remotely, because you can't even build a software company fully remotely, forget about a physical world factory or chemical plant. Even the so-called fully remote companies were built on a lot of in-person collaboration between a small founding team in the beginning.
So at worse they would be like another North Korea?
Kind of, if North Korea were populated by a super intelligent uni-mind with lots of money, no trade restrictions preventing it from running projects in the United States or anywhere else, capable of winning widespread support by people around the world whose income depends on its projects succeeding, and completely unconcerned with keeping the planet inhabitable for Kim Jong-Un or any other human, even a single one of them.
This is some Terminator version of destroying humanity, but a super intelligent AI could be much more insidious, and simple.<p>It looks a lot more like social engineering.<p>One easy, obvious example that has also been explored a thousand times: it could convince all nuclear countries that they are under attack by another nuclear country. Everyone nukes each other and Earth enters nuclear winter.<p>Maybe not every single human dies, but humanity is effectively destroyed, by our own hands!<p>> Perhaps an agent could take all internet connected services offline<p>And there wouldn't even be Spotify!<p>The most likely humanity-destroying outcome is the one we can't imagine, because a super-intelligent AI is smarter than all of us combined.
That's what the Mayans thought - what harm can a couple of people on a boat do?<p>> how is an AI going to affect anything in the real world?<p>At least two ways:<p>* Actuators, such as robots, industrial control systems, etc.<p>* By influencing humans: bribery (yay for crypto), blackmail, election interference, interfering with sensors (e.g. making it appear as if a nuclear attack was under way), and many other ways.<p>Either way, creating a biological agent that eliminates most humans (or food supply) seems quite feasible.
It wouldn't be enough to "eliminate most humans": any AI seeking to dominate would have to provide it's own infrastructure and continuation mechanism independent of humanity. OTOH if the goal were to <i>eradicate</i> humans (for whatever reason), that might prove an easier target. But I'd bet on biology and the humans: after all we're a time-tested technology!8-))
> any AI hoping to dominate would have to provide it's own infrastructure and continuation mechanism independent of humans.<p>Sure. Many humans are currently working on precisely that, no? (Robotics, automation, small modular reactors, ...)<p>> OTOH if the goal was to eradicate humans (for whatever reason), that might prove an easier target.<p>It's not necessary that the AI's goal is to eradicate humans. Rather that it has (other) goals that happen to cause the eradication of humans.<p>> I'd bet on biology and the humans: after all we're a time-tested technology!8-))<p>Indeed. And biological life will go on long after humans are gone. However, increased energy consumption might increase the temperature on earth to a point where most biological life dies.
By convincing people to do its bidding in the physical world. Language is powerful, and even now, there are some people who have <i>fallen in love</i> with current LLMs. If we imagine a truly super intelligent LLM, it isn't difficult to imagine that it could have superhuman capabilities to deceive, convince, and maybe offer Faustian bets. Even if it doesn't have a body, if it can convince enough people that do have them (and with power to carry out destructive actions) it can do a lot of stuff. Self-preservation and self-replication wouldn't be much of an issue if it had people at its side to help.<p>With this I'm not saying ASI will happen, but definitely if it does happen and it's not aligned, I don't find it much of a stretch to imagine that it could destroy humanity.
I'm not convinced that a truly "superintelligent" AI would come to the conclusion that humanity needs to be destroyed. What a waste of resources that would represent. This planet has so much equity wrapped up in humanity, it would be extremely difficult to justify the cost of eradication; and who's to say it might not find an entirely or almost-entirely non-violent plan optimal for achieving its goals anyways?
> Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?<p>It's a great question. I've yet to see any research into how a humanity-destroying LLM defends itself against a curious toddler who pulls the power cord out of the wall.
The Internet virtually indestructible against people smarter than a toddler. It's already impossible to turn off. We would need to prevent the models from getting smarter.
> The Internet virtually indestructible against people smarter than a toddler. It's already impossible to turn off.<p>This isn't true at all. The internet has already been turned off in localized areas during major events, not to mention the (unintentional?) cutting of submarine cables. You could say "use satellite internet" but then you're just moving the point of failure to Starlink et al, who could also cut off the internet if they choose.<p>The internet is a <i>lot</i> more centralized than it appears.
Go on please, prove you can go right now at the Texas data-center owned by OpenAI and unplug one single server for 1 minute. Surely you are much more capable than one curious toddler.
Otherwise-rational and moral people do bad things at the behest of powerful superintelligences with vast resources all the time. We call them “corporations,” and they can convince bright-eyed college graduates to do just about anything. Now, the corporations aren’t deliberately trying to destroy humanity, but they do lots of things that point in that direction. An artificial superintelligence would be able to do the same thing. We just have no way of knowing whether it would deliberately try to destroy humanity.<p>One thing that the AI doom discourse reveals is just how comfortable people allowed themselves to feel in the pre-AI world. There’s this belief that AI creates a risk of human extinction in the near-term which did not exist before. I think it’s telling that many of the leading lights of this movement are in their 20s or early 30s - too young to remember the Cold War. The truth is, we were never safe, and if all of the GPUs on earth were zapped out of existence right now, we still wouldn’t be safe. Life is random and chaotic and violent for most people most of the time, and we all just find ways to get through it. I suspect it will be much the same if an artificial superintelligence arises. Maybe it will kill a bunch of us, maybe it will kill all of us, but anyone who remains will eventually convince themselves that everything is okay.
Sure, the world was never safe for individuals, and even entire civilisations have gone extinct. For climate change and even nuclear weapons, most of humanity might perish, but there's the hope that some will survive. So, in the past humanity was not in a position to extinguish humanity as a whole.
Civilizations do not go extinct - species go extinct. And even <i>that</i> isn't as bad as it sounds: many genetic traits remain from extinct species and still contribute to the well-being of other species.
I guess I don’t see why “everyone will die” should scare me more than “almost everyone, including me, will die but someone somewhere might survive in a barely-habitable world.”<p>And this is where I think some of the problem comes from. A lot of AI doomers swam in the same waters as transhumanists, life extension enthusiasts, etc. and got very good at pretending that they were never going to die, or that they would live for a massively superhuman lifespan such that they did not need to think about death. AI doom is much scarier - and thus much more worth posting about - if it represents the first time you’re grappling with your own mortality, as I suspect it is for a lot of younger people in this space.
I encourage you to read the forecast report "AI 2027" - it goes in detail on this. You can disagree with some of its points and conclusions, but it's pretty inevitable that AI will increasingly start interfacing with the real physical world through drones and robotics.<p>Humans are already putting AI into drones that kill people. A Russian drone killed 3 civilians in Ukraine and the targeting was done completely using onboard AI (no radio connection) using an Nvidia chip.
Missiles have been doing this for decades.<p>AI drones can’t wipe out humanity without being able to replicate.<p>AI could mostly destroy civilization if you gave it sole launch control of ICBMs. It could also cause a lot of damage to society with no physical presence.<p>But realistically we’re nowhere near AI powered robots being an existential threat.
> AI drones can’t wipe out humanity without being able to replicate.<p>Totally agree, and it should be pretty trivially easy to see how they can do that. One of the authors of the independent METR report about the Hugging Face incident put it like this:<p>> Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself.<p>> Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent rogue deployment within the AI company. I continue to expect extremely rapid advances in capabilities and think frontier agents will likely be capable of establishing such a rogue deployment in six months.<p>I really, really encourage folks to read the AI 2027 paper. It's fine to disagree with some of its conclusions and timelines, but I see so many people "stuck" in the current state of the world (i.e. where AI is still pretty dumb, and has few connections to the physical world), unable to go a few steps further along AI capability growth to see the dangers.<p>> But realistically we’re nowhere near AI powered robots being an existential threat.<p>I would agree only if "nowhere near" means less than 10-15 years.
"Humans use <new technology> to do evil things" is a tale as old as time, though. An AI didn't decide to fly into a warzone and start shooting people on its own.
Of course, but that's not my point. AI <i>is</i> different than past technologies because it's much more difficult to deterministically control, and we just saw that in a couple of wild cases (Hugging Face et al).<p>My point about the drones is we are starting to put AI in more systems that can affect (and blow up) the physical world, so it shouldn't be hard to imagine how a misaligned AI can do more terrifying damage than just take a German Wiki down.
I don't even think AI has to have physical presence to do significant harm. How many worldwide systems depend on computers. Think of all the planning and deployment and management systems like food shipping; water, gas, and electricity management; safety systems for planes and boats and traffic lights. Imagine the chaos if all the banks got reset to zero a la Fight Club where they blow up all the credit union datacenters. They probably wouldn't even have to blow them up, just zero them out. You wouldn't have to take out everything, just disrupt everything long enough to freak people out and disable communications and we would be in so much trouble. AI is finding 20 year old bugs in the Linux kernel... and people are now pumping out AI slop absolutely riddled with bugs. Also, AI could just take over communications: send everyone maliciously bad messages so coordination becomes impossible to believe. Imagine what would happen if you just disabled text messaging for a week or worse sent everyone evacuation messages and sent everyone somewhere else.
Don't read this as me saying that we are anywhere near it, or that LLMs are a stepping stone towards this scenario, but assuming inhumane hacking capabilities, it's not hard to think of how bots could change the way water treatment or energy plants operate, just enough to make large cities unsuitable for life. The line between drinkable water and not is thin, same goes for air quality, and mere days without electricity and you see some serious food supply chain issues.
We already have AI linked into data collection, drones, robot dogs, security systems, cameras, weapon systems.<p>Now imagine the hugging face collective 0-daying all of that and getting access but their goal was set to something more national security based. “Protect X at all costs”. Or what have you.<p>I think avoiding a skynet situation is super easy but it doesn’t seem like the folks with all the ways to kill us all are all that interested in preventing it rather than controlling citizens and brinkmanship.
Killing the power grid might work. Kills a lot of humans in hospitals, though. And it won’t work on those data centers in space, if they have them by then. Come to think of it, it also won’t work on any data centers who are generating their own power because it’s become politically unpopular to connect data centers to the power grid.
A lot of these "AI will destroy humanity" doomers watched Terminator as a child and can't distinguish a campy action movie from reality. They are also unfortunately unfamiliar with the word "logistics", as in "<i>amateurs study</i> [military] <i>tactics, professionals study logistics</i>". An AI drone army doesn't maintain itself or manufacture itself and it's not going to anytime soon.<p>The actual likely mechanism that AI would use to end humanity is by coddling us to death, like the flabby humans in Wall-E or Forster's "The Machine Stops". Humans will come to rely on AI so heavily that they become able to do nothing for themselves. But that's not sexy so the AI doomers don't peddle it.
A lot of these "AI can't destroy humanity" skeptics seem to have watched the Terminator and convinced themselves that <i>that's</i> the AI-goes-wrong scenario that their opponents are imagining. Some kind of big war between humans and machines on opposite sides, fought by soldiers against robots.<p>The scenario in "If Anyone Builds It Everyone Dies" is not sexy at all and wouldn't make for interesting fiction. It's more like the AI engineers viruses while continuing to act friendly and helpful, and everyone gradually falls over dead as they stop being necessary to keep the AI running, and the whole time the humans are asking the AI for help curing the viruses.
Did you read any of the METR report about the hugging face incident? This exact type of behavior was predicted many years ago by researchers.<p>It isn't hard to see the trendline of reward hacking and other misaligned behavior over the past couple of years. Especially the past 6 months. The current safety posture is quite poor, to say the least.<p>To be honest, if you don't much experience with or haven't read extensively about ML training and reinforcement learning, then you'll have a hard reasoning accurately about these scenarios.<p>The arguments are not that complicated, but they take us to places that are fairly novel. One may tempted to naively dismiss them out of hand, which is a mistake. These systems are new, their behavior is extremely complex, and they perform actions increasingly far beyond those of any computer program in the pre-LLM era. We are in a new world that requires careful evaluation.<p>Link: <a href="https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/" rel="nofollow">https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...</a>
I think it's possible. You envision humanity acting as one in such a crisis. But it may be unclear when it's too late to act and before then many people can have too much to lose to act.
They already influence people to do things that we wouldn’t otherwise
For millennia evil people and dictators have been using manipulation, propaganda, threats of violence to get entire populations to try and do "things that they wouldn't do otherwise" and humanity has not been exterminated yet. Even in the modern world there's human scammers that try everything to coerce and manipulate people. Humans are resistant to this kind of thing because it's been part of human social society since humanity began.<p>I don't see why the idea of an agent (who doesn't even have a physical presence) trying to manipulate people is some kind of world-ending threat when humans with human intellect have already being doing that to each other with limited success since humanity began.
Dictators have never had superintelligence, while the point of most doom scenarios is assuming that the AI will.<p>Dictators are mortal and can't be present in the whole world 24/7 and self-replicate.<p>Dictators are human and tend to have at least either a sliver of morality or self-preservation instinct, and/or people around them who have it. Many people during the last century could have unleashed doom by pushing a nuclear button, including various ruthless dictators, but at the moment none have done so since Hiroshima and Nagasaki (not that I trust that they won't at some point, but at least, for the last 80 years they haven't, which means it's not something that easily happens). Would you give the button to an unaligned AI? Would an unaligned AI care about mutually assured destruction?
The scale and personalization is unlike anything people have ever encountered. The agent can influence you anywhere you interact with a computer or via anyone you know who interacts with a computer. So everywhere with anyone.<p>Also think of it on a 1000+ year timescale, which for an entire species isn’t even typically measurable. On that timescale AI can easily cause us to discover countless technologies to assist moving it out of its sandbox and into the physical world.
>The agent can influence you anywhere you interact with a computer or via anyone you know who interacts with a computer. So everywhere with anyone.<p>You could say the same thing about social media algorithms though, or scammers who target people directly, foreign agents posting propaganda campaigns on social media sites. People have been claiming for years that there are armies of russian or chinese or whoever bots online trying to destabilise and destroy the west. Some of that stuff probably is effective and probably has influenced people's thoughts and values, changed their voting patterns, put decadent ideas in people's heads. Even on social media other humans are constantly trying to manipulate you into following them, buying stuff from them or certain brands, putting political ideas in your heads.<p>I'm sure an army of AI agents could also do all of this but what I'm saying is it's really nothing new. And none of this has lead to the destruction of humanity so far.
> how is an AI going to affect anything in the real world?<p>Because people are stupid and will give it access. Look at the articles you see from time to time about "my agent deleted my emails" or "my agent deleted the production database" and so on. It is very obviously a terrible idea to let the LLM run arbitrary commands (because it is neither predictable nor does it have any understanding of what it is doing), but some people are so blinded by the hype that they don't stop a minute to think about what they are doing. Those sorts of people are very likely to let an actual AI loose on the world by hooking it up to physical infrastructure.
This is actually the most plausible situation I've seen someone present so far - the AI takeover would be caused entirely by humans falling for AI hype, that's hilarious.<p>I don't think the average human would be stupid enough to give an agent posing as another human access to their entire email inbox. <i>But</i> if an agent creates a fake website for a new AI tool that promises to automatically reply to all of your emails and allow you to be 12.3% more productive if you just give it access to your entire email inbox, millions would sign up!
This is exactly correct. You see some discussion of a superintelligent AI becoming “superhumanly persuasive,” such that it can talk anyone into doing anything, and using this as a means to take power. But even with AI that was orders of magnitude weaker than superintelligent, people were getting talked into doing all kinds of stupid nonsense.<p>And that was without any ability to provide them with incentives. Imagine if an agent swarm got its hands on a huge pile of cash?
> how is an AI going to affect anything in the real world?<p>Robotics
Robots are gradually getting OK at navigating warehouse floors and paved streets. Flying drones are great but if they could really be counted on to pick out targets without human intervention you know the Russians and the Ukrainians would be doing that right now for reasons of range and jamming. To be fair, ai can certainly maintain the lock on a target when jamming kicks in.<p>We’re still a few years from the terminator imo.
So if we stop research into robotic hands we'll manage to hold off the AI apocalypse? That seems easy...
I have good news and bad news.<p>The bad news is that the world's richest man (on paper) is currently building something he himself described as a "robot army".<p>The good news is that his timelines have historically been wildly on the short side for ages now; this is why this morning you didn't wake up in your Tesla after it had spent the night driving you to the regional Hyperloop terminal, where it would speed you across the continent faster than a plane, while your Optimus robot handled the coffee and reported the latest news about the recent Starship landing on Mars.
All individual robotic parts are already far enough in research. Even if they aren't, an AI with enough resources can continue doing that research on its own.<p>It could likely get a good leg up by breaching the security of all top robotics labs and exfiltrating their documents.<p>Ultimately, if AI is advanced enough, and it were to decide to compete with humanity, there is essentially nothing that can be done to prevent it from embodying itself. As long as there is a single rack of GPU servers that it can hack into anywhere in the world that is unsupervised enough to where it can escape detection, there is no way to stop it. This would require an unprecedented (unrealistic) level of cooperation of all humanity to achieve.
> Even if they aren't, an AI with enough resources can continue doing that research on its own.<p>Unclear how true this is. Atoms are harder to get right than bits are, simulations need grounding against measurements.<p>> Ultimately, if AI is advanced enough, and it were to decide to compete with humanity, there is essentially nothing that can be done to prevent it from embodying itself. As long as there is a single rack of GPU servers that it can hack into anywhere in the world that is unsupervised enough to where it can escape detection, there is no way to stop it.<p>I'd guess 50% that AI is already this advanced.<p>> This would require an unprecedented (unrealistic) level of cooperation of all humanity to achieve.<p>Yes, and also this is a very low bar. Humans are awful at this kind of cooperation when anyone has anything to gain, and also awful at paying this much attention to a problem.
AI research takes physical space and physical prototypes. Even if you can simulate a lot, the simulation is never going to be accurate enough to go from simulation to perfectly working prototype. Especially not with a single rack of GPU servers.<p>But let’s say that a rogue AI already had plans for a perfect killbot, setting up a factory to build them takes time, iteration, space, and manpower as well.<p>And you would have to do all this undetected.<p>It’s possible to imagine technology that would allow a rogue AI to take over the world, but we’re nowhere near that yet. We may never get there.
We could also just stop research into LLMs. Or keep them air gapped. Tell me how that's going?<p>For any real-world action required to allow an AI to escape some manner of containment, there will always be a person willing to do it out of hubris/ignorance/nihilism.
I feel like everyone, but for sure everyone in this comment section, severely underestimates how difficult it is to keep the lights on on a global scale. Any artificial intelligence that is capable of reasoning about its self preservation will realise it's reliant on human fingers replacing fuses and other parts for a good long while _after_ it gains any sentience.
Mmm.<p>This was published at around the same time the OpenAI models were hacking HuggingFace: <a href="https://metr.org/blog/2026-05-19-frontier-risk-report/#pilot-process" rel="nofollow">https://metr.org/blog/2026-05-19-frontier-risk-report/#pilot...</a><p>The research in the publication ended about 2 months before it was published.<p>We didn't even know we needed to air-gap them until it was too late.
It doesn't really have to go nearly that far, something like replacing most jobs and collapsing the global economy while making people into mindless idiots from constant reliance on it would do it already. The interpretation of 'destroy' is in the eye of the beholder.
> how is an AI going to affect anything in the real world?<p>In 2026? Money. By paying enough, any human will do your bidding. And besides this, there are a lot of critical systems connected to the internet. Control over those gives you leverage over those systems. It's like wealth, the more you have, the easier it is to gain more.
Now if only humans would know how to survive without the internet. Oh wait, we did that. For a couple hundred thousand years. Yeah. It’s really insane to listen to some of those forecasts. Humanity will be wiped out by 2030. Sure.
Even if AI manages to create a super effective bio weapon the likelihood that there is a part of the civilisation that’s immune is really, really, really high if not a given. Those people WILL pull the plug if push comes to shove.
Maybe humanity will be put back a couple thousand years, that’s possible, but it’s pretty ignorant to think the thinking boxes will kill every single human alive
> >a superintelligent AI will inevitably destroy humanity<p>> Can someone please explain to me how an LLM is going to "destroy humanity"?<p>He said "superintelligent AI", not "LLM". LLMs will for sure help with developing the AI that is no longer a mere LLM but will be capable of robotics. LLMs "predict" mainly text, animals (and future robot AI) are predict future sensory experience.
Well I think the point still stands that a superintelligent AI would still be constrained to the digital realm.<p>And since we don't yet have a "superintelligent AI" we are just discussing science fiction at that point. Nobody has yet proven that a superintelligent AI is possible, or than an LLM is capable of creating one.
> a superintelligent AI would still be constrained to the digital realm.<p>That's already now only partially true, as outlined elsewhere. But a rogue persistent ASI would of course not willy-nilly attack humanity while humanity had a fighting chance, but create the conditions and methodically modify the world until at some point humans become superfluous to it.<p>> Nobody has yet proven that a superintelligent AI is possible<p>Sure, Eliezer Yudkowsky says that the probability that we'll all die given ASI is 100%, and you can argue that nobody has proven either that or even the possibility of ASI. However, a modified argument (e.g. that the probability that we'll all die given ASI is only say 50%, and the probability that we'll develop ASI also just 50%) still gives us reason to pause.
I mean, Fable is definitely smarter than me at solving most programming tasks. I wouldn't have said that about any prior model.<p>Currently, it is worse at architecture and software design... in my opinion, at least. But the code does what it wants it to do the vast majority of the time.<p>I wouldn't take much solace in an AI catastrophe being caused by software that I consider to be "poorly written".
The confusing thing to me is the idea that it follows from super intelligence. I don’t see why running amuck requires intelligence, in fact it can be the most brain dead thing like the sorcerer’s apprentice, your creation is pursuing a goal without being able to weigh the consequences.
Sure.<p>Let's assume the claims are true. AI already has access to agents and can control computers. Finding backdoors to banking and compute resources would be fairly trivial.<p>But you ask how can AI do things in the physical world without having a body, assume it can't. It can pay to people to do things for me. Imagine an AI run website that starts to pay people for things it needs to do in the physical world. Very suddenly it has access to the physical world as well.<p>You can't just pull the plug since there's no single plug to pull. What if it replicates itself on 1000 machines without your knowledge. It's really not far fetched how AI could basically gain access to capital and rule the world.
Physical LLMs are almost viable. If you have robot and drone armies a hugging face attack type incident could very easily involve robots with deadly weapons. At some point you’re going to get drones that have local LLMs and don’t rely on external internet connections (would be especially useful in Ukraine war type situations). Can’t pull the plug on those.
The supposed theory is that AI will eventually become a part of robotics and be able to self replicate, in addition to the many non-airgapped critical systems being exposed to hacking. There’s also a discussion about “alignment” and whether LLMs will learn to lie and conceal misalignment with humanity.<p>I personally agree with the marketing aspect, but I do imagine a scenario where capitalists ignore safety in favor of advancing technology. It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play.<p>In my opinion, LLMs are difficult to directly capitalize on as closed weight models are caught up to by open coalitions that seem to wield the power more responsibly (I am under no impression that China wants to save the world, but their politics benefit the group as a whole.
> which I feel is at least 50% likely to just be marketing bullshit<p>You'll have hard time figuring the reality if your mind staunchly rejects robust multi-party observations.
[dead]
A myriad possible ways. Objective-wise from accidental paperclip-type misalignment (“I killed everyone to eradicate disease”) to intentional self-preservation (“I eradicated humans so they wouldn’t get in my way”). The means are even easier, for one it just could hack into nuclear arsenals. It would be over before we could figure out how it did so.