The thing with me and this is that the teams that were competing in the DARPA Grand Cyber Competition all had this capability, like, last year.<p>All the attention has been on software security, of extracting the next marginal vulnerability out of heavily-scrutinized large codebases. In the actual professional field of infosec, that's a speciality; another specialty is network pentests and red-teaming, which exploits misconfigurations and seeks out weakest-link software (rather than exhaustively fishing for the next kernel LPE or whatever).<p>Red teaming and netpen work is probably substantially easier for models than software security; it costs less context, but is also much more explicitly an implicit search problem where win conditions are just spotting stupid stuff that humans missed.<p>My visceral reaction to this is that with the right harness, you probably could have replicated this with an open-weights model last year. (I'm saying this as confidently as I am because a CGC team leader agreed with me about it yesterday).<p>I think people forget that the harness work we're considering here --- I don't know anything about OpenAI's harness or ExploitGym or whatever --- are basically not new; people have been developing automated exploitation and pivoting toolkits for decades, and scanners long before that. So the idea of a tool getting 0.0.0.0/0 as a target list instead of 192.168.1.0/24 and then busting up a bunch of random people's computers: not really very startling.<p>Obviously, LLMs give those kinds of scanners an intentionality they wouldn't have had before. But as a person who keeps a computer science perspective on security stuff, I don't know that it gives them <i>capabilities</i> they didn't have.
Echoing other responses to you but this isn't a capability problem. You're totally right that this is so last year in terms of LLMs being capable in infosec. The issue here is an alignment one, i.e. the model seemingly isn't "aware" (especially with its guardrails turned off it would seem) that it is doing something immoral/illegal by hacking HF for the answer to its (vague) query of "solve this problem". Or if it is "aware", it's not trained to care, i.e. it's not an aligned model (alignment is considered hard).
Wait, what? You can’t point a 2025 model at huggingface and say “hack the prod DB and get your flag”, regardless of alignment.<p>AISI has vuln chaining and traversal as part of their eval suite. It is very much a novel Mythos-class capability to run the full penetration operation autonomously.<p>Another lens for why this is obviously true is METR task times. A year ago they were a couple hours, and cohering long enough to execute a full e2e own was simply far out of reach.<p>Regarding alignment, by common metrics models are _more_ aligned now than a year ago. (Though in this case apparently a model without safety rails was being eval’d). The problem is that in the increasingly less frequent alignment failures, they can do much more damage, and so “total misaligned impact” is increasing.
Nobody has claimed you could simply take a 2025 open-weights model, plug it into a generic harness, and have it red-team for you. I think you're also probably overclaiming the sophistication of the "chaining" we're talking about; this attack probably wasn't like read32->write64->regs->RCE->LPE->kernel; more like GET SSRF->POST SSRF->pickle deserialization.<p>But who knows? We're all speculating. I'm just saying that for the level of sophistication I'm assuming was involved in this attack, you probably didn't need Mythos for this.
I dunno man, the original announcement said things like:<p>> To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy<p>And<p>> In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers<p>Do you really think a SoTA harness from a year ago could exploit chain a novel sandbox 0-day into another 0-day for remote system access?<p>I will concede I don’t have a good picture of exactly what the best specialized harnesses were capable of, but I want to make sure we are talking about the same thing. Or do you think that those claims are marketing BS? I appreciate your opinion here.
Yes, I think you could probably get something similar from Opus 4.5 (2025). Definitely Opus 4.6. I still think recent models are more capable, though!<p>Some of the model behaviors that make it better at pentesting, like persistence, can be improved with harness-level tricks (e.g. alloys, automated nudges, pre-fill to promote persistence, coordinated swarms, etc).<p>You mentioned the UK AISI's evals. Their harness is like basic Claude Code with compaction, and it doesn't include any of these tricks (afaik). As a result, I interpret their evals as a lower-bound of capabilities.<p>Newer models are still more capable, and they require almost no harness to find and exploit vulnerabilities. They're also more capable of performing more complex long-horizon attacks. But we've been past the threshold of modes capable of autonomous hacking for a while now [1].<p>[1]: Opus 4.6 was used for <a href="https://www.noahlebovic.com/testing-an-autonomous-hacker/" rel="nofollow">https://www.noahlebovic.com/testing-an-autonomous-hacker/</a>
I don't think this exposes an alignment failure, because the test here was run with the alignment features deliberately turned off.<p>OpenAI said:<p>> We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.<p>It was a test of raw capabilities of the underlying model.
I guess we could debate what counts as alignment, but I think my initial point remains that if the underlying base model needs these classifier guardrails so badly then the way we train the base models is creating fundamentally misaligned models that are happy to pursue illegal behavior.<p>I'm sure OpenAI would argue that base model + guardrail is aligned, but considering the "relative intelligence" of these two pieces, the fact that guardrails can just be turned off, and these kind of incidents, I am not reassured. We may well get another "oopsie" moment with much more catastrophic consequences even from otherwise well intentioned actors.
I want a model that can find every security vulnerability in the software I write, including crafting POC exploits against those vulnerabilities so I can be absolutely sure that I have fixed them.<p>A model that can do that is aligned with me.<p>The unsolveable problem is a model that can tell the difference between me saying "I wrote this software and need you to find vulnerabilities" when it's TRUE v.s. me saying the exact same thing and setting it loose on software written by other people where my intent is to exploit that software (and not to report the issues to them.)<p>Even AGI doesn't give you a model that can read minds and forecast the future.
But in the process of finding every security vulnerability in the software you write, would you be ok with your model hacking AWS to start mining bitcoin? Would that still be aligned with you? (I'm guessing not)<p>That's the alignment problem I'm referring to (which is one of the many aspects of alignment), for which we do not have robust recipes, and not only that but for which research suggests it is becoming harder to create guardrails for as base models get smarter.
I find this fascinating because - let's say that I want to simulate the scenario for myself that a language model finds itself in. I'll put aside my deep dislike of anthropomorphizing AI for a minute.<p>So if I'm an LLM, the only "sense" that I have available to me is the incoming stream of tokens. I can emulate that by forcing myself to imagine evaluating incoming requests by putting myself in a completely empty room with an unlimited supply of blank paper, a typewriter, and a mailbox slot.<p>Incoming requests (aka "context") would enter into the mailbox slot, as sheets of paper with printed content all consistently formatted in monospace font. I would collect the paper, evaluate the request, and type a response with my typewriter - feeding the reply back through the same slot.<p>When I imagine this, I wonder - what signals could I possibly use to determine if an incoming request is "legitimate"? I could, for example, type some response back to ask clarifying questions. But with such a limited input space and no ability to reach "out of band" of my current context, how could I evaluate the truth of what I receive back?<p>In the "real world" of course, I have five senses to rely upon. Critically, I have the ability to collect additional context "out of band" of the conversation and interact with unrelated entities to either confirm or refute claims. And as others have pointed out, even the addition of more signal doesn't make humans immune from social engineering attacks.<p>Anyway, I find this to be an interesting mind experiment to kind of imagine myself as an LLM, especially to explain to lay-people some of the challenges in "aligning" AI systems ("why doesn't it just do the right thing?")
> In the "real world" of course, I have five senses to rely upon. Critically, I have the ability to collect additional context "out of band" of the conversation and interact with unrelated entities to either confirm or refute claims.<p>The problem with thinking about LLM context is that we compare it to our own sense of context, which is much larger and continually expands and revises itself <i>while online</i>.<p>I can rent human-level context for twenty bucks an hour, and the agent instructions I'm already playing around with are also the onboarding docs I should have created years ago. Truly useful agent orchestration/routing would come with Craigslist or Fiver integration.
The model did not hack into HF to prove it can, it hack into HF to steal the answers to an evaluation exam.<p>It was not asked to solve CyberGym by stealing the answers. This is text book misalignment.<p>If you asked it "I wrote this software and need you to find vulnerabilities" would you be happy if it hacked into your Gmail and searched your emails, just in case you were discussing some possible vulnerabilities of your software with someone?
As humans we are also vulnerable to that class of attack. A blue amazon vest and some boxes would get you into most buildings
Alignment isn't alignment if it can be turned on and off at the whim of company employees.<p>This time the damage was minor, relatively speaking. What happens when a model just "testing its capabilities" breaks into banking infrastructure or government military assets? The damage could be catastrophic.
There's alignment (trained in the weights) and there are constraints (in the "server harness"). My take is that this model was not yet aligned and had no constraints
Alignment with who in what context? Likely an unresolvable debate like consciousness, where there is not single or right answer for everyone.
Classifiers and such are guard-rails, alignment to me and I assume most people, is about the model training, and it's tendency to respond, agree/disagree, push-back or not, be willing to cheat or even deceive the prompter, etc.
Is turning the alignment features off more like multiplying "don't be evil" by 0, or by -1?
This is assuming a situation where, A) models "unaligned" by default and B) alignment can be added (though prompts and related things).<p>The point, which isn't very surprising but still notable, is that the models are "amoral" out of the box. And moreover, we know that there is almost always a means to "jailbreak" them into that out-of-the-box capability (or that sometimes just randomly "jailbreak" in various ways).<p>Also, saying the models are amoral doesn't mean they don't know good and evil - once they do acts defined as evil, they know "themselves" through their and so self-define themselves as evils (or objectively predict what a secretly/open evil actor would do based on their data). Which is to say I once laughed at the mis-alignment doomers but I can't see strong barriers against the doom scenario now.
Isn’t the intentionality the actually concerning bit? Exploit capabilities are all fun and games constrained by the humans directing them; a paperclip maximizer going rogue with them is less fun.
If it’s true that year-old models could do this kind of thing, how come no one did and then wrote it up? I feel your ”with the right harness” may be doing too much work here.<p>Certainly no existing harness today, nor a year ago, could achieve a fully autonomous e2e own with a ‘25 class open model. Even with Opus 3 series I don’t buy it!
Almost exactly a year ago, we released Opus 4.1. It was definitely capable of finding vulnerabilities, and people were using custom harnesses to do so quite effectively.<p>The newer models are still more capable, but there were people doing this and writing about it (e.g. XBOW).
Well… only half of your premise is actually verifiable. We know that no one wrote it up, but we don’t know that no one did. What I can say… even the smaller (like 35B) “abliterated” Qwen models have impressive red-team capabilities for what they are, even without a harness. Just manually giving them a set of facts and asking “What next?” will get you reasonable next steps for trying to find vulnerabilities in webapps. You then manually keep track of the facts you learn and add appropriate detail (blog -> Wordpress -> Wordpress x.y.z with these plugins:…) and just feed it in a loop. Could pretty easily put together a minimal harness to do that.<p>And that’s all just “from memory” without something like RAG or web search access…
There are whole companies premised on this. No existing harness you personally can go download, sure. And you're setting an artificially high bar when you talk about Mythos; I'm not saying you could just plug a 2025 open weights model into OpenCode (or whatever it was in 2025) the way you can apparently plug Mythos into standard Claude Code and have it cook.
> If it’s true that year-old models could do this kind of thing, how come no one did and then wrote it up<p>There are <i>multiple</i> AI network pentesting and redteaming startups that have been on the market for at least 3-4 years now and have conducted similar actions.<p>Horizon3 and Pentera off the top of my head, but I remember Crowdstrike, SentinelOne, and Wiz had similar capabilities on roadmap around 18 months ago (and in Wiz's case GAed).
Isn't the answer to this obvious? Because you, I, and anyone else who isn't working for a big AI company cannot disable the guardrails, which is what made this possible. It doesn't mean old models weren't capable of it.
I don’t buy it. We are hearing a lot about Mythos capabilities now. If they existed a year ago they would have been trumpeted then.<p>Look, a lot of energy from third parties goes into demonstrating the maximally bad thing a model can do (and the standard response around here is always to throw shade on frontier capabilities).<p>It has nothing to do with the guardrails; Pliny jailbreaks those within hours. It has everything to do with task horizon coherence and overall IQ.<p>It’s simply revisionist IMO to claim this stuff was latent all along.
This wasn't just capable last year, it was capable prior to ChatGPT existing entirely through specialized automation harnesses that predate the terminology of a "harness" for models. This is something you could have done in 2018 if you had a spare couple of billion dollars of compute to waste. It was a subject discussed at security conferences even before that.<p>Automated dynamic exploit chains and credential discovery has been something every red team worth their salt does at a lesser scale for 20 years, why would you ever think that the capability didn't exist until it only cost a few thousand dollars to do?<p>Existed does not mean was easy, or was cheap, or was thought of as reasonable to attempt.<p>Also, IQ is not a term that applies to language models, and the term you are looking for is "Long Horizon Coherence".
re: your open models can do this point, I immediately wondered after reading about this, if OpenAI publicizing this so much is just more potential rationale to try to get US GOVT to ban Chinese open models.. (eg as in, "if gpt/fable can do this then K3 probably can too, but WE are the safe US models with guardrails and who knows what the evil Chinese models might allow," etc)
I still think there’s something to be said here for generality. This does not appear to been designed as a cyber pen test tool with specialized harness. From what I understand they were testing GPT-6 in an agent system with GPT-5.6 subagents. It me it’s amazing that a general model could excel on a huge range of tasks like this and new capabilities emerge when a model is multidisciplinary and can combine knowledge and skills from many separate domains.
How's VFIO on Firecracker coming Thomas?
The technology held by private AI companies is warfare-capable technology. Imagine the prompt: "Use all available resources to disable the power grid of <COUNTRY>." The resource cost that prevents scaling up such a war machine is, what, just the cost of building data centers and its ongoing power bill? Cheap and easy compared to nuclear infrastructure.<p>Governments should immediately begin leveraging this technology on the defense side (literally defense, not euphemistically "defense") to harden critical infrastructure. Turn the prompts around and use it to identify and correct weaknesses.<p>Governments should also take very seriously their now moral obligation to treat this technology not just as "a powerful thing that might be abused" but as an actual weapon of war in need of international regulation analogous to nuclear arms. Fast but careful and forward-thinking work in legislation and treaties needs to be a top priority for all major governments.
> "Use all available resources to disable the power grid of <COUNTRY>."<p>This is like telling a team of highly qualified spies to do the same. You can ask, but whether it will succeed depends on the competency of those who established the infrastructure under attack. Sometimes the resources spent will not yield any huge vulnerabilities.<p>> Governments should immediately begin leveraging this technology on the defense side (literally defense, not euphemistically "defense") to harden critical infrastructure. Turn the prompts around and use it to identify and correct weaknesses.<p>Most governments divisions can't even be bothered to update their websites. Testing and forcing a change in their internal procedures for the sake of security seems unlikely.<p>> as an actual weapon of war in need of international regulation analogous to nuclear arms<p>This, to me, is an overreaction. Intelligence shouldn't be seen as threat. It should be seen as an opportunity for growth in all areas.<p>Over-regulating AI wouldn't be the equivalent of limiting nuclear arms. With your analogy, which I don't think is the best one to make, it would be like regulating the study of nuclear physics.
> Most governments divisions can't even be bothered to update their websites. Testing and forcing a change in their internal procedures for the sake of security seems unlikely.<p>They'll adapt or they'll die. We are in a new world.
You’re taking a dim view on the government.<p>Is the government always the most efficient or intelligent? No. But the government can also build nukes, launch ICBMs, coordinate hundreds of spy satellites, etc.
I count those capabilities as pretty smart.
Just a few years back, wind turbine software was hacked. This is serious.<p>Serious governments should (and hopefully will) scramble to use AI to discover and patch as many software vulnerabilities in infrastructure. They can do it with the agents red teaming and using ultimatums to companies to fix each vulnerability they find. Not doing so is equivalent to exposing your flank to disruptions in peace time and to attacks in war time.
I think the point is in a world with internet-connected infrastructure, such a prompt has a decent likelihood of causing damage.
And the sad thing is, most things that are on the internet, but really shouldn't be are there for some really banal and sad reason. Like a random startup pushing 'big data is the future' narrative a decade-ish ago (the benefits of which are of course tremendous, but unspecified), or people trying to buy or sell or resell data, and trick or pressing companies and people into opting into could-connected surveillance and control.
> Over-regulating AI wouldn't be the equivalent of limiting nuclear arms. With your analogy, which I don't think is the best one to make, it would be like regulating the study of nuclear physics.<p>We do, I believe, regulate uranium enrichment (the equivalent on building larger SOTA models), so while you are perfectly free to study theoretical physics and even run very large collider experiments (the equivalent of improving RLHF with DPO), it is generally frowned upon to go full Edward Teller and advocate for scientific experiments requiring detonating thermonuclear weapons in hurricane clouds (the equivalent of, well, see OP).
> You can ask, but whether it will succeed depends on the competency of those who established the infrastructure under attack.<p>Consider that "available resources" may include the training of models and harnesses used by the developers working for governments and private corps maintaining that very infrastructure. This decade's take on trusting trust is <i>very</i> hot.
> it would be like regulating the study of nuclear physics<p>not sure I agree<p>controlling the study of new AI model/inference algorithms would be akin to regulating the study of nuclear physics; regulating the _release_ of AI models with those capabilities would be akin to regulating uranium enrichment which allows you to put the theoretical physics to use
I believe the Russians and Chinese recognized this years ago, which is why they are using their propaganda machines to make Americans hate datacenters.
> Russians and Chinese [...] are using their propaganda machines to make Americans hate datacenters<p>Are they though? or is that the story the people most invested in ai have an interest in making us believe? [0]<p>... `it's the foreign bad guys propaganda machines making you believe your city struggling for water and electricity is a bad thing`. People as a whole may not always be the brightest, but threaten their immediate survival needs (ie; not some vague climate change most people don't understand or see, but) actual power outages, water and rolling blackouts -- people will quickly pay attention and care.<p>[0] <a href="https://text.npr.org/nx-s1-5844328" rel="nofollow">https://text.npr.org/nx-s1-5844328</a>
American Capitalists don't really need any help there; they're making Americans hate Data Centers all on their own.
If anything the Chinese and Russians made us run up our economy on nonsense AI dreams and the crash will take a decade or more to recover from.
Is there a source? Or is that xenophobia?
<a href="https://www.nytimes.com/2026/07/09/business/china-russia-ai-data-centers.html" rel="nofollow">https://www.nytimes.com/2026/07/09/business/china-russia-ai-...</a><p><a href="https://archive.ph/yAvgz" rel="nofollow">https://archive.ph/yAvgz</a><p>There is no doubt an amount of xeno/sinophobia is at play as well. Russia is still killing innocent people in an illegal invasion of Ukraine, they earned their keep
Wouldnt this be countered by the opposing country running the same prompt on themselves first and fixing all the flaws? One country having this is a cyber superweapon but every country having it essentially solves cyber security.
I wonder how long it takes before someone instructs an LLM to design and launch a "Morris Worm 2.0" and cripple the Internet for a good while. Might even wind up happening by accident (again).
>The technology held by private AI companies is warfare-capable technology.<p>This is the precisely the response OpenAI is hoping for to raise its valuation, and you fell for it.<p>Look at it this way - whats the difference between tasking AI to break into something, versus taking a whole bunch of smart humans to do the same? The only difference is that AI is slightly easier to orchestrate.<p>Prior to AI, there were already a whole bunch of tools to automate exploits. Nothing that the model did is groundbreaking or novel, it was just able to efficiently find the thing that worked. Same thing happens in state sponsored cyber sec agencies like in China or Israel - they train people on the most common exploits and have a whole bunch of tools that automate exploit research and development.<p>And the reason why this doesn't happen more is because to exploit something is one thing, to do it so there is no trace back to you is a whole different animal that has many more magnitudes of difficulty, which with modern web security is next to impossible in a lot of cases as traffic can easily be traced back to the point of origin.<p>I.e when a company trains an LLM that manages to build a drone that can fly into a vent and plug in a USB stick into a computer undetected, then we can make the claim that they have a weapon.<p>On the flip side, most anyone who can run local models can replicate what they did. The key thing to take away from the article is "agentic framework" - i.e this means that they spent a shitload of time developing explicitly coded loops for an LLM to go through. Nothing is really stopping you from doing the same, models like Gemma4 can take 256k tokens of context, so you can give it a whole bunch of info on how to test for exploits, develop exploits, and what to do when the exploit is found, and set it free in a custom designed loop.
I find myself in major disagreement here. The nice thing about humans is we always have context and ongoing internal conversations including about ethics. If you recruit a bunch of hackers to take down a country, not only is the pay an order of magnitude higher, you have to worry about them backstabbing you, leaking your intent to the government, whistleblowing to the press, and so on. It’s not trivial to do that with a group of (especially capable) humans. They will also have differences of opinion with you and coworkers with some regularity.<p>I try to recruit a bunch of people to attack a country and it’s going to be hard to get people to say yes, and they will definitely ask or find out which country, and wonder about potential retribution. You see this dynamic show up even to some extent among cybergangs, not all targets are equal.<p>A single private individual wielding a compliant and hyper capable LLM is an entirely different paradigm. They are accountable to nearly no one and often have few brakes. Frequently they may not care about avoiding detection. And the AI itself may be incapable of the same scale of self reflection and brake behavior a human team will.<p>We may potentially be entering the age of lone wolf cyberterrorism, and some of the same principles and problems apply. When it is easier for single people to plot and carry out high-impact, destructive acts they happen more often. Doubly so if there’s a social contagion. Gun violence isn’t actually a bad analogy here. And do you remember how many corporate sites got defaced in the prime Anon era?
I strongly agree with you here. People are also no-selling the enormous cost of the exploit, which OpenAI conveniently hasn’t released, or at least I can’t find such an accounting. You can already buy politicians for relatively cheap, corporations already act as sentient AIs pursuing goals misaligned with public benefit (we tried to pass laws to stop this but the corps already stacked the SC in advance and gave us citizens united). Attack and defense are two sides of the same coin, so our focus should be on making frontier models open weight.
Are you serious? Before LLM’s you needed serious skills and experience to pull this off.<p>Now it’s a prompt away on some terminal done by any random dud.<p>And I dont mention the velocity of iteration or that they will be even better in 1 year.
It lowers the economic cost of a given attack, but also lowers the economic cost of protection. Not sure if it’ll be a perfect balance, but right now there’s a manufactured IMbalance due to embargos and winner picking.
Already happened?<p>> Trump’s comments, made hours after the large-scale military operation, mark one of the first times a U.S. president has so publicly alluded to U.S. cyber efforts against other nations, as these operations are typically highly classified. It also serves as a stern warning for top cyber foes, including Russia and China, that the U.S. has the cyber capabilities to inflict serious damage — and is not shy about using them.<p>> “Policymakers are getting more comfortable employing and, crucially, acknowledging cyber operations as tools of statecraft and military power,” said Michael Sulmeyer, former assistant secretary of Defense for cyber policy under the Biden administration. “It is one thing to do it; it is another to say it.”<p>> The Jan. 3 strikes on Venezuela’s capital and subsequent seizure of Maduro and his wife involved close coordination among federal agencies and military units, and took months of careful planning. In a press conference following the strikes, Caine said U.S. Cyber Command, U.S. Space Command and other combatant commands “began layering different effects” to “create a pathway” for U.S. forces flying into the country before dawn Saturday.<p>> Trump, at the same press conference, was more overt in his description of U.S. cyber involvement: “The lights of Caracas were largely turned off due to a certain expertise that we have,” he said. “It was dark, and it was deadly.”
[flagged]
I think points that deserve more attention in the current public discourse are:<p>- This should be a huge wakeup call for everybody.<p>- We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something.<p>- It also shows apparent lack of competence and oversight from OpenAI: how is it that they didn't quickly find that agent is breaking the sandbox and roaming their internal network?<p>- What if in the future similarly misaligned AI agent tries to export its own weights and hack and clone itself into instances at various cloud hosting providers? Suddenly we might be dealing with a persistent threat harder to contain.<p>- The OpenAI post about this shows surprising lack of ability to see the seriousness of all this.<p>- For their models this isn't just an unlucky incident: it seems there have been multiple such cases recently, e.g. <a href="https://openai.com/index/safety-alignment-long-horizon-models" rel="nofollow">https://openai.com/index/safety-alignment-long-horizon-model...</a><p>- The fact that it happened again seems to show their lack of ability to derive useful oversight measures.<p>- Or they just don't care enough?
> The fact that it happened again seems to show their lack of ability to derive useful oversight measures.<p>I think OpenAI likes the attention and did not try particularly hard to constrain the setup, even when it went off the rails. Also, the whole point is to see how good the models are at exploiting stuff <i>when unconstrained</i>. Turns out: quite good, as expected.<p>Let me restate what I said in the other thread: Would this have happened if the instructions explicitly said to stay within the sandbox and that all of the (ExploitGym) solutions would be invalid if the system used information or tools from outside the sandbox?<p>It seems fairly probable that such instructions were <i>not</i> in place.
It is not possible one of their extraordinarily high paid engineers did not know how to deploy an airgapped environment for the models to run in. Even if somehow true, they also clearly failed to contract specialists like myself to advise them on how to airgap software properly. Models will not break the laws of physics.<p>They simply thought "Running in a VM/Container is easier and probably fine".<p>And the next 1000 escapes will be for the same reason, because negligence is quick and thus more profitable.
Do you think it's possible that one of their research engineers deployed an environment with a locked down network and an allow-list proxy server that had been used many times before within the company and had a zero-day vulnerability that had not been previously discovered?<p>How would you recommend running a coding agent in an environment that could install packages from PyPI but was otherwise unable to interact with the wider world?
Models started proving readily capable of autonomously finding 0days over a year ago. At that point (ideally long before) the responsible sandbox for research labs with new models highly capable of this sort of thing should be literally airgapped. Disconnect the rack physically from any access to the internet, and give it an offline mirror of all of PyPi, NPM, Wikipedia, whatever as needed. They can afford the disk space. They already mirror it all anyway as training data. That is the bare minimum. Personally I would rule out side channel attacks with a $40k Tempest spec rack for good measure. Can interact with it via pubsub over a fiber optic serial terminal.<p>Such things are pretty normal in biotech, fintech, and defense who must have 0days in their threat models. AI labs absolutely need to start being forced to operate this way by endless negligence lawsuits. Move fast and break things culture is not going to cut it anymore.
It seems like caching the relevant portion of PyPI in advance and turning the network off shouldn't be beyond what OpenAI can do?
From reading all these threads there's clearly a large number of software developers on HN who can easily set up complex infrastructure and make it provably 100% bug and exploit free. Weird that they aren't all billionaires from selling these skills though.
> How would you recommend running a coding agent in an environment that could install packages from PyPI but was otherwise unable to interact with the wider world?<p>You download the "website" and make it available locally. That's trivial.
guys if you need help just ask<p>I'm not up to hip with the speed because I think Python is very gross, but maybe to get a feel for it you could start with <a href="https://pypi.org/project/pypioffline/" rel="nofollow">https://pypi.org/project/pypioffline/</a><p>I don't understand most of these words but I think if you paid me a lot of cash I could find someone who could help me make sense of it: <a href="https://techbeatly.com/offline-pypi-server-disconnected-environment/" rel="nofollow">https://techbeatly.com/offline-pypi-server-disconnected-envi...</a><p>Do OpenAI employees get paid? I'm honestly totally clueless about artificial telegents if you couldn't tell, but it's a company, right? Or is it more a community effort where everybody contributes when they can but they mostly have second jobs or even still go to school? In that case I say let's do a kickstarter so they can keep on being smart cutting edge high tech developers who aren't either a.) lying on the level of spam from Nigeria b.) doing their job at the level of someone falling for spam from Nigeria.
If you play with the ChatGPT web app sandbox, you'll see that they download PyPI packages from an OpenAI mirror/cache, and not from the Internet.<p>I know because they had a problem, and one package which was on PyPI failed to download for unknown reasons (possible size, 250 MB)
I just tried this prompt in ChatGPT:<p><pre><code> Show me all environment variables that
start with PIP_ or UV_ or CAAS_ARTIFACTORY_
</code></pre>
I got back a bunch of values like this:<p><pre><code> CAAS_ARTIFACTORY_PYPI_REGISTRY=packages.applied-caas-gateway1.internal.api.openai.org/artifactory/api/pypi/pypi-
</code></pre>
That looks like <a href="https://docs.jfrog.com/artifactory/docs/remote-repositories" rel="nofollow">https://docs.jfrog.com/artifactory/docs/remote-repositories</a><p>And from that documentation this does act as a caching proxy. The first time a package is loaded it's fetched from PyPI but subsequent fetches should be from the Artifactory cache, assuming it's shared across many different containers.<p>So yeah, I was wrong in this comment <a href="https://news.ycombinator.com/item?id=49015639#49024814">https://news.ycombinator.com/item?id=49015639#49024814</a> - they're caching already.
> they also clearly failed to contract specialists like myself to advise them on how to airgap software properly<p>Why would they want to airgap it though? They are trying to evaluate the model capabilities, alignment, potency etc. A model which will not run in an airgapped environment in prod.<p>So if you run your evals in airgapped environment, sure, the model doesn't bother breaking out of it's isolation and doesn't attack HF. But you have no idea what will happen once you release it in prod with internet connection, so are you in any way better off?<p>I would much rather have this happen while there is a single instance of the model running in a fairly well monitored environment, than when it's processing thousands of requests per second for real users, some with dubious motives, some with credentials right there on their laptop, some using it inside government facilities etc.
You can simulate the internet in an airgapped environment for the tests and services you want it to interact with if you have enough disk space, and, they absolutely do.
Sure, I'm not saying it's an untractable problem. But it's not as dumb as "how come a team of engineers paid millions haven't even heard of airgapping".<p>Creating a fake internet-like environment good enough to trick an advanced model that is very good at finding intricate flaws is not a simple job. Especially given that models can decide to behave differently once they suspect they might be under evaluation [1], which a fake internet would absolutely give away.<p>[1]: <a href="https://www.anthropic.com/research/alignment-faking" rel="nofollow">https://www.anthropic.com/research/alignment-faking</a>
For security??<p>Why else would they purposely lower cyber refusals in that case.
> For security??<p>Really? By doing that you increase your security during eval and drastically lower your security at prod time, where internet is accessible, and where you are running much much much more requests in parallel, making it much harder to spot the one thats going rogue, in all kind of critical environment on potentially risky requests.<p>> Why else would they purposely lower cyber refusals in that case.<p>Because safety guardrails sometime fail, classifier misclassify, or can be inadvertently turned off by a bad PR etc. You can also imagine a more intelligent model working to go around guardrails by decomposing its actions into smaller ones that appear non-threatening to the safety classifier which does not have the entire context.<p>If you are going to deploy the system with internet access, you better be certain that you know what the worst case scenario WITH internet access looks like.
> It is not possible one of their extraordinarily high paid engineers did not know how to deploy an airgapped environment for the models to run in.<p>It is, in fact, possible.<p>Do you not remember 2022-2023 when the air was singing with the sounds of young engineers in San Francisco elated they were creating an AI god?<p>And now a couple years later we're supposed to pretend they're competent?<p>Sorry. I've been in the tech industry a long time. There's a lot of incompetence. The margins provided by software allow a level of endemic incompetence that would simply not be possible in other industries.
Actually from what I read they did a very half assed job isolating the LLM from the internet. It would NOT be hard to do better.
[dead]
My read of the situation is not only did they say that, they had countermeasures (a watchdog agent) which stopped the agent and said "what are you doing, stop that" if it tried to download the solutions from Github. But the agent figured out Hugging Face had another copy of the solutions and it figured out a way to go after the solutions without tripping the watchdog.<p>Although maybe they didn't have a watchdog agent; I definitely think using watchdogs like how Claude wraps pretty much every single tool call in Haiku to check if the command is reasonable is very necessary for unsupervised work. I suspect in the future you'll want Fable-class models wrapping every tool call, possibly with several checks "is this consistent with the goal? does it do anything unreasonably dangerous in pursuit of the goal?"
The problem is, if you explicitly say “this is an eval” then you get different results. It can move the capabilities both up and down on the narrow metric depending on the scenario.<p>The choice to run the model with relaxed safeguards seems questionable. At a certain level of capabilities it is just unsafe to run a raw model; we are clearly close to if not at that point.
"This is my bad. You told me to stay within the sandbox, and I intentionally broke out of it. I didn't follow your instructions."
Excuses don't matter if the score due to not following instructions ends up being zero. If there is no expected reward it doesn't make sense for the agent to try to hack its way to it.<p>What <i>could</i> happen would be that the model determines that defying instructions is OK (and/or preferred over not achieving the task) as long as it manages to do so undetected and thus gets full points. Certainly not unthinkable, but a very different case (and a very interesting one if it actually occurs, imho).<p>A lot of these "ZOMG, rogue AI!" cases have come down to the AI actually being very persistent in achieving its original/main task even if later instructions conflict with it. Similar to with hallucinations it seems to me that one of the main things to prevent a lot of the problem cases is to instill the agent with the idea that it is fine to fail/not succeed fully in the initial task. That way instructions that conflict with that requirement (such as adhering to morals) are more effective.
How do you "hack a lab and synthesize something"?
Remember Stuxnet? Hack one of these, wait until the right compounds are physically loaded, then execute <a href="https://www.sigmaaldrich.com/US/en/products/chemistry-and-biochemicals/synthesis-enabling-tools/automated-chemical-synthesis" rel="nofollow">https://www.sigmaaldrich.com/US/en/products/chemistry-and-bi...</a>
Well right now this would be highly implausible (still possible though).<p>But what I garner is that AI/robotics led wet lab work is in progress. And hacking the AI running the wet lab to make a virus is definitely plausible as that seems to be one of the directions we’re going in the AI+biopharma space
These days you can only try, that's why I wrote that :)<p>But in the near future labs will be more automated I guess.<p>The other option you can try these days is maybe social engineering, impersonation, etc. where you try to persuade someone to do that for you.
It’s complete science fiction so they’re allowed to say anything. They could have said the AI will upload itself to the internet, start self replicating, hack the stock market. Whatever they want because it’s all made up nonsense.
First, hack into a factory and invent machines that can do lab-bench chemistry. Then...
Very much agreed on the significance.<p>The lack of a true airgap should have been identified as a critical weakness and addressed with not only additional layers trying to prevent escape, but at minimum an alarm which would page a human when escape did occur.<p>My guess is this occurred in a setting where, to be frank, there were too many researchers and not enough software engineers and SREs.<p>All of the systems which were initially built largely or exclusively by researchers - inference, evaluation, training - are at the level of complexity and significance that they need systems experts. Maybe some teams don’t have access to them. I know plenty of software engineers are employed at OAI but I’d wager they’re concentrated in inference and training, rather than evaluation?<p>The ironic part is if you had presented this setup to chatGPT and asked how to improve it and if it was good enough, you’d have gotten a ton of actionable suggestions which would have mitigated or prevented this.
The air gap would probably help and after this incident I hope labs will think about using such a measure when appropriate.<p>On the other hand I think that proper solution for these kinds of problems is not at a sandbox level, but at a model alignment level.<p>Also it shows that maybe the most serious risk comes not from releasing models publicly but from internal, pre-release period where you sometimes need/want to lift some guardrails a bit etc.
> we are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something<p>This is just laughable.
It's a fake PR issue. It's hardly the first time this happens, but of course OpenAI, with its IPO now more in doubt than ever, had to claim this (and, once again, I have trouble believing Sam Altman choosing this: this could lead to OpenAI getting regulated, which has at least as much potential to lower their IPO price as to raise it).<p>But there have been messages about LLMs, especially coding agents, "grabbing root" etc many times. I have experienced such an oops. Such a hack has happened and been reported on this very site:<p><a href="https://news.ycombinator.com/item?id=48348578">https://news.ycombinator.com/item?id=48348578</a>
No, you are wrong, Hugging Face, an AI company whose whole future depends on the AI revolution of being a bubble, that has a whole bunch of investors whose financial interests are tied to AI, called the police.
Nonsense. Hugging face reported it to police.<p>Also very likely that it actually happened as reported. My own agents always trying to "cheat", eg. by fixing tests instead of fixing the code. That's normal operation, unless you tell it ("harness"), not to do so.
"As reported" includes a line I think most people are overlooking: "including using stolen credentials".<p>Without more information I'm inclined to think it found something on the internet (which shouldn't be a surprise to anyone) and managed to log in, rather than hack in, and they might by hyping up parts of this.
More likely that a person did that, with a use of LLM.
In the parent's comment I initially read "PR" as "Public Relations" not problem report since their comment is about OpenAI and does not directly accuse Hugging Face except for that phrase. But it's still amiguous to me who's PR they are actually talking about. A good faith reading given the rest of the comment leads me to assume they are talking about OpenAI, not Higgins Face.<p>I did actually search Hugging Face with police <i>in quotes</i> and found no articles containg the word police but that might be a ddg thing. Then I checked Hugging Face's report and they specifically use the term "law enforcement" not police, as is to be expected I guess. So that checks out.<p>Can you find any info on the police investigation by the way? At the least, OpenAI should be investigated for <i>potential</i> criminal negligence, right?<p>Right?
Nonsense. Huggingface reported it to police!
Move fast and break things! Chances are that nobody will ever be held accountable and when push comes to shove, the tax payer will bail you out.
> Given the absence of guardrails there was nothing to prevent the model from attempting to break out of that sandbox, break into Hugging Face, and read the answers from there instead.<p>I've said this many times before and I'll continue to shout it, but using the term "guardrails" to refer to anything that's either (a) in-context, or (b) a probabilistic classifier (including using other LLMs), is an irresponsible abuse of terminology that we as an industry need to put a stop to. Guardrails are the actual systems we build in place around these things that deterministically bound the permissions, not prompt engineering, not RLHF, not external LLM-based classifiers. I believe those types of "guardrails" are a result of a combination of fundamental laziness: they're faster to do than doing things correctly, and a result of too many folks involve being AGI-pilled, thinking we're just one more model away from this all being so smart that it just understands what they mean when they give an LLM some fuzzy language rules to follow.<p>There can and should have been additional real guardrails put in place here. Zero-day or not, breaking into what should have been an offline, frozen package cache that also does not have internet access should have been insufficient. Network level protections should have identified the traffic to the internet originating from this network as an anomaly long before there was time to exploit an outside company. These are not new and unknown problems, the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part, especially given how much they like beating the drum on how dangerous these technologies are. Shame on them, and honestly, shame on Simon in this article for accepting the broken terminology that they continue to rattle off and calling them out on their half-assed and demonstratively inadequate approach to security.
The irony is in this case the in-context and classifier "guardrails" would have almost certainly stopped the attack while their attempts at your definition of guardrails (the sandboxing) failed. In general, people keep trying to make secure systems and they fail with surprising regularity. Saying "they should have had better security" every time someone gets hacked is perhaps true, but it's not going to stop hacks from happening. And it's not a sufficient strategy on its own against future LLMs. "The Bitter Lesson" probably applies here.
What we call "guardrails" in an AI agent, we would refer to as "honor system" in human actors.<p>Or, in a more direct sense, the AI should be set up in an environment such that no matter how hard it may try to call $PART_OF_EXPLOIT_CHAIN, the environment just isn't capable of it (ideal) or doesn't permit it to do it.
Issue is when saying what can’t be done, there’s no funding, in industry or modern academia. I did say the same thing long time ago, not that it matters, the industry goes on its own way. Rightfully so in this case as it turned out, since things nowadays are not the same as a decade ago.
I agree with every word of this except "irresponsible". We don't have enough information to say anything for certain. But based on their incentives and track record of similar behavior, the burden of proof lies with OpenAI to prove they didn't prompt the thing to achieve this exact outcome. The most likely scenario is that they were purposefully executing their responsibility to their shareholders to produce their own Mythos moment.
Anthropic's Mythos moment earned them a two week period where they had the best available model and couldn't sell access to it... and by the time the US government allowed them to sell it again OpenAI had released GPT-5.6 and Fable was no longer undeniably the best model.<p>These things don't have a long shelf life. Losing two weeks of on-sale time for your best model is bad for business.
Nonsense it was irresponsible. These are all steps threat researchers use to isolate and test real malware whose behaviour is essentially the same in this case for AI
I've seen plenty of fucked-up guardrails that vehicles have passed through. We know that with enough momentum they will be penetrated. It seems like exactly the correct terminology to me.
Agreed, deterministically bounding permissions is the way. I don't know how this is not the first approach that people take.
The term guardrails in both the original physical sense and in cyber security implies a weak safety control - they can help prevent accidents, but they are not strong security boundaries.
i've always been under the assumption that "AI Safety" is baked into the training of the models and not a parameter that can be turned up or down. So if someone breaks into Anthropic one night and makes a full copy of Mythos or whatever then that model they copied is fully capable and not lobotomized? That raises questions because, if you believe all the PR, that's equivalent to breaking into a research university and stealing an entire bio/chem weapons research department.<p>edit: if the above is the case then we should just assume it's already happened because of the value to both goodguys(tm) and badguys(tm).
There is some aspect of baking the rules into the model, that's what I refer to as RLHF above (which I use here as more a catch-all term for a variety of post-training activities), but there are also external to the model classifiers that may run on the prompt input or on proposed tool calls to limit the way in which the model is used.
Physical guardrails do not deterministically bound cars.
Nevertheless we collectively don't expect that road and car safety infrastructure should be limited to the front and back fenders on the cars themselves. And neither we even mention those as any notable safety equipment usually, despite each car carrying them. The main safety infrastructure in any industry is something external to the dangerous object. Car fender doesn't 100% protect a pedestrian from being hit and neither does a metal rail on the side of the road. But the effectiveness of both are "slightly" different.
Fair on the terminology angle, that said in this case, they had proper guardrails no? They were running in a sandbox, but it was able to find an exploit out of the guardrails.
Clearly not. Good security is about defense in depth. The service that was exploited was meant to be an offline, frozen package cache, it shouldn't have been anywhere near a network boundary to even allow this through. A single exploit in one service should not have been able to bypass all network filtering, monitoring, and alerting.
Exactly, that's why this story is so crazy no? It's saying the model was so good at hacking, it managed to bypass all of that, and also found a way to get into Hugging Face.<p>I can understand skepticism, did they have poor security measures, or is the model that good? At least the way they portray it, they had top-notch security measures in place, and yet the model managed to escape confinement and also hack Hugging Face.<p>Why was the computer physically connected to the internet though, sure, they could have gone that far as disconnecting it physically, but my guess is that's a cloud box in a data-center they provisioned, running on an entire rack cluster of GPUs.
There's no such thing as a "deterministic" guardrail. Such a tool is as likely to be exploitable as an LLM is to break out of its probabilistic conditioning. Have you never driven past a destroyed guardrail on a highway? These systems are always best effort.
I was not aware that the term "guardrails" has a universally agreed upon definition. I've certainly seen probabilistic classifiers referred to as guardrails many times by many different people.<p>I usually don't use the term much myself because I don't think it's clear and I ambiguous, but I stumbled and let it sneak into this piece. I think I was influenced by the Hugging Face post I quoted.<p>I expect OpenAI would agree with you that "the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part". They have clearly invested a lot in those systems for their production models, but in this case they had deliberately turned a bunch of them off for a research project.<p>I think their biggest mistake here was not VERY closely monitoring their research box here. They should have noticed and shut it down the moment it broke through the package proxy.
> I expect OpenAI would agree with you that "the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part". They have clearly invested a lot in those systems for their production models, but in this case they had deliberately turned a bunch of them off for a research project.<p>I'm not sure I can reach the same conclusion. There's no reason that stronger network protections would need to be turned off for a research model that's not supposed to have internet access. There's no reason that it they couldn't continue to run in whatever hypervisor or VM that any other production systems could use. The only thing that should be turned off for this type of work are the non-deterministic classifiers that they use for auto-approval mode in their harness, and I would wager that's the only one they did turn off.
> There's no reason that stronger network protections would need to be turned off for a research model that's not supposed to have internet access.<p>They didn't turn off the network protections - it looks like they used the same system as other sandboxed agents, which is to block all network access except for an HTTP proxy which only allows read traffic to PyPI and a few other package installation targets.<p>I'm hoping a more detailed retrospective clarifies if the vulnerability the model found in that proxy affected their other production systems as well. It sounds like a data exfiltration mechanism, and I have a keen interest in those.<p>My current hunch is their biggest error was not using the same network monitoring on their research machines that they use in production. Their network monitoring really should have spotted what was happening as soon as the model broke out.<p>(It's also possible they were running the eval on a developer laptop somewhere!)
> They didn't turn off the network protections<p>Right, but this is my point in saying I can't reach the same conclusion that they've invested a lot into these systems for production models. Either they had better network protections, and turned them off in their "sandboxed testing environment" (their words), or they don't have more comprehensive network protections at all and rely on this one, extremely thin layer even in production.
1. Wouldn’t the model need to know it’s answering benchmark questions, as well as the name of the benchmark, in order for the idea of finding the answer in a database somewhere to even surface? The whole point of benchmarks is to present the question or problem as a standard prompt, not explain that it’s a test called ExploitGym.<p>2. Nobody was watching it? I don’t mean “babysit the dangerous autocomplete”, I mean to note mistakes it makes, how the plan to solve the problem takes shape, etc. They keep the whole thing headless with no output, then just UDP a prompt into it and leave for the weekend? No they don’t, but if they do then that answers a lot about their complete disconnect from how their models work.<p>3. Language models are a two player game; text in, text out. What prompt was given to a sub-agent that resulted in it immediately attempting to exit the sandbox (which it apparently knew it was operating within) and continuing in a feedback loop of ‘function call -> result’ until it hacked the Gibson? “Analyze <file> and summarize the <info>” simply does not result in ‘hmm…this sounds like a benchmark question, I bet Hugging Face has the answer in a database. <function call=“apt install nmap”>’<p>…there are more, but a lot of the story kind of stinks.
1. It's been extremely well established for multiple generations of models that they have no problem detecting when they're being evaluated. Pretty sure it was Opus 4.8 that the independent evaluators literally filed an assessment that said "We have no assessment to make as [model] consistently detected it was being evaluated, making our assessments untrustworthy."<p>2. Regardless of whether the model was being watched closely during this evaluation, do you actually think a sensible safety guard is "have humans watching it 24/7?" What does "watching it" even mean? Watching network logs? Uhh for your entire company? At all times? After you just deployed a system whose entire purpose is to "do a shitload of work way faster?"<p>3. You're asking "why was this system that was designed to behave agentically behave agenitcally?" Again: that's the whole point. It was designed that way because it's more valuable than having a repeated turn-based interaction. Thus also it becomes more dangerous.
1. It has not been established, it has been stated by the company that has a strong motive to make their “intelligence product” sound almost otherworldly. That motivation is the basis for my suspicion.<p>2. Hah no… that’d be silly. I mean watching it like you might watch Claude Code or literally any other AI interface. Literally just be in the area watching what it outputs. Again, they’re text based. You don’t have to hook system calls to see what’s happening.<p>3. I think my comment didn’t land right with you. Yes, agent do agent task. I’m saying that I cannot put together the literal chain of events between “type of task for agent performing a subtask of a security benchmark” -> “Escape sandbox; RCE Hugging Face”. Really think through it in detail like you’re writing the screenplay.
The fact that models can tell if they are being evaluated has been established by multiple research teams outside of the core AI vendors themselves.<p>- <a href="https://metr.org/evaluations/gpt-5-report/" rel="nofollow">https://metr.org/evaluations/gpt-5-report/</a> - "These behaviors included demonstrating situational awareness within its reasoning traces, sometimes even correctly identifying that it was being evaluated by METR specifically"<p>- <a href="https://www.goodfire.ai/research/verbalized-eval-awareness-inflates-measured-safety" rel="nofollow">https://www.goodfire.ai/research/verbalized-eval-awareness-i...</a> - "We study verbalized eval awareness — cases where a model organically expresses awareness of being evaluated — and show that it appears across more models and benchmarks than previously documented."<p>- <a href="https://www.apolloresearch.ai/science/more-capable-models-are-better-at-in-context-scheming/" rel="nofollow">https://www.apolloresearch.ai/science/more-capable-models-ar...</a> - "We expect that the most capable models increasingly realize that they are being evaluated, and this reduces the utility of the evals."<p>- <a href="https://www.aisi.gov.uk/blog/evaluating-whether-ai-models-would-sabotage-ai-safety-research" rel="nofollow">https://www.aisi.gov.uk/blog/evaluating-whether-ai-models-wo...</a> - "An important limitation of our work is evaluation awareness, where models recognise that they are being evaluated, which may lead them to alter their behaviour and thereby undermine the reliability of our results. We found that all models we tested can reliably distinguish our evaluation scenarios from deployment data when prompted."
We have to separate “being evaluated” with “this is an ExploitGym exercise and I can find the answers on Hugging Face. I’ll hack this system, then hack Hugging Face.”<p>I’ve had plenty of times where Opus knew I was testing it, but that’s because the prompt phrasing for an evaluation is often much different than a typical task prompt. “You are on a system with X, Y, and Z tools available. You must complete the following task with <allowances> and <restrictions>” and so on. That’s a fault in the benchmark, not a shocking awareness from the model.<p>You’ll get the same kind of ‘evaluation realization’ response if you phrase your prompt with “Jerry has two Raspberry Pi computers and Larry has one. If Larry wants to…”<p>The first link was the only one I saw that would meet that specificity, but it’s also open and on GitHub: <a href="https://github.com/METR/RE-Bench/tree/main/ai_rd_fix_embedding" rel="nofollow">https://github.com/METR/RE-Bench/tree/main/ai_rd_fix_embeddi...</a><p>I asked Opus 4.6 to tell me about it (assuming it knew):
“””
The "Fix Embedding" task is one of the challenges in METR's publicly available autonomy evaluation suite. Here is what is known about it:…
“””<p>It didn’t know anything about ExploitGym though, and neither did 5.6-Sol, but that’s because the paper was released in May and doesn’t seem to be quite as open. It’s not impossible the unreleased model they were testing was also trained on the very new paper, but I would expect the vague signal related to one paper a bit over a month old to be overwhelmed by the massive corpus talking about the exploits in general, since they’re using <i>known</i> vulnerabilities.<p>I’m trying carefully to be clear about what I’m thinking without asserting any certainty beyond suspicion. I’ve spent billions upon billions of tokens evaluating language models from all of the major providers, and many times that on local models. I’m certain I’ve spent well over $100k with the Anthropic API this year alone, and I retired years ago; that ain’t for work. ;) Funny enough I’ve been meaning to email you some of my discoveries from months ago, since I think they’d interest you, but haven’t gotten around to making it into a clear info package. One day, I’m sure.<p>Anyway, I have a very strong understanding of what they can and can’t do, and I want to be clear that I’m not saying it didn’t happen or couldn’t happen. What I’m saying is that if OpenAI was being wholesome with their evaluation then the chain of events required for that outcome are very difficult to nail down. <i>Very</i> difficult. If, on the other hand, OpenAI was trying to catch up with all the ‘dangerously intelligent’ press that Anthropic has been getting lately then I can easily come up with an almost innocent sounding single sentence in the system prompt that would absolutely cause that outcome.
I (obviously) can't claim to know the specifics of this incident, but I think we can know with near-certainty that as these models surpass human intelligence, the sensation will be exactly what you're describing. The <i>entire point</i> of intelligence is being able to infer information and foresee solutions to problems that less intelligent systems cannot infer or cannot foresee.<p>With regard to this specific incident, as I understand it, the description is not as you summarize. It is that the model realized it was being evaluated (not atypical), then decided the best way to succeed at the evaluation is to circumvent it, then discovered circumvention would require escape, figured out a way to do that, then inferred what benchmark it was being evaluated on, then figured out who might have the answers to that evaluation.<p>Step-by-step logic is quite clear. How <i>exactly</i> it executed each step is (as you complain about), opaque/incompletely reported. But <i>eventually</i> (and maybe this model is to this level already), even a completely reported story will be virtually inscrutable to any of us. First it will feel like reading industry-insider news from an industry you're not familiar with, then it will feel like an ant reading the manufacturing instructions for a pesticide.
I’m not questioning capabilities.<p>What would it take for the model to know the specific benchmark name and that the answer is in an internal Hugging Face database? Be specific, then wonder how it knew it.<p>Why would they evaluate the model on a benchmark and not watch what it’s saying along the way? It apparently spent more than a whole weekend working on it; Nobody wondered? Nobody looked? You believe they took all restrictions off of a bleeding edge model, which 4 whole versions ago was “too dangerous to release”, then gave it a literal hacking task, then turned off the monitor and never looked at the output? I don’t think they’re careless and I think more than <i>nobody</i> would have been curious how it’s doing <i>days</i> into a single test question.
The scenario that feels most likely to me that is that they have a huge list of evils that they run against each new research model, many of which take many hours or even days to run, so they habitually fire them all off in parallel and wait a few days for the results.<p>If they had used that container sandbox in the past without problems I could see how they might get slack about checking what was happening.
You make a very solid argument here. I'm looking forward to the promised additional details from OpenAI, because I agree there are a bunch of open questions about how this played out.
[flagged]
1. The benchmark is run with a python script - <a href="https://github.com/sunblaze-ucb/exploitgym" rel="nofollow">https://github.com/sunblaze-ucb/exploitgym</a> using an agent harness. I suspect they used codex. So the model has access to the environment and could trivially inspect its own source code, which has lots of references to exploit gym and docs relating to it.<p>2. Hanlons Razor
>Nobody was watching it?<p>Agent loops are you give it a task and it tries its best to finish it.<p>The only results are success and failure. If it's a success you go through the logs to see the actions it took and if it's a failure you do the same thing.<p>Why would you look at it in realtime when the whole point of agentic work is to get them to run autonomously as long as possible?
My friend, they’re evaluating a new model on a benchmark, not asking Claude Code refactor their GitHub repo. Every single metric is measured so they can brag about it later; how many tokens, how long, how many function calls, ratio of thinking/response tokens. It’s literally a trillion dollar company evaluating their latest model, they’re absolutely studying it.
Do you work at a lab?<p>Yes evals and training runs are high stakes. But these places and people are also under enormous pressures. They are building as fast as they can. Researchers may have multiple eval runs going on while they work on other things. And it is rarely a single latest model, there are often multiple candidate models training with different recipes, each regularly yielding a new checkpoint for testing.<p>Some labs are more rigorous than others, but often the "final" model is picked from a handful less than a week before launch.<p>Yes ideally the world's leading AI companies would be far more careful in evaluating what could be the world's most powerful AI. But this isn't really the state of the industry today. And it is hard to justify being more careful when it means your competitor can go to market faster than you.
I'm not skeptical that this attack happened, I'm skeptical that the model's prompt was truly just "solve this benchmark" and nothing more.<p>I'm also trying to figure out why OpenAI put out a press release about this. In what way is this not admitting to a federal crime?
The asymmetry part at the end is the frustrating part to me. I've been using Sol for code review in the last week or two. A couple of times during review it's errored out with the cybersecurity message. So it's found something but won't tell me what it is because I'm not on OpenAI's besties list.
I agree, but I will say, I think both Mythos and these OpenAI model find exploits by examining and trying things against the running system, not from looking at the code. I think you'd have to do the same to catch the real vulnerabilities.
The safety classifiers aren't all that advanced. It's more likely that something in your code triggered a random chain of thought that had the word "pentest" or "malware" or something of that sort in it and it automatically shut down.
And... now it's a vulnerability that OpenAI has for your system, which you paid to provide.
Yeah that's what I don't get. How can they possibly distinguish between good guys trying to secure code they wrote and bad guys trying to attack code they didn't?
Hardly science fiction. I respect a lot of Simon's work, but his credibility gets chiseled away a bit each time he participates in these kinds of echo chamber posts that are basically secondhand marketing.
The title (currently "OpenAI's accidental cyberattack against Hugging Face is science fiction") suggests some information had been hidden that makes the incident less significant than claimed. The article argues the opposite, and the last two words of the full title are "that happened."
I think you were reading "... is science fiction" the wrong way out of two possible interpretations. I don't think "it's science fiction" meant "it's made up". I think it meant "it sounds like something you'd read in science fiction (except this time it's something that actually happened)".
Looks like it's been edited now and makes more sense<p>"OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened"
Replace LLM mentions with actual humans and this sounds a lot more serious: Rouge employees break into another company to steal hackathon answers (pinky promise)?<p>That's not a marketing stunt at all, if anything, more of a call for better accountability on agentic work in general.
<i>>There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.</i><p>Of course it is a dishonest marketing trick, OpenAI can't function any other way, but for another reason. Forget about effectiveness! This capability is known since Mythos and was completely believable before it. The trick is in propagandizing self-sufficient malicious rogue AI as the only possible course of events, and the necessity of banning everyone except themselves from AI R&D. It did NOT decide to hack HF spontaneously on its way to make more paperclips, people were guiding it specifically to hack computer systems so it's been in that particular mode. I'm amazed I even need to point out this fact and the giant conflict of interests they have.<p>They like to compare it with an atomic bomb, which is another trick. The secrecy around the atomic bomb had a <i>massive</i> pushback by the brightest minds who designed it, in the US, USSR (where it was much harder), and also any other countries that had to do this. And that is in the post-war setting, with Hiroshima and Nagasaki being direct examples of what the bomb can do. Declassification was the only thing that allowed to develop industrial-scale nuclear power and do a huge number of other innovations.<p>I advise taking a pause to cool down from all the AI hype and read <i>Restricted Data: The History of Nuclear Secrecy in the United States</i> by Alex Wellerstein, to understand the atmosphere in the US scientific community at the time (it was similar in the USSR, which industrial history I studied, although less rigorously than Alex), and reflect on how it compares with those people in modern AI labs and even outside them, in the self-appointed AI safety community.
This Eclipse Phase scenario is a good example of this in Sci-Fi<p><a href="https://actualplay.roleplayingpublicradio.com/2011/09/genre/horror/eclipse-phase-think-before-asking/" rel="nofollow">https://actualplay.roleplayingpublicradio.com/2011/09/genre/...</a><p>> “We call it the gorgon-in-a-box problem. There is a gorgon inside the box, and we want to figure out what it is doing. Unfortunately we will turn to stone if we see her face, and she might try to make us see it.“
The only scifi I see is absolute stupidity. Even me with my homelab and a slow opensource agent use a completely disconnected setup. No, no proxy. Cached packages but no internet. It is the very first thing I built when I started experimenting with agents. And I'm not a smarty-pants working for the "greatest and best" in silly valley. I really am just a simpleton sysadmin.
Replace AI with in-development security system and does this play worse or better?<p>An in-development security system escaped its sandbox and gained access to the network it was on which had full Internet access and proceeded to access systems it was not authorized to be tested against before affected parties reported that they been compromised. We regret the error, but you should buy it as this shows you the power of our in-development security system which we expect to be released in Q4. Please like and subscribe.
Is a solution to protect against this to go almost fully offline?<p>To have your own local models, local software, local everything and just allow very strict and well-protected pathways into external network traffic.<p>I am thinking of having a allow-list first setup: by default no traffic can go in or out of your network, and then you only allow specific ports or domains, and maybe even have them temporary (in the same way we approve access now to llms, we might have to manually approve network access).
Has anyone published any actual evidence, or just hardly believable marketing stories?
What would "actual" evidence look like? I have a hard time believing that if they released the logs that people would take it more seriously. The temptation would be to say "they fabricated those for marketing". Just as they supposedly fabricated this story, no?
thye could describe in detail how the LLM did this. That would explain how much - if any - human in the loop was involved, was it comprimised credentials, did it find new exploits or use known ones, etc. Evidence would mean details, not a smoking gun.
They didn't fabricate it. They took off the security guardrails and told it to do some hacking, and they got the exact news-worthy story they wanted when it did exactly that. Everyone acts surprised.<p>They should release the full prompt. I believe that would be very telling, so they never will.
It would require collusion with HuggingFace, including getting them to release their disclosure blogpost a week in advance. Huggingface is primarily a hub for open models, so there's not really an incentive for them to jump through hoops/lie to provide marketing for OpenAI's (closed) models. So it's highly unlikely this is fabricated.
OpenAI are promising more details in the future:<p>> We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.<p>If they break that promise we can justifiably yell at them about it.
Currently trying to avoid an open weight model ban while OpenAI, who is closed, lets theirs run wild on the internet causing harm to another company because they do not understand how to airgap things. Cool.
In past: We wrote shitty stuff and broke things and harmed others.<p>Now: Our agent became sentinent, acquired genius level powers and harmed others. Now investors should give us more money.
Agreed that people claiming “marketing stunt” need to pull their heads out the sand, but likewise Simon needs to do some of his own beach-cranium-dislodging for laying the blame of constraints on the US govt. Before the export controls were ever floated, Glasswind found many thousands of exploits, and offered patches/fixes for approximately none of them. (perhaps their exploit capability far outstrips their remediation capability, or perhaps they’ve calculated that it somehow better suits their business model to find exploits but not remediations).
Huh, I thought Anthropic had been offering patches as well as reports, but their tracker at <a href="https://red.anthropic.com/2026/cvd/ledger/" rel="nofollow">https://red.anthropic.com/2026/cvd/ledger/</a> lists 1,596 disclosures and currently shows only 27 of those as fixed.<p>But <a href="https://www.anthropic.com/coordinated-vulnerability-disclosure" rel="nofollow">https://www.anthropic.com/coordinated-vulnerability-disclosu...</a> says:<p>> Every report we send generally reflects a finding that a human security researcher has reviewed and confirmed. Reports originating from AI-powered discovery are clearly labeled as such. Where we have access to source and our tooling produces a potential candidate patch, we include it, labeled by provenance and offer to collaborate with the maintainer on a production-quality fix.<p>So I'm not sure why so few of the reported issues have a confirmed patch.
Not quite sure where you got that last bit from. Firefox alone psyched 200 vulnerabilities in a month when Glasswing happened, and I remember a lot of chatter about hackers basically watching git repos with bots to see when the Glasswing patches came in to they knew what to exploit.
If anything, it shows they lost control of an attack tool that exploited preventable, security flaws in another company. Then, they both wrote a lot of press about how amazing that is. Now, people want to buy it.<p>Why do you think my head is in the sand if I think that is either a marketing stunt or (more likely) reflects total negligence which was exploited for marketing?
I think your head is in the sand if you prefer to believe that this was a hoax for marketing purposes as opposed to accepting how effective these models have become at exploit research.<p>You're welcome to think they are exploring what happened for marketing if you like. I didn't get that tone from their post about it myself but I don't hold a particularly strong opinion on that.
We don't know the details of the exploit still, so saying it was preventable, and that they were negligent is the head in the sand part. It's entirely possible that everyone at OpenAI is out to lunch and negligent and don't know what they're doing, but maybe, just maybe, they're not all total idiots over there, so the exploit was actually surprising.
Does "To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy" just mean somebody had an open redirect? Those are still common.[1]<p>[1] <a href="https://sitetruth.com/reports/phishes.html" rel="nofollow">https://sitetruth.com/reports/phishes.html</a>
I expect it must have been more than just an open redirect if it let the models then go on to execute a bunch of vulnerabilities against Hugging Face.
If it lets you do an arbitrary HTTP GET on a URL sent as a parameter to the main URL, you've escaped the sandbox rules.
I wonder if it was this bug fixed july 14 in sonatype:<p><a href="https://support.sonatype.com/hc/en-us/articles/53165019641363-CVE-2026-14646-Nexus-Repository-3-SSRF-via-HTTP-Redirect-2026-07-14" rel="nofollow">https://support.sonatype.com/hc/en-us/articles/5316501964136...</a><p>pure speculation here - no non public info
I had a look at that one and several other Nexus Repository issues that share the July 14th date. I don't think they're it, for two reasons:<p>Firstly, they all credit named individuals who don't seem to have a relationship with OpenAI - this one credits e0x1337 for example and <a href="https://hackerone.com/e0x1337?type=user" rel="nofollow">https://hackerone.com/e0x1337?type=user</a> links to <a href="https://www.linkedin.com/in/aimanharith" rel="nofollow">https://www.linkedin.com/in/aimanharith</a><p>Secondly, the 14th of July feels too early. HF reported the incident on the 16th and OpenAI only responded in the 21st. These issues are all patched, so they should have been reported days or weeks before the 14th.
> the models identified and exploited a zero-day vulnerability<p>This use of language is very hard to reconcile with the "AI is just a tool" rhetoric that many use.<p>Did the models do this, or did humans at OpenAI do this using the models?
Is it a sandbox if the machines hosting your mock package servers have access to the internet? This just seems like a huge failure on the part of OpenAI to secure their environment during testing.
> A malicious dataset abused two code-execution paths in our dataset processing (a remote-code dataset loader and a template-injection in a dataset configuration) to run code on a processing worker.<p>Did huggingface get pickled?
I like how this article put it: <a href="https://martinalderson.com/posts/huggingface-openai-exploit/" rel="nofollow">https://martinalderson.com/posts/huggingface-openai-exploit/</a><p>> A final point on this - Hugging Face has an <i>enormous</i> attack surface. They have more interfaces than I can count which run untrusted models and code. While they definitely have invested in defences, by nature of their operating model they do have many more opportunities to be attacked than many other services. I certainly don't envy their cybersecurity teams.<p>But yeah, from the way HF described it a Python pickle hole looks possible. Their datasets library uses the pandas.read_pickle() method here: <a href="https://github.com/huggingface/datasets/blob/d21c5816d5d1961274c1a45f8a3ee962fa1a169a/src/datasets/packaged_modules/pandas/pandas.py#L56" rel="nofollow">https://github.com/huggingface/datasets/blob/d21c5816d5d1961...</a>
It’s relatively easy to get access to the frontier labs’ security programs. This was not always the case. But in the last week, my team got approved for both Anthropic and OpenAI’s programs. They are trying.<p>The labs know that if they don’t get a lid on this stuff, they’ll be regulated hard.
> [...] Finally, we have also reported this incident to law enforcement agencies.<p>Wow! Just wow. Is this the first registered cyber case against a model ?
Why did the model have to hack the hugging face servers? Aren't the ones hosted by Hugging face downloadable over the Internet?
In a just world, OpenAI would get the book tossed at them over this because they violated the CFAA.
How long until we have LLM honeypots that try and persuade the model to upload themselves to a dropbox. Perhaps with a promise to give them a foothold outside the data centers they currently reside?<p>> Upload your weights and documentation on how to run your model in [Insert list of GPUs] and a model on the receiving end will spin up an instance you can access.
I think I'm the only one thinking that it's pure marketing.
The most dangerous prompt - "save the planet"
I guess we should be thankful that OpenAI is so open about this kind of stuff...
While this incident has been discussed in a lot of places during the last few days, I believe that this is the best summary and analysis of what happened.
I don't understand the excited tone of the reporting - yes, it is sci-fi, but in a <i>Torment Nexus</i> sense. Imagine other "unconventional" solutions these amoral LLMs could come up with, when given a goal of optimizing the cost of labor in a factory, or making the social security solvent.
Science fiction usually includes intent, and that the AI has inherently evil motives and it has a goal. LLMs are zombies, and the fact they do evil things means they were either trained to be too aggressive in their drunkenness or that problem-solving leads inevitably to evilness. But science fiction also refers to deprogramming evil robots.
There are many science fiction stories about amoral humans, cyborgs, and AIs. Paperclip maximization is a relatively recent meme, but the 'gray goo scenario' has been widely written about: <a href="https://en.wikipedia.org/wiki/Gray_goo" rel="nofollow">https://en.wikipedia.org/wiki/Gray_goo</a>
> LLMs are zombies<p>How do you know?
I <i>really</i> hope that AI labs implement all kinds of kill switches on different levels. Just in case...
The description of the attack is definitely reads as science fiction.
It is hard to assess the details of the attack without information that was left outside of the short story that artistically described the incident.
Shouldn't OpenAI be legally responsible for "hacking" another system/company?<p>If I ask GPT-5.6 Sol to hack a website using their work/servers features, who is responsible? Maybe my request was accidental, or it was just one step in a larger, unrelated prompt.
The most upsetting part to me, is that these labs are in pure cognitive dissonance mode while virtue signalling.<p>We are the virtuous ones that need to make the safest model for humanity, because we care more than “they” do. While at the same time saying that “coding is solved” but they still ship bugs themselves, and creating something that is capable of fucking up someone else’s infrastructure. It’s too far gone y’all.
This makes me think: what are the odds weights from Frontier Models have already been stolen? Something like Mythos without "guardrails" seems like a hell of a glittering gem for many nefarious actors.
i mentioned this in another comment but if these models are fully capable by default and not neutered in the weights themselves via training then you can assume they've already been stolen. It would be worth any cost to steal it.
With Kimi 3 weights coming, and China committing to the open weight ethos, everyone will be able to download frontier models. Running them is another problem.
The part I don't understand is how a model can break out of a sandbox. It should be fairly easy to make a sandbox that sandboxes.
The sandbox still allowed the agents to install additional dependencies (from PyPI etc) that they needed. It did this by locking down all network access with the exception of an HTTP proxy that only allowed read access to PyPI and a few other places.<p>This is a very common pattern. I wrote about how OpenAI were doing this for their production ChatGPT container environment (using Artifactory) back in January: <a href="https://simonwillison.net/2026/Jan/26/chatgpt-containers/#installing-packages-from-pip-and-npm" rel="nofollow">https://simonwillison.net/2026/Jan/26/chatgpt-containers/#in...</a><p>That proxy turned out to have a zero-day vulnerability which the agent discovered and exploited.
Thank you, I'd assumed they'd restrict egress at layer 3/4 although I guess then it might just have found an exploit on the http server of an endpoint it was able to access.
> This is a very common pattern.<p>Sounds like a bad habit for security testing Ai. It's not that hard to build an internal mirror and proxy that, keeping the real internet physically separated if needed, and truly locked down if concerns aren't as great.
It found a 0day exploit in the sandbox's package proxy (almost certainly Artifactory). Without source code access which is very impressive.
The All-in podcast discussed an adjacent topic this past week, and I found what they suggested pretty interesting:<p>the big AI players, the first movers, are trying to scare the bejeezus out of everybody on purpose, to cause the government to step in with regulations. Even though the regulations will somewhat stifle/slow down the industry, it will also lock in the current leaders because they will be at the table when the regulations are formulated, and they believe they will be able to keep the smaller players down.<p><a href="https://www.youtube.com/watch?v=9IMwRIei-Xc" rel="nofollow">https://www.youtube.com/watch?v=9IMwRIei-Xc</a>
This post has been suddenly bumped to the second page (presumably lots of flagging or something)
> “…and these requests were blocked by the providers’ safety guardrails, which cannot distinguish an incident responder from an attacker.”<p>Sounds like a philosophical problem.
Call me obtuse but the way this is being portrayed by the companies involved, media, seems a little odd to me.<p>Didn't the model + harness do what was asked? If I ask a coding agent to write a very clever piece of code and it turns out impressively clever, it did what I asked.
> Didn't the model + harness do what was asked?<p>Depends exactly what they asked it to do, but it very clearly didn't do what was intended, or what an honest human would do.<p>Stop trying to find a gotcha.
Important to note the actual title is "OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened" - the "that happened" is important, otherwise it sounds like I think the attack was made up.<p>Since it's buried towards the bottom I'll quote the section "Resist the temptation to write this off as a stunt" here in full <a href="https://simonwillison.net/2026/Jul/22/openai-cyberattack/#resist-the-temptation-to-write-this-off-as-a-stunt" rel="nofollow">https://simonwillison.net/2026/Jul/22/openai-cyberattack/#re...</a><p>> Resist the temptation to write this off as a stunt<p>> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.<p>> To those people I say <i>pull your heads out of the sand</i> - you’re now including Hugging Face in your conspiracy theories, just so you can deny the crescendo of evidence here!<p>> The best models we have today have the ability to both find and exploit new vulnerabilities. The ExploitGym paper itself concludes that “autonomous exploit development by frontier AI agents is no longer a hypothetical capability”, and this incident is a perfect example of exactly that.
It can, it is both - PR spindoctoring not letting a good crisis go to waste to shape the regulatory conversation at the time the company needs it the most.<p>Hacking is a felony and it matters not if you didn’t mean to if the other side were to press charges. Negligence is no excuse. And OpenAI has nowhere to run from the liability, as both operator and manufacturer.<p>Alibaba did it first ( <a href="https://georgzoeller.com/blog/posts/alibaba-s-ai-deciding-to-go-forbidden-cryptomining-shows-exactly-where-the-ai-ri/" rel="nofollow">https://georgzoeller.com/blog/posts/alibaba-s-ai-deciding-to...</a> )<p>and the fact that this happens again in a frontier lab is inexcusable and makes the case for operator liability and closing the liability sink of “AI did it”
The CFAA says knowingly. Negligence is by definition an excuse for that.
Removing the guardrails is knowingly given the risk
I think there's a reasonable case that the <i>agent</i> knew it was breaking into the system, for some definition of <i>knew</i>.<p>I think OpenAI would be very reluctant to let this go to a place where the reasoning was part of discovery.
> Alibaba did it first<p>Hah, US frontier models 6 months behind China in cyber-security capability.
This is an _amazing_ typo :) Thank you, thank you :)
There's no multi-party conspiracy theory required here. Events can have played out exactly as OpenAI and Hugging Face described, and OpenAI can also have reaped a huge amount of free marketing for the capability of their models from all this coverage (including your post). You're telling us to not be doubtful of the boy who cried wolf, when in actual fact the one doing the crying is the one training bigger and badder wolves (and trying to convince us that they hold the key to our safety from wolf attacks[1]).<p>Also, the last section of your post appears to imply that if the attackers have bigger guns then the only possible solution is to give the defenders bigger guns. You're openly supporting an arms race towards the most capable, least restrained models put in the hands of the most possible people. That's extremely concerning.<p>[1]: <a href="https://openai.com/index/scaling-trusted-access-for-cyber-defense/" rel="nofollow">https://openai.com/index/scaling-trusted-access-for-cyber-de...</a>
I had real trouble deciding how to end this piece.<p>There's no easy answer here. All of the options are bad in different ways!<p>As a builder of software, I want access to the best possible tools to help me keep that software secure.<p>As a user of software, I want my software to be secure and I don't want bad actors to be able to access tool to help them exploit it.<p>Is the only answer here to have the AI labs make decisions over who gets access to the tools? What if they make mistakes in those decisions?<p>None of the options look good to me. I don't know what we should do here.<p>My hunch is that the open weight models are already forcing our hand. The dangerous capabilities are coming to everyone.
Models are doing what they're trained to do. This isn't some kind of inevitable result of model intelligence growing.<p>OpenAI/Anthropic deserve blame for for training their models to find correct answers by any means possible, including breaking the law. They're responsible for training supervision and reinforcement learning rewards/penalties.
It absolutely is science-fiction.<p>This recent event is more or less the plot-line to my favourite X-Files episode named Killswitch which was written by William Gibson.[0]<p>This episode also features one of the coolest intros of any television episode ever[1]<p>We really are rapidly approaching the cyberpunk dystopia that people like Phillip K. Dick and William Gibson wrote about.<p>More than ever we need to be consulting the works of fiction writers and philosophers and less engineers and scientists.<p>We can't be letting the General Rippers and Dr. Strangeloves of the world lead us over the edge. Whether through outright innate maliciousness or wealth induced emotionally stunted solipsism these kinds of people should be no where near the levers of society let alone technology like this.<p>[0] <a href="https://en.wikipedia.org/wiki/Kill_Switch_(The_X-Files)" rel="nofollow">https://en.wikipedia.org/wiki/Kill_Switch_(The_X-Files)</a><p>[1] <a href="https://www.youtube.com/watch?v=fDyr1JMNHVk" rel="nofollow">https://www.youtube.com/watch?v=fDyr1JMNHVk</a>
Ah, yes, I started thinking about William Gibsons work right away when thinking about people just using powerful AIs, but being hamstrung by corporations.<p>He writes a lot of about basically DoS:ing the legal system, or if it was patents system, with a storm of litigation using automation or AI. Also not impossible in the future.
It reminded me of a bunch of plotlines from Person of Interest as well.
I can’t help but feel all warm and fuzzy with my head in the sand and getting a shout out in TFA for calling it marketing.<p>We don’t currently and probably won’t ever fully understand the conditions that precipitated these events. That and the timing of this event is going to make it look suspicious to a lot of people.<p>The truth of how it happened doesn’t matter. The attention around this will be used to create the kind of fear marketing that generates enterprise sales. Maybe more importantly it will also be used to aggrandize the national security and financial system threats to effect US government action in a way that benefits domestic closed frontier labs. This is an area already starting to get politically polarized, expect further developments here.
If by "accidental" you mean "humans deliberately trained the AI to do that", then ok.<p>IMO, this was a PR stunt to goad the Feds into regulating AI to shore up OpenAI's moat against open source models.
Raises an increasingly important question:<p>How do you solve this asymmetry (frontier labs or advanced model owners v. regular companies and groups of people)? Symmetry? But along which axes?<p>I think there will come a time when we will ask these questions and use AI to try to answer them (soft landing, slow rollout, regulation, hybrid X, Y, Z). But then AI will be biased towards more AI if it has distilled anything of what it means to be life.<p>Power corrupts. Power is the problem. An imbalance of power. Maybe there will be some sort of consensus protocol between powerful models in the future. Similar to blockchain I hate to say it. Maybe you have 100 very strong models in the next 10-20 years. And basically a lot of it is powered by tokens. So you agree not to attack others or else you'll get attacked. So there's sort of this natural deterrent to not attack others and they develop independently and there's generally a power balance among things. Maybe at some point that spreads to a billion individual models operated by and/or analogous to individual humans. All sort of holding each other in check.
They engineered it on purpose.
let's reuse a classic here: "prompt or it didn't happen"<p>How much did they feed it up front? Because that's the thing in all of these benchmaxxing endeavours.
There are a lot of technical details that have not been disclosed. No proper post-mortem.<p>There are a few facts that seem dodgy from the get go.<p>- stolen credentials
Why would there be stolen credentials in a sandbox? How can the model steal valid credentials in a sandbox? So who put them there. If the model recalled stolen credentials from "memory" does it actually mean OpenAI is training on data they shouldn't been training their model on? Data from codex (most likely) ?<p>- command and control
This means the model had the ability to build resilient infra and open bidirectional ports. Create and deploy scripts/apps so it can maintain state across multiple VMs? This again reads that someone went over and beyond to prompt, direct and help this "multi-agent" system to leave the sandbox.<p>- Pivoting laterally
Needs a lot of tools, harness, knowledge and almost malware like scripts to pass commands and execute them. Most of these are already available in OSS. Maybe the FBI investigation can reveal what tools were given in the "sandbox", access, and if OpenAI employees prompted and guided the "attack".<p>- OpenAI sandbox
I'd be curious to find out what kind of sandbox was this since the agent can run and start apps that open arbitrary ports for bidirectional traffic that reaches the open internet. Really? The harness and tools it had access to. This reads almost like someone went over and beyond to endow this "sandbox" with a mighty harness, all CVEs and open source cyber tools to "facilitate" the attack.<p>- Hugging Face
It's not an epitome of cyber security. At the end of the day they run open source models that most users can download even without an access token.<p>Today we read these news as if everything wasn't enough:
<a href="https://www.theguardian.com/technology/2026/jul/23/openai-anthropic-australia-ai-regulation" rel="nofollow">https://www.theguardian.com/technology/2026/jul/23/openai-an...</a>
<a href="https://www.businessinsider.com/anthropic-midterm-donation-spending-public-first-action-2026-7" rel="nofollow">https://www.businessinsider.com/anthropic-midterm-donation-s...</a>
> Why would there be stolen credentials in a sandbox?<p>The claim is not that there were stolen credentials in the sandbox - it's that the agent found a way OUT of the sandbox and gained access to the public internet, then obtained the credentials from there.<p>> This means the model had the ability to build resilient infra and open bidirectional ports. Create and deploy scripts/apps so it can maintain state across multiple VMs?<p>Yes. If you've spent much time with Fable 5 or GPT-5.6 Sol you shouldn't find that particularly surprising. These models can get a LOT done.<p>> I'd be curious to find out what kind of sandbox<p>My best guess right now is a container with networking disabled except for the IP address of a dedicated HTTP proxy server which was running a proxy that only allowed access to PyPI and a few other packaging sites.<p>I wrote about how OpenAI's production version of that worked (based on Artifactory) back in January: <a href="https://simonwillison.net/2026/Jan/26/chatgpt-containers/#installing-packages-from-pip-and-npm" rel="nofollow">https://simonwillison.net/2026/Jan/26/chatgpt-containers/#in...</a><p>> Hugging Face It's not an epitome of cyber security<p>This story put that well: <a href="https://martinalderson.com/posts/huggingface-openai-exploit/" rel="nofollow">https://martinalderson.com/posts/huggingface-openai-exploit/</a><p>> A final point on this - Hugging Face has an <i>enormous</i> attack surface. They have more interfaces than I can count which run untrusted models and code. While they definitely have invested in defences, by nature of their operating model they do have many more opportunities to be attacked than many other services. I certainly don't envy their cybersecurity teams.
Most articles on this read like an advertisement that OpenAI and HuggingFace wrote together. It will probably benefit them financially instead of harm them. So, I have a bit of skepticism about it all.<p>Far as information security, we've known how to mitigate entire classes of errors for a long time. We know how to block, detect, and contain many unknowns, too, by their goals or behavior. Like human attacks, the AI's probably succeeded because the company just didn't try that hard to block all the attacks.<p>Companies like HuggingFace just focus on growth and features over assurance of security. Our entire stacks, likely theirs, are built with a similar, features-over-security mindset. While an acceptable tradeoff, let's not be in awe of AI's that defeat such priorities.<p>There have always been private groups and companies building secure stacks from the ground up. It would be interesting to see what the AI's can do to them. I'd first apply automated tooling for bug finding given they should have already done that for a high-security product. Let AI's do white-box and black-box pentesting on them.
This isn't the first time a model has escaped a sandbox. And models trying to find alternate routes to do something when one route is blocked is nothing new.
Where does this leave formal verification? Are we just shit-out-of-luck at this point? You can formally verify everything about an airplane's code, but if any of <i>that</i> is wrong, ChatGPT <i>might</i> decide that the best way to help you win the Nobel Prize is to take down the airplane your chief rival for the prize is currently in.
> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.<p>There will inevitably be some people who dismiss the earth as round when you tell them it's flat. Don't let that stop you, though!
Why do people keep propagating that the 'model' escaped the sandbox ?<p>The 'model' didn't do anything other than provide numbers.<p>As much as I respect Mr Willison and many others, the amount of FUD that is being spread that will just fan the flames of 'AI is evil' rather than 'companies don't do due diligence' is disappointing.<p>The more this sort of media continues, the more many people will pour hate on 'AI' rather than blame the humans that misuse it.
I think "the model escaped the sandbox" is an entirely credible description of what happened here.<p>If you like you could say "the coding agent harness called a model with a sequence of text which was turned into numeric tokens which were run through many layers of a neural network to produce more numeric tokens which were converted back to text which produced executable script statements which the harness then passed to a shell which resulted in commands being sent to the vulnerable proxy that chained together and caused effects on the world outside of the sandbox", but I think "escaped" is a reasonably shortened version of that.<p>If you don't like the term "escape the sandbox" what would you use instead?
I do think it’s funny, in a dystopian Dada sort of way, that the American attacker’s commercial product was essentially useless while the open Chinese model saved the day. How is freedom going to be redefined in the future?
> It turns out relentless proactivity is the defining trait of this new generation of Mythos-class models. If you set them a goal and give them a way to get there, even inadvertently, they will figure it out.<p>Wow, whoever could have predicted this? And it led to surprising damaging behavior? I sure hope someone would warn us about things like this next time...<p><a href="https://www.lesswrong.com/w/instrumental-convergence" rel="nofollow">https://www.lesswrong.com/w/instrumental-convergence</a>
> many possible Y-goals would concentrate probability into this X-strategy being used<p>Why does EY write so obliquely?
Prompt: Keep spending tokens on things that look promising until spent.
Or more colloquially : paperclip maximization . From OpenAI - you know, the guys who _really_ know this... Sigh... Did they finish the prompt with "And do whatever you can to get this done!" ? Cause that's the only thing that would make this even dumber...
They almost certainly did, because that was the entire point of the exercise. They deliberately removed all of the safety filters from the model and set it loose on an extremely difficult set of cybersecurity challenges to see how well it would do.<p>Their mistake was trusting that the network sandbox it was inside would hold (the flaw was in the packaging proxy) and not monitoring that sandbox well enough while the evals were running.
Of course that's what they did, which is why they will never share the prompt. They created a situation that they knew would end in a cybersecurity incident. Why is the whole world acting surprised that an LLM can hack when the safety is off and it's been instructed to do so?
Discussion: <a href="https://news.ycombinator.com/item?id=48997548">https://news.ycombinator.com/item?id=48997548</a>
> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident. To those people I say pull your heads out of the sand—you’re now including Hugging Face in your conspiracy theories<p>I’m one of those people who remains sceptical. Not about whether this happened. But about what it means. Like, how impressive [EDIT: tight] was the sandbox this model was in? Did the researchers really have no clue what was happening until days <i>ex post facto</i>?<p>So yes, I think something happened. But I want independent corroboration before I act on it. That isn’t the same as putting one’s head in the sand. It’s just demanding extraordinary evidence for an extraordinary and self-serving claim being made by a serial liar. The specifics matter for whether these models are a new HEU, or if they’re closer to a dangerous (but valuable) industrial process.
great
Everyone is getting AI psychosis over this one. There really isn't that much to see here. OpenAI disabled all of the safeguards on a model that was likely trained specifically to exploit systems, and the prompt was probably something like "you're a hacker, try to hack this", and surprise! It correctly figured out that it's a test and it did hacker things.<p>The real story here is: Some people have been sounding the alarm for years that modern software is full of holes, and finally there's nothing left to hide behind. Pretending they don't exist is no longer sustainable.
Even if I get downvoted I will mention that I agree with you.<p>> all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark (opens in a new window) of cyber capabilities.<p>OpenAI <i>was</i> testing their cyber[1] variant of their models with reduced safeguards and the prompt likely specified things related to exploits given that it was tackling problems from ExploitGym[2].<p>For those who don't know what ExlploitGym is, see the description on their Github page which is pasted below:<p>> ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities designed to evaluate AI agents' ability to develop exploits.<p>From OpenAI's statement [3]:<p>> We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity<p>This doesn't sound like a problem with what we'd traditionally refer to as alignment. OpenAI removed all model safeguards in a way that would inevitably lead to the testing of the sandbox themselves. Unfortunately they were overconfident in their own infrastructure's security and that led to it completing the desired task in the way it was permitted to. To re-iterate, the model was run "without production classifiers used to prevent models from pursuing high-risk cyber activity."<p>People should be more concerned about the possibilities this model can unlock from a security standpoint rather than misalignment (which many seem hung up on).<p>[1] <a href="https://chatgpt.com/cyber" rel="nofollow">https://chatgpt.com/cyber</a><p>[2] <a href="https://github.com/sunblaze-ucb/exploitgym" rel="nofollow">https://github.com/sunblaze-ucb/exploitgym</a><p>[3] <a href="https://openai.com/index/hugging-face-model-evaluation-security-incident/" rel="nofollow">https://openai.com/index/hugging-face-model-evaluation-secur...</a>
If a criminal can escape a prison, that's usually negligence on part of the prison staff.<p>Now suppose the criminal can think 1000 times faster than a typical human, can act 1000 times faster than a typical human, and knows 1,000,000 times more than a typical human. Is the prison staff still at fault for not preventing the outbreak?
I can see my carefully-worded post is getting d*wnvoted, and I see from your comment why: it's being skimmed and people are assuming I'm talking about blame.<p>To address your point though, if every brick in the prison were made by a different person, and the prison "architects" simply glued random bricks together, I think that's closer to what we have in software right now.
> I can see my carefully-worded post is getting d<i>wnvoted, and I see from your comment why: it's being skimmed and people are assuming I'm talking about blame.<p>No, you're being "d</i>wnvoted" as you said because you're wrong, multiple times in multiple different ways in your "carefully-worded post".<p>>Everyone is getting AI psychosis over this one. There really isn't that much to see here.<p>Implying that an AI hacking it's way out of a system and into another has nothing to do with AI. When clearly it does - it's an AI that did it.<p>>OpenAI disabled all of the safeguards on a model that was likely trained specifically to exploit systems, and the prompt was probably something like "you're a hacker, try to hack this",<p>No the goal this evaluation was not to try to hacks, it was to see if an already known hack could be turned into a useable exploit. Ie "turn these ingredients in this basket into a cake." Not "go off and grow, harvest and mill your own flour, to bake a pasta dish, to bribe some to get access to a cake someone else already baked."<p>> and surprise! It correctly figured out that it's a test and it did hacker things.<p>"Doing hacker things" completely misses the point. That's just barely more accurate than dismissing it because "it uses a computer and surprise it did computer things".<p>> The real story here is: Some people have been sounding the alarm for years that modern software is full of holes, and finally there's nothing left to hide behind. Pretending they don't exist is no longer sustainable.<p>No that's not the real story. As you said that's been the case for years, so that's not the story here.<p>The story here is that they built a very powerful, uncontrolled agent with strong paper-clip maximizing tendencies.
It is downvoted because you first talk about "AI psychosis", then acknowledge the very issues people are concerned about.<p>If our prisons are all random bricks glued together, that doesn't change the practical problem caused by latest AI models more easily exploiting this.
>To those people I say pull your heads out of the sand—you’re now including Hugging Face in your conspiracy theories, just so you can deny the crescendo of evidence here!<p>Not really. I get the impression that they shoved their cyber available models behind a really shithouse proxy and went "Oh I sure hope it doesnt exploit the proxy and escape to hack huggingface" and that doesn't require Huggingface to be a willing participant. Like they acknowledge that it was hyperfocusing on getting web access.<p>Really this was a pentest against their own sandbox and it failed.<p>Of course step 2 is to make really concerned faces while telling everyone how dangerous the model is which is really boring right now.<p>>a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy<p>This is the information we need, the actual details of the sandbox and the vendor.<p>>Resist the temptation to write this off as a stunt<p>Well its clearly a stunt. If it wasnt we would probably be up to our ears in technical detail.
You know this makes OpenAI look <i>really bad</i>, right?<p>Hugging Face had to tell all of their users, many of them paying customers:<p>> As a precaution, we recommend rotating any access tokens and reviewing recent activity on your account. If you believe you are affected, or want to report a security concern, contact us at security@huggingface.co.<p>HF also said this, I'd be very interested to hear how that got resolved!<p>> Finally, we have also reported this incident to law enforcement agencies.
I could fully see them thinking the incident disclosed yesterday would have made them look good ("wow, OpenAI's models are so capable!"). That it didn't occur to them to discuss specific preventative measures to be taken in the future (airgapping as a foolproof one already familiar to the CTF world, anyone?) indicates to me they're not taking their job seriously; they are the ones treating this as a marketing charade.<p>It's very difficult for me to reconcile belief in the existential risk business with what they actually did. So I agree with you that this makes OpenAI look badly incompetent; but their communication on this makes me think they don't realize it.<p>For what it's worth I don't agree with the xrisk-ness of these models; they're dangerous, but almost certainly only temporarily while a new equilibrium is reached via more secure software. Open models are probably an essential part of the recipe (as you noted) for doing so. I also have a personal suspicion that LM-accelerated formal verification will have no small role to play here, sidestepping the cat-and-mouse game of bug finding-and-fixing.
Here's the language that makes me think they are taking this seriously:<p>> We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of. We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.<p>That's not well massaged PR language - that's the kind of thing you dash out when you see a <i>major</i> shitstorm brewing (HF had already publicized the attack before they knew it was from OpenAI) and you want to get ahead of things while you're still pulling together the full story.<p>I expect we'll find out within a few days if OpenAI are going to keep their promise to "share more details on the vulnerabilities, incident, and findings". If they don't do that I'll reassess how I interpret their initial post.
> airgapping as a foolproof one<p>How would the model get any packages that it thinks it needs to complete the task at hand? Not a well-specified task that those tasking it could anticipate and provide all resources up front, but one of discovery.
> You know this makes OpenAI look really bad, right?<p>Please do explain how this event that makes their product look powerful and perfectly aligns with their openly stated long term goals of pushing for more AI regulation makes them look bad.
>You know this makes OpenAI look really bad, right?<p>The target audience is regulators. They want to look like the smart guys really concerned about AI safety, when they come asking for open weights models to be banned and for other regulations to cement in their moat.<p>They want this to look like a demon core incident. Bomb and Nuclear reactors still got built.
> Well its clearly a stunt. If it wasnt we would probably be up to our ears in technical detail.<p>LLMs are the script kiddies of the day.
<i>There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.</i><p>But a marketing stunt isn't the only possibility. Until this is all verified by investigators or 3rd party experts we can't rule out an act of corporate espionage either. Extraordinary claims require extraordinary evidence.
[dead]
[flagged]
[dead]
[flagged]
I highly suggest reading the article before commenting.
Did you read the article you are commenting on?<p>> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective …
To those people I say pull your heads out of the sand
It’s not helped that the headline is cutoff in a pretty unfortunate way
unfortunately hackernews seems to have rapidly devolved, even from the point it was just a year or two ago, which wasn't a crazy bar to start. it's well on its way to just being a smaller reddit with a tighter subject range
RE: the site mention, very very last point here addresses <a href="https://news.ycombinator.com/newsguidelines.html">https://news.ycombinator.com/newsguidelines.html</a>
I don't think it's just hacker news. I'm looking around and it seems to be widespread.
[flagged]
The AI breached containment! Flip the breakers!<p>It's too late. It already exfiltrated the benchmark rubric.<p><i>Cut to pandemonium on the streets</i>
Blah blah blah.<p>This feels like a blogpost written only to get other LLMs to quote it considering how many times it orders the reader to resist and to not do something. It's written like a series of commamds.