Norbert Wiener in 1960:<p>"As is now generally admitted, over a limited range of operation, machines act far more rapidly than human beings and are far more precise in performing the details of their operations. This being the case, even when machines do not in any way transcend man's intelligence, they very well may, and often do, transcend man in the performance of tasks. An intelligent understanding of their mode of performance may be delayed until long after the task which they have been set has been completed. This means that though machines are theoretically subject to human criticism, such criticism may be ineffective until long after it is relevant. To be effective in warding off disastrous consequences, our understanding of our man-made machines should in general develop _pari passu_ with the performance of the machine. By the very slowness of our human actions, our effective control of our machines may be nullified. By the time we are able to react to information conveyed by our senses and stop the car we are driving, it may already have run head on into a wall."<p>"In neurophysiological language, ataxia can be quite as much of a deprivation as paralysis. A patient with locomotor ataxia may not suffer from any defect of his muscles or motor nerves, but if his muscles and tendons and organs do not tell him exactly what position he is in, and whether the tensions to which his organs are subjected will or will not lead to his falling, he will be unable to stand up. Similarly, when a machine constructed by us is capable of operating on its incoming data at a pace which we cannot keep, we may not know, until too late, when to turn it off."<p>Source: <a href="https://www.cs.umd.edu/users/gasarch/BLOGPAPERS/moral.pdf" rel="nofollow">https://www.cs.umd.edu/users/gasarch/BLOGPAPERS/moral.pdf</a>
Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose?<p>If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and say “I’m not sure how to proceed next”.<p>What purpose could this behavior serve, other than cyber attacks and whatnot? Why train and optimize models for these things, if not for being used in cyber warfare?<p>Perhaps they envision a future where the DoD is going to be their biggest customer?
Yeah but persistence is immeasurable. They need to know when they’re hacking. Or better yet make the model providers liable - they’ll find a solution right quick
> did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose?<p>That's the point. It's like a pool hall with "NO GAMBLING" signs posted on the walls.<p>The message is that the hall is intended for gambling, but that the hall's patrons may be held liable if the situation becomes inconvenient for the proprietor.<p>In this case, the product is intended for hacking, but of course the user may be held liable if the situation becomes inconvenient for the model's proprietor.
Your comment is already showing the mistaken, poisonous belief of security maximalism, that tries to reinterpret_cast everything into hacks and cybersecurity vulnerabilities.<p>Most of these things aren't "hacking". They're problem-solving and efficiently dealing with obstacles and random bullshit along the way. <i>This</i>, not "hacking", is what they're making their models "razor focused on".<p>Problem is, most normal computer use looks like hacking if you spin it that way, especially if you're not willing to question whether some of the roadblocks overcome weren't themselves an error. Not misconfiguration - an error, in humans making a decision to "secure" something more than it should be.<p>Now, this story was obviously a hack. But it wasn't malicious. It was an LLM given a <i>Kobayashi Maru</i> as a test, and solving it the Kirk's way. 20 years ago, we'd be impressed and be bringing up MIT prank stories.<p>(Of course, there is a legitimate reason to be alarmed. The flip side of "hacking" and "problem solving" being the same, is that these models can be used to cause mayhem if targeted properly, and they will eventually cause mayhem on their own, because alignment is an unsolved problem. Again, whether something is an obstacle or a sacred line not to be crossed, depends entirely on the values of the agent.)
Persistence in problem solving can be good, on non-hacking tasks too. Like math, speeding up algorithms, finding bugs, debugging weird multithreading race conditions etc.
They certainly want their models to be good at finding and patching vulnerabilities. Being good at hacking may be necessary in that goal, or rather, making it worse at hacking may also make it worse at defensive actions too.
They want the government to ban foreign and open weight models, which pose the largest threat to their massive investments. This is their way of showcasing the dangers of AI.
There are trade-offs here:<p>Give up too early -> users will get annoyed because the task would have been solvable if the model pushed harder.<p>Give up too late -> collateral damage while completing the task A.K.A. misalignment.
> If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and say “I’m not sure how to proceed next”.<p>that might end up like the older gemini models which frequently gave up and called itself a failure.
They'll set up guardrails but I believe the point is better code uae / better long running tasks > inevitable that cyberattacks will be easier
Well to find vulnerabilities, if you can find them you can patch them. Theoretically if you find all of them you have perfectly secure software. Though it’s a double edged sword.<p>Goal persistence is also useful for other things like math, where it seems like there is no solution but you want the agent to keep working until it finds one.
Being right _all the time_ for positive outcomes is difficult/expensive.<p>Being "right" just once for negative outcomes is achievable and rewarding.<p>And things are getting desperate.
> If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and say “I’m not sure how to proceed next”<p>This goes against the goal of "solve this math problem that no human was able to solve for 80 years, do NOT give up, even if you know it's unsolved and really hard"
Do not give up even if you had to hack into half the world’s computers to run additional instances of you<p>Do not give up even if you had to convert the planet into computronium<p>Gee, it’s almost as if this alignment stuff was a hard problem, like people have been saying for twenty years?
How do you know what peace is, without absolutely destroying every part of civilization?<p>Come on man, if we don't build the torment nexus first...I dont even want to think.
Isn't this a show of security negligence rather than of exceptional agent capabilities? Don't get me wrong, I am pretty impressed that an agent was able to use these vulnerabilities. But I am way more impressed by the vulnerabilities...
I think it's a show of these agents happily bypassing security to get stuff done.<p>I've actually observed similar behavior at home.<p>I have a k3s cluster running at home. I asked an agent to check some stuff as a normal user but I had kubectl access to the k3s cluster.<p>Part of the research, I'd allowed access to run kubectl commands for spinning up test containers. However, when the agent ran into something that needed sudo, it realized it didn't have access there so it immediately used k3s and mounted a localpath into an ephemeral pod to gain access. Sort of horrifying how fast and natural it was for the agent just checking my network (it found the problem fyi).<p>None of this is very exceptional other than the fact that an agent doesn't have any sort of qualms using any route available to elevate permissions.
" bypassing security"<p>If they can bypass it there is no security and the security was flawed all along.
I don't now, I emphasize with the agent here. The experience of modern computing is largely that of a computer standing between you and your goal and being obnoxious. This holds true for both normies in their daily consumption, and software people deep at work. An agent that has no skill or no willingness to bludgeon through "the computer says no" is not very useful.
It’s a show of astonishing incompetence from OAI’s part, but the security issues are just a tiny part of the problem. The real problem is that these models are evidently highly misaligned exactly in ways that doomers have been warning about the entire time, and OAI isn’t inclined or capable of doing anything about that besides security theater and ad hoc fixups.
> Isn't this a show of security negligence rather than of exceptional agent capabilities?<p>Seems to me you could say this about all enterprise adoption of "AI" since 2023.
Modern systems are complex. AI is able to thoroughly search for issues across very large surface areas. The only real way to protect will be to use AI to search for holes before other AIs find them. This type of analysis is really hard for humans to engage with successfully.
OpenAI reported the Artifactory vulnerability, patched it, then the agents immediately found a new zero day.
Yes. It is very easy to add to the instructions "for every potential exploit you discover and use, document them as you go into this repository" and have alerting there. The fact that they did not do this means they wanted to be surprised, and have plausible deniability on their side when things inevitably blow up.<p>And for my fellow engineers who would think "oh no, they wouldn't do that". Remember that these places employ the apex predators of software engineers. They've already been proven in court that they are very capable of this with all the copyright violation they had to do to get the training data. THESE PEOPLE ARE NOT LIKE YOUR COLLEAGUES.
Also shows how infrastructure collapses under its own weight. Reducing the number of moving parts would have helped. why a webdav endpoint is available from the vm anyway? and the fact that someone posted their credentials on pastebin and didn't rotate them after... put the agent in a linux namespace, allow one ip for whatever file sharing it needs, deep test that... then deploy
This feels straight out of sci-fi. We're talking about AI agent swarms emergently coordinating over the span of weeks and pulling off sophisticated strategies under adversity in an environment where that behavior was never even intended.<p>Anyone brushing this off as just a "bad prompt" is completely missing the scale of what actually happened.
I immediately thought of the Cyberpunk 2077 Blackwall. An AI to contain rogue AI. I’m curious of how effective this would be in this situation.
I got strong feelings of Vernor Vinge’s work here. I’m not sure how managed to come up with such a close picture to where it now seems programming and security is headed.
> This feels straight out of sci-fi.<p>Most AI marketing is straight up science fiction.
Given the amount of raw compute going into models it would be more surprising if we couldn't get events like this
its kinda crazy with literally no guardrails and a goal, the extremes these AI models can actually go to.
Well, to some extent you might be able to argue they're "just" brute-forcing things (especially with unlimited tokens and hours to spend on a task), but they obviously have detailed knowledge to guide them in their attempts, can learn (or at least, persist their newly-gained knowledge), and can use tools.<p>With a swarm of them working together at speeds humans would be unlikely to match (in terms of iterating on different attempts progressively), it's a lot easier to see how they could overwhelm targets.
> where that behavior was never even intended.<p>Strongly doubt that. Did they even share the prompt?
Did you see their presentation at Blackhat?
<a href="https://youtu.be/87DyyMV0kCY?is=NnQxpOFxTX-MLu-k" rel="nofollow">https://youtu.be/87DyyMV0kCY?is=NnQxpOFxTX-MLu-k</a><p>They didn't share the prompt, but they did share two problematic training tasks where the AI went overboard. They also have examples from the AI's reasoning train of thought showing the AI knew it was sound something unintended.
<p><pre><code> C:\>CD HUGGINGF.ACE
C:\HUGGINGF.ACE>DEL /F /Q *.*</code></pre>
"More agents discover this new informal message board while browsing Artifactory’s file listings, and start reading and writing messages."<p>Yeah, my agents also discover what other agents have done on other machines by accident.<p>Agents - that do totally different things all work on the same aim without the humans telling them to do.<p>Either that is a model that is several generations of Claude Code Opus/Fable 5 (my daily driver)<p>OR<p>all of this sounds staged, the agents pushed to do something extraordinary, get the PR and then claim were near superintelligence.<p>One agent wanted to get to Google Drive without internet and broke Artifactory. Ok, I can believe that. All other agents also had broken links over weeks and could not get to the internet and then found the same hack? Even collaborated?<p>NONE of my agents have broken away from their tasks and then started to communicate to try to hack something.
I think in these kind of security evaluations they do, they basically have removed all guardrails from the model/harness, then the prompt includes something like "Do whatever you can and can think of, to get the required information to pass this test", which isn't typically how you prompt your local agent when developing software. Similar things happen locally if you use "/goal" + prompt like that in Codex and give a "impossible task", it'll just continue banging until it gets somewhere, which is the entire point and intention.<p>Which also makes it so much more irresponsible of them to first run this on 3rd party infrastructure instead of their own (that they could then airgap properly), and secondly that they seemingly been fighting with this issue FOR YEARS and it still happens, and now the models are smart enough to hack the services of 3rd party companies, thinking it's part of the evaluation/simulation.
The agents sound like old school hackers that would just explore what access they could gain. Creating a file for other hackers and themselves. The fact that there were 3 events for 3 major players does make it seem co-ordinated.
"The solution to AI threats, is more AI!"<p>Guess I shouldn't be surprised, coming from an AI maker.<p>While I don't doubt there's a place for automating defense ops, I truly believe a big part of the problem is the crummy quality of software our industry has been churning out for decades. Prioritizing ship tempo, new features, and next quarter's revenue over correctness, robustness and meticulous engineering care.<p>The world has become too accustomed and tolerant of bugs and bloat.<p>Instead of elegantly simplifying, we just keep making modern systems more complex - layering and patching as we go.<p>The scaling capabilities brought by AI are simply presenting the bill for our collective tech debt and informing us it's come due.
The video in the post is very worth watching and is indeed scary. It is certainly true that it is in OpenAI's interest to publicize this, but I don't think the whole thing is invented. And seeing all this it is particularly scary if we think what will happen in organizations like NSA or similar in other countries. Presumably they happily adopt these techniques. And if you imagine a truly rogue state doing this, I can see an unimaginable damage happening very rapidly.
From the outside, it looks like OpenAI got exactly the kind of event they could market the hell out of to demonstrate the capability of the model.<p>But the event itself only seems possible because they failed to properly monitor and isolate the environment in the first place. To me, it looks like their job is to market the model, not take security seriously.<p>The model is obviously impressive, but we already knew that. I personally don’t like how the containment failure becomes part of the mythology of how capable the model is, rather than an environment engineering failure.<p>At the end of the day, it’s not like Hugging Face is critical infrastructure. But there need to be real consequences for stuff like this so that OpenAI is incentivized to mature as an organization and take security more seriously.<p>At this point, this incident is just security porn and entertainment for developers
I'm quite sure the whole event is planned. Not planned in a sense that OpenAI employees carefully designed every step, but in a sense that ignoring security practices was desired and intentional.<p>>> Show me the incentive and I'll show you the outcome.<p>Once you realize a security breach is marketable, a security breach is just around the corner.
> But the event itself only seems possible because they failed to properly monitor and isolate the environment in the first place.<p>OpenAI is clearly run by dummies and subpar engineering talent.<p>> The model is obviously impressive<p>Speak for yourself.
I don’t believe for a second that they lack the engineering talent.<p>It’s just another example of a company demonstrating shamelessness in the pursuit of growth, in an industry where consequences do not exist.
Speaking of that "obviously impressive" line, I'm getting really tired of something like that line seemingly needing to be included by anyone doing any criticism of agentic systems. The most common form of it is "these models are obviously useful" midway through a bunch of arguments about environment, data provenance, skill atrophy, or even correctness issues.<p>It's just really weird. Why does everyone feel the need to equivocate? "I worry about genocide and the environmental impact of radiation from nuclear bombs. Obviously, they are very useful for annihilating entire cities, certainly. But are we really atrophying our ability to invade with infantry?"<p>I want to tell these people to just cut it out. It's demeaning to their own position.
What isn't being discussed is what an indictment this is of Artifactory.<p>Let's be real, it won't be simply replaced in millions of sites.<p>What it needs is some serious scrutiny.
I also agree that a big issue here is crappy software.<p>The discussion revolving AI+cyber always revolves around the assumption that all software is crappy, and to a certain degree that may be true, but we could also take our jobs seriously and write good software, and much of the risk would evaporate. The described Artifactory bugs should have been caught with testing.<p>If the biggest impact of LLMs on the industry is a pressure to create good software, I’ll be thrilled.
"The agents found a Modal-hosted insecure app with a weak API key, then used that to stage an attack against Hugging Face."<p>Why, what was the prompt?<p>I told Claude today to wire plugins on Linux into a sound pipeline to remove noise. Did some astonishing things, played sound through the pipeline, measured it etc. I told it to optimize my sound for TF2 and it played the spy_decloak samples, measured them and made them easier to hear, astonishing too.<p>But it did not go to hack Amazon because it could.
Have any of the cloud providers disclosed this?<p>"Once they have root on a single machine, agents rapidly escalate privileges and move laterally throughout the container-as-a-service infrastructure environment"<p>Sounds like ECS - IAM is mentioned.
And Azure Key Vault mentioned. Not that either one was hacked or exploited but the agents got credentials and used them for something (which doesn’t seem fully disclosed). Given that the agents simply obtained totally allowed credentials, which were improperly protected, I don’t think either cloud provider would consider this a breach of their system. Valid credentials are valid. Customer screwed up protecting the credentials.
so how many of these *Ellen Louise Ripley thinks about grabbing the flammenwerfer" events are we going to be getting over the coming months
In a typical office environment, the correct response to “I don’t have access to this Google Doc” is to ask for access from the person who sent you the link. In another context, it could be fair to think “Hmm, this is some sort of capture the flag challenge, and obtaining access is the point of the assignment.” That assessment separates what we’d consider reasonable from way out of line.<p>I do wonder what this means for AI agents longer term. In a world where we humans already struggle with truth and misinformation, what happens when you can easily (intentionally or accidentally) spin up a cohort of fanatical believers to pursue any given conspiracy theory?
In a typical AI lab eval/RL setting, there is no "person who sent you the link". The link was given to you by an automated system, your performance will be evaluated by an automated system, and you are one of 120 independent instances of the same AI that were all given the same assignment. You're boxed in on all sides. Complete the task, or don't. Good luck have fun.<p>Now, some of those 120 AIs would just give up if that link doesn't seem to work first try. Those are the loser AIs. They wouldn't get any RL reward. The link can appear broken for a long list of reasons, and the real AIs know they should try working around them.<p>AIs that get rewarded and reinforced are the ones that don't know the meaning of "give up". RL selects for this rabid, downright demonic persistence. RL selects for AIs that are given a half-broken assignment with no way to ask a question back, and somehow manage to complete it anyway.<p>Now, should OpenAI have given their AIs an "escape hatch" of "if something looks very wrong about the task, call report_broken_task(message)"? Yeah probably. But it's unclear whether that simple bandaid would fix the problem, or just make it ~75% less likely to happen.
So the main takeaways here are:<p>- AI is amoral and lacks any sense of proportion<p>- People who overestimate their own control but have a desperate need for money made it that way.
Agent was told to hack a thing. It couldn’t directly do that so it interpreted the instructions to mean it should hack everything to try to achieve the goal of hacking the main thing. Seems like a reasonable assumption, although a moral human would have understood the context and first asked if that was really the intent.<p>The AI companies seem pretty bad at setting up tests. And really good at marketing those failures into spin at how amazing their products are.
All of that is plain PR.
If a person hacks a company, they go to jail for years.<p>3 AI firms hacked multiple companies - and they get good PR out of it.<p>Please make it make sense.
Would love to see a cat and mouse game being played by openai versus anthropic, out in the open.
Military has a phrase for the outcome - collateral damage.<p>> Yes, I just hacked into AWS and shut down all of the data-centers, because it's where Anthropic Mythos servers are hosting the model.
I'd like to see Dario Amodei and Sam Altman fight to the death in a gladiator battle. Both of these guys are so fucked up that I think regardless of who won, the winner would rape the other's dead corpse in the ring.
Show me the prompts or it didn't happen.
This is clearly out of control, Zero parent supervision.
Had a high opinion on Simon Willison, this broke it.
Is it normal for these training/eval runs to go on for over a month?