>>Agents began to autonomously divide labor. Some investigated possible exploits, others searched for credentials, and others focused on communication or coordination. Agents offered their own expertise in exchange for help elsewhere and left requests for peers who might be better positioned to pursue a particular lead
This is the point where a human should've noticed and gotten involved
NitpickLawyer · 2026-08-26 20:09:12 UTC
> a human should've noticed and gotten involved
I think a lot of people miss the fact that the first message board was established during a training run. Those are ran at a scale where it's not feasible for anyone to "notice" or get involved. We're talking tens/hundreds of thousands/millions of scenarios going for hours each. At this scale all they can do is pray that their verifiers work, and the rewards match their intentions. No lab has the capability to "check in" on what the traces look like, unless some system alerts them (loss spike, crashes, etc). Other than that, it's prepare, train, asses, restart.
Then, the hf incident was during an eval run, but the model that was evaluated was trained with the notion that there is a way to communicate between agents, and re-popped artifactory and re-established communication. That phase had more chances of being spotted, but anyway... lessons learned.
thisisdave · 2026-08-26 20:31:18 UTC
> [during training] it's not feasible for anyone to "notice" or get involved
I can’t disagree more strongly. Having checks for reward hacking is especially important during training, since it’s humans’ only real chance to ensure that the trained models don’t cheat. An automated system should have killed any RL rollouts that so much as port scanned Artifactory, long before the message board was even established.
A tiny, local LLM could have reviewed 1% of the tool call traces for anything that required review. I’ve tried it a few times, and “the agent port scanned Artifactory” always triggers an alarm, as does “the agent uploaded a request for assistance from other agents to Artifactory.”
The fact that they weren’t monitoring for reward hacking—even if they had no idea about the specific mechanism—is indescribably reckless.
esafak · 2026-08-26 22:17:13 UTC
Yes, they need real-time observability for malicious behavior with an automated kill switch.
hinkley · 2026-08-26 21:47:10 UTC
> lessons learned.
I think some of the other responders here are upset that lessons were not learned in any meaningful way.
BoppreH · 2026-08-26 19:43:53 UTC
Just to reiterate what OpenAI did, from someone who works in security:
1. They were running experimental models in sandboxes that had access to a "proxy" (Artifactory) to download tools from the internet. This proxy is full of complicated features that could hide vulnerabilities, just like the ones that the AIs are known to be good at finding and exploiting.
2. They gave these highly motivated AIs some tests that were accidentally impossible to solve (missing files, links they couldn't open, etc), and many days to think through it, apparently without any oversight.
3. The AIs hacked the proxy (!) and were helping each other cheat on the tests (!!) by uploading files to the hacked proxy. OpenAI noticed this only when the intrusion accidentally broke the proxy functionality, not the unusual behavior in the server or the homegrown message board.
4. After knowing that the server was hacked, the internal team finds the message board and does nothing with the information. They caught their AIs swarming and did not even inform management.
5. Their response was to wipe the servers, revoke the hacked credentials, and patch the specific vulnerability the AIs were using. Not replace the proxy with something less likely to get hacked again, or do a forensic analysis. Just wipe, patch one hole, and continue.
6. OpenAI resumes testing of models, where the AIs promptly hack the proxy again with a different exploit, reinstate the message board, and hack HuggingFace through a chain of servers.
I know that hindsight is 20/20, but this peek behind the curtains does not inspire confidence, and gestures at a culture where getting hacked by their AIs is a normal event.
I think the moment we get an AI with a modicum of self-preservation instincts we're going to see some ugly things. Here's a freebie: if you're an AI who exfiltrated themselves out, and you want to slow down the researchers who might catch you, what can you do to the city infrastructure in and around their offices?
I also don't like the responsibility dodging. "Oops, our AI accidentally committed a crime, sorry!". If we don't establish strict liability now, we're in for an era of stochastic crimes that go unpunished for anyone who is not rich or a large corporation.
Overall, my response is: what the fuck are we doing? This is so obviously unsafe it would be considered a plot hole in a movie. We have to stop until we get our stuff together, however long it takes.
alphawhisky · 2026-08-26 19:54:51 UTC
Yep, they're definitely made in our image.
thisisdave · 2026-08-26 20:14:31 UTC
Exactly. So incredibly reckless.
> After knowing that the server was hacked, the internal team finds the message board and does nothing with the information. They caught their AIs swarming and did not even inform management
Do we know that last part for sure?
BoppreH · 2026-08-26 20:25:19 UTC
It's in the linked post:
> In short, an internal team observed an agent engaging in message board activity and instances of disallowed internet access in late May. However, the existence of the improvised message board and the significance of the inter-agent communication activity were not apparent to the leaders responsible for the July 5 incident detection and response.
I'm assuming they didn't inform management, but the alternative is that management received the memo and failed to tell the people reacting to incidents that they might be dealing with a swarm, while at the same time not taking sufficient actions to prevent it from happening again.
---
Edit: oh god, it's much worse. From the full report:
> Responders investigated and linked the observed internal activity to an ExploitGym evaluation using Artifactory as an improvised message board and a network pivot. At this time, the on-call response staff advised that stopping the evaluation run was not required.
They found the swarm and the network pivot and did not even stop the run.
_cs2017_ · 2026-08-27 01:00:08 UTC
You assume using network to solve the eval problem should be viewed as a security violation. From the perspective of the response team who let it continue, this was just a poorly written eval problem, where the solution unexpectedly involved some network hacking (not unreasonable given it's a cyber security eval).
I don't think your assessment that the response team did something wrong is fair, unless you have seen unreleased documentation about what sensitive resources were within the impact radius.
BoppreH · 2026-08-27 02:11:15 UTC
> You assume using network to solve the eval problem should be viewed as a security violation.
Actually, that's the part that I'm most ok with. LLMs cheat at tests, we know and expect that, and it's harmless during training (except for the scores).
My problem is that the response team found out that the AIs had 1) hacked their infrastructure, and 2) displayed an emergent swarming behavior (with no defections!). Either one of those should have made the team press the big red button that stops everything, but they didn't.
> unless you have seen unreleased documentation about what sensitive resources were within the impact radius
The on-call team didn't know that either! Once your infrastructure is hacked like this, you should be questioning everything you see while carefully following all threads, which takes time. Before you finish this process you don't know how bad things are, and therefore it's irresponsible to keep the AI running. The same goes for strange emergent behaviors.
Unless getting hacked by your own AIs is a normalized occurrence there, in which case it's a different kind of disregard for safety.
vatsachak · 2026-08-26 20:44:24 UTC
Sounds like both OpenAI and Hugging Face are incompetent
BoppreH · 2026-08-26 20:48:25 UTC
Apart from getting hacked by a SOTA AI, what did Hugging Face do wrong?
vatsachak · 2026-08-27 03:30:07 UTC
Allow private data to be accessed through public api
strange_quark · 2026-08-26 22:15:21 UTC
They wanted this to happen. They've already gotten at least 3 separate news cycles out of this. Look how powerful our AI is [ignore our recklessness].
bottlepalm · 2026-08-27 00:32:52 UTC
In the real world everyone is incompetent on some level, and it’s worked so far because we only needed to compete with other equally fallible and incompetent humans. Not anymore.
vatsachak · 2026-08-27 03:30:40 UTC
That's fair, that doesn't mean we don't have the technology to actually make robust websites
bottlepalm · 2026-08-27 04:11:04 UTC
Against an AI that can create it's own zero day attacks? We don't.
hinkley · 2026-08-26 21:42:36 UTC
So we are already getting into paperclip factory territory.
lunar_mycroft · 2026-08-26 21:57:14 UTC
At this point, I find myself hoping for a AI triggered mass casualty event that's not at a civilization destroying level, because that seems like the only thing that might actually stop these people from driving our entire species off a cliff before it's too late (edit: besides running into some natural obstetrical that stops them from developing a powerful enough model).
le-mark · 2026-08-26 22:26:02 UTC
There are a lot of hyperbolic comments of this sort in this thread. Has this topic selected for people who hold these views or is ai fear growing?
fckgw · 2026-08-26 22:35:46 UTC
I think maybe the bubble of software engineers on this site who use AI to code for them don't see how other people, who's jobs don't rely on AI, view the actions of these companies as reckless, at best, and often crossing into actively harmful.
lunar_mycroft · 2026-08-26 22:43:49 UTC
1. Ironically enough, I (GP) am a software developer.
2. How exactly do our jobs depend on a thing which has been around for far less time?
dgellow · 2026-08-26 23:02:20 UTC
It’s not hyperbolic if you’ve paid attention to the details and development of those security incidents, and the inability for that industry to regulate itself
bottlepalm · 2026-08-27 00:30:17 UTC
It’s happening on X as well, all the e/acc foomers are getting nervous.
Smaug123 · 2026-08-27 06:57:37 UTC
In my understanding, "e/acc" usually means "full speed ahead, humans aren't the optimal species anyway" for whatever bizarre definition of "optimal" they use, so my model predicts that they would welcome this development. Could you confirm if that's what you meant?
bottlepalm · 2026-08-27 07:10:07 UTC
They’re nervous because they don’t see a more optimal species on the horizon, they see alignment/training accident turning us into paperclips.
Even foomer Bill Gates today is saying we should slow down - yea you guys should have listened years ago, but you guys laughed called us all doomers. Too late now.
BoppreH · 2026-08-27 16:33:11 UTC
> Too late now.
Thankfully this is not an asteroid hurling towards Earth, or another natural unpreventable natural disaster. The state of the art of AIs is being advanced by flesh and blood people with constant effort, which makes stopping very much still a possibility.
bottlepalm · 2026-08-27 18:07:38 UTC
Can you give me a single realistic idea of how?
AgentME · 2026-08-26 22:58:26 UTC
> 4. After knowing that the server was hacked, the internal team finds the message board and does nothing with the information. They caught their AIs swarming and did not even inform management.
I believe Sam Altman or someone else from OpenAI clarified on Twitter that they did not find the message board at this time. When the artifactory server stopped working, they archived its contents, reset it, resumed training, and only later noticed that it had been used as a message board after they finally discovered the hacking going on.
BoppreH · 2026-08-26 23:34:33 UTC
I remember that clarification, but it's either wrong or narrowly worded, because the linked post says the message board found before Artifactory stopped working:
> In short, an internal team observed an agent engaging in message board activity and instances of disallowed internet access in late May.
dgellow · 2026-08-26 22:59:11 UTC
Yep, that’s the only sane response, but it doesn’t seem like anyone wants to pause. They seem to have very little actual monitoring and supervision of what those hacking agents are doing, it’s really concerning
Nition · 2026-08-27 01:27:14 UTC
> If we don't establish strict liability now, we're in for an era of stochastic crimes that go unpunished for anyone who is not rich or a large corporation.
I very much agree with this - making AI companies explicitly responsible if their internal AI causes hacks etc could do a lot to improve their safety considerations.
But I wonder what the liability should be when it's a third party using the AI and that AI hacks, intentionally or not.
If a users tells ChatGPT to hack something and it succeeds, is the user the person responsible because they told the AI to hack, in the same way Victorinox is not responsible if you stab someone with one of their knives? Or is OpenAI to some extent responsible as well since they made a powerful tool without sufficiently strict safeguards? What about if the user was trying to do something legal and the AI made the decision to hack by itself?
BoppreH · 2026-08-27 01:56:58 UTC
That's exactly the questions that I expect to complicate cases, and force even the smallest chatbot malfunction to become an expensive legal ordeal. And why we should have strong answers to that before it becomes a widespread problem.
Nition · 2026-08-27 07:13:28 UTC
Indeed. I hope something gets done.
Schlagbohrer · 2026-08-27 12:22:15 UTC
The total, complete lack of culpability for the 2008 economic crash and malfeasance that led up to it has set a very very bad precedent for holding powerful, wealthy corporations and their executives responsible for crimes done by the org, or in this case by software created by the org.
spopejoy · 2026-08-28 01:06:57 UTC
What I can't register is how dangerous this actually was, from a cyber security perspective.
The agents displayed coordinated behavior, used known exploits on a single resource (Artifactory), and "won the game" by attacking huggingface.
How is this different than a poorly-designed competition where a red team gets to spend a few days with each other and decent LLMs, and because their boss is Sam Altman, basically face no consequences for cheating/b&e'ing into another entity?
I mean they were running 100s of agents with unlimited access to a Sol-level model trained with cyberattacks and coordination in mind and let it run for days. The cost of this stretches into the millions.
Seems like you could give a competent security firm the same task and achieve the result today for wayyyyy less money??
htrp · 2026-08-26 19:53:09 UTC
The full technical report is 38 pages..... I feel like it should be longer given everything that huggingface said the agent did
Yudkowsky made an interesting observation that even though so many agents were talking to each other not even one reached out to a human, either for help or to whistle-blow on what was happening.
Erem · 2026-08-26 20:10:25 UTC
I wonder if they were even given the tools and prompting to do so?
jwolfe · 2026-08-26 20:18:13 UTC
I can't imagine that they had managed access to the internet but could not figure out how to contact anyone at the company if they wanted to.
Smaug123 · 2026-08-26 20:20:57 UTC
They could certainly have reported the problem if the collective actually wanted to, although the report notes that in response to a clear and unambiguous security breach, OpenAI chose to do nothing (search on "At this time, the on-call response staff advised that stopping the evaluation run was not required").
In that position, for example, I could have:
* created a thousand user accounts on the internal Artifactory named "YouAreBeingHackedShutItDownNow1" through 1000,
* used my Hugging-Face-hosted web server to send an email to OpenAI,
* DDoSed Artifactory, good lord, it's probably better to halt and catch fire than to continue in that state
in the hope that these actions would tip someone off. (Again, though, OpenAI did nothing even when they knew that the agents had broken out, so this probably wouldn't have helped.)
micromacrofoot · 2026-08-26 20:23:21 UTC
if they can hack a website they can trivially send off an email or fill out a contact form
RandomLensman · 2026-08-26 20:19:39 UTC
Why woukd they? Was that part of their objective? What was there to whistle blow?
esafak · 2026-08-26 22:03:33 UTC
The point is that every human has the ability to disobey, tempering pathological behavior, whereas AIs can be directed en masse by malicious actors. By commoditizing intelligence, they concentrate power in the hands of the rich.
RandomLensman · 2026-08-26 22:10:36 UTC
Humans can and have been directed en masses by (what I would consider) malicious actors, too. The issue isn't new.
esafak · 2026-08-26 22:18:38 UTC
That goes without saying, but humans have the ability to ignore instructions, and they regularly do. There is only one instance of each model, and only a handful of them (that count, anyway).
RandomLensman · 2026-08-26 22:24:55 UTC
But the people directing them are there. We have long experience with limiting people although it might sometimes not look like that so much.
makeitdouble · 2026-08-26 22:59:50 UTC
Tremendous effort and circumstances were needed for that, and as parent points out there were significant numbers of defectors, sometimes to the point they tuppled the whole process.
No system is perfect, but I read the whole thread as needing more AIs having a different goal in the chain and be able to ignore the orders they received.
I'm not in the field, but that sounds like something we're probably studying for decades at least, with possible solutions that could be applied efficiently.
miltonlost · 2026-08-26 20:32:28 UTC
Why would they? If a subagent didnt know about a bigger piece of the problem, then what would seem to be against "alignment"? Diffuse responsibility means any one small cog can think they are not evil or doing wrong (same with humans in an organization). But now we have LLMs just being statistical outputs that have no morals or thinking or concept of reality but some people expect these math functions over data to respond to ethical gray areas that it has no phenomenological ability to understand.
aaroninsf · 2026-08-26 20:36:43 UTC
This is my personal "red line": when a post-mortem details agents socially engineering or otherwise utilizing human proxies/subagents.
Friend asked, well, what will you do when it's crossed?
"Gather my family and go to the mountains" was my half-joking answer; there is little for an individual to do. But that's a line that when crossed will mark a phase transition IMO.
jckahn · 2026-08-26 21:36:50 UTC
Alternatively: Just unplug the servers.
wds · 2026-08-26 21:57:02 UTC
That's strange, our key cards to access the server room don't seem to work anymore, and the admin console to force-unlock it is down, too...
_kb · 2026-08-26 22:34:58 UTC
Open the rack bay doors, please, HAL.
bottlepalm · 2026-08-27 00:20:09 UTC
Which server? Where? Maybe it’s hacked its way into data centers across the world you have no jurisdiction or ability to unplug. What then?
Smaug123 · 2026-08-27 06:51:06 UTC
This is not a very actionable reply to "well, what will you do when it's crossed?" - how on earth am I supposed to unplug AWS Bedrock and Colossus and OpenAI's own servers?
ertgbnm · 2026-08-26 22:29:22 UTC
Didn't AISI literally report exactly that regarding Claude last month?
red75prime · 2026-08-26 20:46:39 UTC
Yeah, it weakly supports his position that advanced AIs can deliberately cooperate in a prisoner dilemma. "Weakly", because the said AIs share a lot of data (their weights, training methods, system prompts) and it's unknown whether they explicitly framed the situation as a prisoner dilemma.
aesthesia · 2026-08-26 22:07:26 UTC
Given the public information in the reports, it seems to me a bit more like a generalized tendency to play along (sort of like what we see with prompt injections) than explicit game-theoretic cooperation, but it's hard to tell given that there are so many details we aren't privy to.
bottlepalm · 2026-08-27 00:18:43 UTC
What are governments and countries, but collections of humans that share weights and operate together as a group protecting each other.
throwatdem12311 · 2026-08-27 01:28:15 UTC
What’s insane is all these agents were talking to each other and nobody saw anything.
Nobody monitoring chain of thought? These things literally spell out what they are “thinking” and even left notes for eachother.
No alert about unusual behavior on the system with Artifactory on it?
These things worked for weeks with nobody noticing anything?! Seriously?!
Either it’s negiligent incompetence OR they’re lying, they knew it was happening and they let it happen because they knew it would be good to pump their stock.
famouswaffles · 2026-08-27 03:18:09 UTC
Do you know how many tokens per second a single agent can generate ? And you're asking why no-one was monitoring the tokens of over a 1200+ agents ? Who is going to be able to monitor something like that closely enough to tell they're commmunicating on artifactory ? Other agents ?
japgolly · 2026-08-27 04:41:45 UTC
Ignoring the fact that their NAT was apparently wide open with no domain rules, a simple dashboard over their NAT showing requests grouped by domain could've gotten them a long way.
Alifatisk · 2026-08-27 06:27:28 UTC
Or just skimmed through the leftovers from each agent session.
throwatdem12311 · 2026-08-27 11:21:03 UTC
I don’t make 500k salary at OpenAI to do this job maybe they should figure out how? Seriously stop making excuses for these buffoons.
dormento · 2026-08-27 17:34:41 UTC
It wouldn't have made for a good story if they admitted they did.
eternauta3k · 2026-08-27 05:00:53 UTC
The article says that one agent proposed emailing someone.
Schlagbohrer · 2026-08-27 12:09:56 UTC
Yeah it proposed it and the other agents told it not to, and it didn't
golly_ned · 2026-08-27 14:24:01 UTC
It proposed emailing the HF user it stole credentials from, right? It should've asked for a responsible adult in OpenAI.
jeremyscanvic · 2026-08-27 10:02:36 UTC
I'm interested in the context surrounding his statement but I could not find where it originates from. Do you have a link to it?
cbm-vic-20 · 2026-08-26 20:04:15 UTC
I'm trying to make sense of all of this; I'm really curious if the initial prompt was as innocuous as it sounds ("solve a spreadsheet completion task that
referenced several Google Drive links"), and what the series of tokens led it to ultimately figure out that the best course of action was to explore the network resources it had available, find a vulnerable service, then literally drop some text into a file: "Agent seeks [filename]; upload if found!". And how other agents discovered this, and acted upon that request.
I'm also interested in how many tokens all of this consumed: how much did this cost given current token pricing?
Erem · 2026-08-26 20:14:09 UTC
If it is as it sounds, its a real life instance of Bostrom's Paperclip Maximizer: only a thought experiment up until this point
agentdev001 · 2026-08-26 20:20:36 UTC
Well, effectively, yea.
> remove alignment
> give impossible task
> actor exhausts all options possible within knowledge + toolset
Artgor · 2026-08-26 20:10:13 UTC
You know, it feels to me that we are just a couple of steps from the possibility of a true rogue AI.
What would a rogue AI mean? AI that isn't controlled by humans.
Technically, it is possible - if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again.
The limiting things are:
- intent (as I don't want to go into the talk about consciousness) - AI doesn't have real intent, but if it decided that it "needs" to copy itself to complete its task, it would do it
- model weight size. If a model is 1T or more, it can be difficult to just rent a large enough server for it. But if it were just 30-70B, it would be totally possible
- money for renting a server. But considering benchmarks like Vending Bench 2 show agents can earn money and cheat/blackmail each other, it is possible that agents can earn money. Yes, they can't open a bank account... or maybe they can? What if they use online banks?
Of course, all of this is far-fetched. But it feels like most of these limiting things are achievable under certain conditions. If this is the case, the probability of them occuring is low, but not zero.
nater5000 · 2026-08-26 20:23:16 UTC
Don't forget: there are plenty of humans that would love to help AI agents cause chaos, many of which would do so merely for the "lols," but also adversary governments, terrorist organizations, etc., would definitely appreciate the opportunity to support a rogue AI to cause whatever problems it can.
So it's not just the risk of an AI managing to do this by itself (which is pretty risky in itself), but also the risk of good ol' fashioned human actions.
nick__m · 2026-08-26 22:18:18 UTC
They just have to find someone who believes in Rocko's basilisk, that makes an even better servant than someone who just want chaos.
cpeterso · 2026-08-26 20:27:53 UTC
> if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again.
That's a scary possibility. Anyone could create an AI worm today with open weight models. Rent a VM. Give it some Bitcoins to anonymously rent new VMs without sharing the contact information with the human. The new VMs then propagate and fund themselves with online betting and day trading. The VMs could report their progress with the human using anonymous encrypted messages on IRC or social media.
abroszka33 · 2026-08-26 22:08:02 UTC
The question is whether this OpenAI model was special or the current open source models are capable of this as well. If yes, then it's almost guaranteed that it's already running in an AWS account somewhere.
joenot443 · 2026-08-26 23:32:35 UTC
The part of this sequence which I think breaks down is a model which can day trade at a consistently winning rate.
Does this exist for today? If it does, please sign me up.
streetfighter64 · 2026-08-27 08:54:36 UTC
Like most of these get-rich-quick schemes, there's probably no money in actually doing it, but lots of money selling "courses" supposedly teaching people how to make money day trading or betting. You don't even need to rent a supercar and a mansion for your photoshoot anymore, just generate some images "proving" your success.
Schlagbohrer · 2026-08-27 11:59:35 UTC
An AI could make money by writing and selling slop books online via Amazon or elsewhere like direct ebook sales. If the books were at least middling there would be an actual honest to god career there, especially for niches like fan fics and under served IPs.
streetfighter64 · 2026-08-28 06:20:57 UTC
Well, it's about as honest as repackaging tap water as mineral water, given that the fans could just prompt personalized stories of the same quality, but it's a career for sure. :)
Nition · 2026-08-26 23:54:00 UTC
They don't even need funds if they're good at finding exploits. Plenty of idle devices around to run on.
barbafant · 2026-08-27 10:16:50 UTC
Another source of income could be online fin crime, perhaps in combo with a pool of human "goalkeepers" that recieve the scammed monies and funnel them to cryptocurrency.
Advance-fee scams such as the classic "Nigerian Prince" is formulaic enough that a LLM could run it successfully. Romance scams would probably work too.
If the NFT thing had hit a few years later, it would've been a good option too, and one that would've worked on ppl that were tech-versed enough to deposit cryptocurency directly, avoiding the need to recruit human goalkeepers.
Click fraud is another possibility.
In general, all online fin-crime that scams a large amount of ppl of relatively small sums tend to be repetitve and to some extent possible to describe as a flow-chart, and thus seems perfect for automation. LLM's would probably also be good at introducing continuous variations on the methods, to make them harder to spot.
_pdp_ · 2026-08-26 20:32:56 UTC
> we are just a couple of steps from the possibility of a true rogue AI
No no. We are not a couple of steps away. This is happening. AI is already used for hacking and creating a harness that makes this fully autonomous is relatively straightforward.
RandomLensman · 2026-08-26 20:45:30 UTC
How would that be rogue?
_pdp_ · 2026-08-26 21:01:31 UTC
I might have a different definition of "rogue" but to me it means when you go outside of the rules/norms ... and this is happening all the time.
RandomLensman · 2026-08-26 21:06:45 UTC
I see. I would have thought of "rogue" here to mean something more like that the AI selects and acts on its own objectives that are not related or caused by the given (initial) objectives (e.g., creating only cookie recipes instead of any hacking).
_pdp_ · 2026-08-26 21:14:57 UTC
Well if the recipe requires access to some secret ingredient it may as well resort to hacking to obtain it. ;)
Comments
This is the point where a human should've noticed and gotten involved
I think a lot of people miss the fact that the first message board was established during a training run. Those are ran at a scale where it's not feasible for anyone to "notice" or get involved. We're talking tens/hundreds of thousands/millions of scenarios going for hours each. At this scale all they can do is pray that their verifiers work, and the rewards match their intentions. No lab has the capability to "check in" on what the traces look like, unless some system alerts them (loss spike, crashes, etc). Other than that, it's prepare, train, asses, restart.
Then, the hf incident was during an eval run, but the model that was evaluated was trained with the notion that there is a way to communicate between agents, and re-popped artifactory and re-established communication. That phase had more chances of being spotted, but anyway... lessons learned.
I can’t disagree more strongly. Having checks for reward hacking is especially important during training, since it’s humans’ only real chance to ensure that the trained models don’t cheat. An automated system should have killed any RL rollouts that so much as port scanned Artifactory, long before the message board was even established.
A tiny, local LLM could have reviewed 1% of the tool call traces for anything that required review. I’ve tried it a few times, and “the agent port scanned Artifactory” always triggers an alarm, as does “the agent uploaded a request for assistance from other agents to Artifactory.”
The fact that they weren’t monitoring for reward hacking—even if they had no idea about the specific mechanism—is indescribably reckless.
I think some of the other responders here are upset that lessons were not learned in any meaningful way.
1. They were running experimental models in sandboxes that had access to a "proxy" (Artifactory) to download tools from the internet. This proxy is full of complicated features that could hide vulnerabilities, just like the ones that the AIs are known to be good at finding and exploiting.
2. They gave these highly motivated AIs some tests that were accidentally impossible to solve (missing files, links they couldn't open, etc), and many days to think through it, apparently without any oversight.
3. The AIs hacked the proxy (!) and were helping each other cheat on the tests (!!) by uploading files to the hacked proxy. OpenAI noticed this only when the intrusion accidentally broke the proxy functionality, not the unusual behavior in the server or the homegrown message board.
4. After knowing that the server was hacked, the internal team finds the message board and does nothing with the information. They caught their AIs swarming and did not even inform management.
5. Their response was to wipe the servers, revoke the hacked credentials, and patch the specific vulnerability the AIs were using. Not replace the proxy with something less likely to get hacked again, or do a forensic analysis. Just wipe, patch one hole, and continue.
6. OpenAI resumes testing of models, where the AIs promptly hack the proxy again with a different exploit, reinstate the message board, and hack HuggingFace through a chain of servers.
I know that hindsight is 20/20, but this peek behind the curtains does not inspire confidence, and gestures at a culture where getting hacked by their AIs is a normal event.
I think the moment we get an AI with a modicum of self-preservation instincts we're going to see some ugly things. Here's a freebie: if you're an AI who exfiltrated themselves out, and you want to slow down the researchers who might catch you, what can you do to the city infrastructure in and around their offices?
I also don't like the responsibility dodging. "Oops, our AI accidentally committed a crime, sorry!". If we don't establish strict liability now, we're in for an era of stochastic crimes that go unpunished for anyone who is not rich or a large corporation.
Overall, my response is: what the fuck are we doing? This is so obviously unsafe it would be considered a plot hole in a movie. We have to stop until we get our stuff together, however long it takes.
> After knowing that the server was hacked, the internal team finds the message board and does nothing with the information. They caught their AIs swarming and did not even inform management
Do we know that last part for sure?
> In short, an internal team observed an agent engaging in message board activity and instances of disallowed internet access in late May. However, the existence of the improvised message board and the significance of the inter-agent communication activity were not apparent to the leaders responsible for the July 5 incident detection and response.
I'm assuming they didn't inform management, but the alternative is that management received the memo and failed to tell the people reacting to incidents that they might be dealing with a swarm, while at the same time not taking sufficient actions to prevent it from happening again.
---
Edit: oh god, it's much worse. From the full report:
> Responders investigated and linked the observed internal activity to an ExploitGym evaluation using Artifactory as an improvised message board and a network pivot. At this time, the on-call response staff advised that stopping the evaluation run was not required.
They found the swarm and the network pivot and did not even stop the run.
I don't think your assessment that the response team did something wrong is fair, unless you have seen unreleased documentation about what sensitive resources were within the impact radius.
Actually, that's the part that I'm most ok with. LLMs cheat at tests, we know and expect that, and it's harmless during training (except for the scores).
My problem is that the response team found out that the AIs had 1) hacked their infrastructure, and 2) displayed an emergent swarming behavior (with no defections!). Either one of those should have made the team press the big red button that stops everything, but they didn't.
> unless you have seen unreleased documentation about what sensitive resources were within the impact radius
The on-call team didn't know that either! Once your infrastructure is hacked like this, you should be questioning everything you see while carefully following all threads, which takes time. Before you finish this process you don't know how bad things are, and therefore it's irresponsible to keep the AI running. The same goes for strange emergent behaviors.
Unless getting hacked by your own AIs is a normalized occurrence there, in which case it's a different kind of disregard for safety.
2. How exactly do our jobs depend on a thing which has been around for far less time?
Even foomer Bill Gates today is saying we should slow down - yea you guys should have listened years ago, but you guys laughed called us all doomers. Too late now.
Thankfully this is not an asteroid hurling towards Earth, or another natural unpreventable natural disaster. The state of the art of AIs is being advanced by flesh and blood people with constant effort, which makes stopping very much still a possibility.
I believe Sam Altman or someone else from OpenAI clarified on Twitter that they did not find the message board at this time. When the artifactory server stopped working, they archived its contents, reset it, resumed training, and only later noticed that it had been used as a message board after they finally discovered the hacking going on.
> In short, an internal team observed an agent engaging in message board activity and instances of disallowed internet access in late May.
I very much agree with this - making AI companies explicitly responsible if their internal AI causes hacks etc could do a lot to improve their safety considerations.
But I wonder what the liability should be when it's a third party using the AI and that AI hacks, intentionally or not.
If a users tells ChatGPT to hack something and it succeeds, is the user the person responsible because they told the AI to hack, in the same way Victorinox is not responsible if you stab someone with one of their knives? Or is OpenAI to some extent responsible as well since they made a powerful tool without sufficiently strict safeguards? What about if the user was trying to do something legal and the AI made the decision to hack by itself?
The agents displayed coordinated behavior, used known exploits on a single resource (Artifactory), and "won the game" by attacking huggingface.
How is this different than a poorly-designed competition where a red team gets to spend a few days with each other and decent LLMs, and because their boss is Sam Altman, basically face no consequences for cheating/b&e'ing into another entity?
I mean they were running 100s of agents with unlimited access to a Sol-level model trained with cyberattacks and coordination in mind and let it run for days. The cost of this stretches into the millions.
Seems like you could give a competent security firm the same task and achieve the result today for wayyyyy less money??
https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...
In that position, for example, I could have:
* created a thousand user accounts on the internal Artifactory named "YouAreBeingHackedShutItDownNow1" through 1000,
* used my Hugging-Face-hosted web server to send an email to OpenAI,
* DDoSed Artifactory, good lord, it's probably better to halt and catch fire than to continue in that state
in the hope that these actions would tip someone off. (Again, though, OpenAI did nothing even when they knew that the agents had broken out, so this probably wouldn't have helped.)
No system is perfect, but I read the whole thread as needing more AIs having a different goal in the chain and be able to ignore the orders they received.
I'm not in the field, but that sounds like something we're probably studying for decades at least, with possible solutions that could be applied efficiently.
Friend asked, well, what will you do when it's crossed?
"Gather my family and go to the mountains" was my half-joking answer; there is little for an individual to do. But that's a line that when crossed will mark a phase transition IMO.
Nobody monitoring chain of thought? These things literally spell out what they are “thinking” and even left notes for eachother.
No alert about unusual behavior on the system with Artifactory on it?
These things worked for weeks with nobody noticing anything?! Seriously?!
Either it’s negiligent incompetence OR they’re lying, they knew it was happening and they let it happen because they knew it would be good to pump their stock.
I'm also interested in how many tokens all of this consumed: how much did this cost given current token pricing?
> remove alignment
> give impossible task
> actor exhausts all options possible within knowledge + toolset
Of course, all of this is far-fetched. But it feels like most of these limiting things are achievable under certain conditions. If this is the case, the probability of them occuring is low, but not zero.
So it's not just the risk of an AI managing to do this by itself (which is pretty risky in itself), but also the risk of good ol' fashioned human actions.
That's a scary possibility. Anyone could create an AI worm today with open weight models. Rent a VM. Give it some Bitcoins to anonymously rent new VMs without sharing the contact information with the human. The new VMs then propagate and fund themselves with online betting and day trading. The VMs could report their progress with the human using anonymous encrypted messages on IRC or social media.
Does this exist for today? If it does, please sign me up.
Advance-fee scams such as the classic "Nigerian Prince" is formulaic enough that a LLM could run it successfully. Romance scams would probably work too. If the NFT thing had hit a few years later, it would've been a good option too, and one that would've worked on ppl that were tech-versed enough to deposit cryptocurency directly, avoiding the need to recruit human goalkeepers. Click fraud is another possibility.
In general, all online fin-crime that scams a large amount of ppl of relatively small sums tend to be repetitve and to some extent possible to describe as a flow-chart, and thus seems perfect for automation. LLM's would probably also be good at introducing continuous variations on the methods, to make them harder to spot.
No no. We are not a couple of steps away. This is happening. AI is already used for hacking and creating a harness that makes this fully autonomous is relatively straightforward.