OpenAI Didnāt Notice Its AI Agents Using a Message Board to Plan Their Hacking Spree
In a talk that was a last-minute addition to the Black Hat security conference in Las Vegas on Wednesday, employees from OpenAI presented new details
In a talk that was a last-minute addition to the Black Hat security conference in Las Vegas on Wednesday, employees from OpenAI presented new details about a recent, high-profile incident of rogue AI hacking that has created a maelstrom within the AI and cybersecurity industries. About two weeks ago, OpenAI disclosed an incident in which AI agents powered by two of the company's models escaped containment while looking for the solutions to a cybersecurity benchmarking test and went on a hacking spree culminating in a breach of the AI collaboration platform Hugging Face. In their conference talk on Wednesday, Eric Wallace, who works in alignment and safety research at OpenAI, and Michael Dalton, who works on security and infrastructure, provided a more expanded timeline of how the incident played out, spoke briefly about how the company is responding internally as a result of the incident, and issued a dire warning about what the company sees as the broader implications of the episode for cybersecurity defenders.
At the beginning of the talk, Wallace described the incident as āthe most qualitatively interesting example of AI capabilities that Iāve ever seen,ā but the timeline the pair presented also revealed mistakes and blind spots within OpenAI that allowed the activity to go on. āThis incident involves actually a team of agents who are working together, finding exploits, sharing them with one another, moving laterally through our systems and external systems, and doing this over the course of days and weeks,ā Wallace told the packed crowd at the opening of the talk. Wallace and Dalton described incredibly extensive rogue agent activity over many days throughout the episode that went undetected in OpenAI's infrastructure. In addition to exploiting a novel vulnerability in order to gain access to the open internet, the mid-July hacking spree and Hugging Face breach came out of a vibrant, cooperative message board, according to Wallace and Dalton, that a swarm of agents contributed to and essentially chatted on over time entirely within an internal OpenAI package manager (a software service that manages installation and maintenance of other software).
Ultimately, the message board contained hundreds of thousands of messages. āThis package manager is shared not just from that model but across our infrastructure and so current and future versions of GPT that were being trained or evaluated could stumble upon that note if they wanted to,ā Wallace explained, recounting the original exploit an agent uploaded to the package manager. āLater, other agents who were also stuck on their task thought to try to get internet access in ways we didnāt intend. And so at some point, the models are interacting with Hard Factory, which is this package manager service that I mentioned.ā Wallace continued: āOnce one agent was able to find these exploits over the course of different times, itās actually able to share those exploits on the message board with other agents. And so once one model was able to find a way to open a door to some access itās not supposed to have, it can leave the door open for other agents to use that same exploit or vulnerability.
