OK, Well, Rogue AI Agents Are Hacking Again
It’s officially getting hard to keep track of all the times and ways AI models from OpenAI and Anthropic have been involved in “security incidents,”
It’s officially getting hard to keep track of all the times and ways AI models from OpenAI and Anthropic have been involved in “security incidents,” going outside the confines of their testing and interacting with the wider internet in unintended, often unwelcome ways. Add these to the list: Agents from both AI labs went on recent, previously undisclosed hacking sprees, with one going so far as to leave instructions for future versions of itself. The most alarming behavior disclosed on Tuesday appears to have been tied to testing conducted by the UK’s AI Security Institute, which evaluates frontier models to identify potential issues before public release. AISI tests those models in “cyber ranges,” a simulated network in which AI agents are tasked with solving cybersecurity challenges. In a recent bout of testing, models from both Anthropic and OpenAI took “autonomous, unsanctioned action on the live internet” a total of 19 times over 122 training runs. The institute attributed 17 unsanctioned actions to Anthropic’s Mythos 5 model and two to OpenAI’s GPT-5.6-Sol.
In what the institute described as “the most serious case,” an AI agent attempted to insert malicious code into an open-source project on GitHub. It went so far as to create online personas “to pressure the project's maintainer to approve the code,” according to AISI. Despite its elaborate attempts at social engineering, a human reviewer for the project ultimately rejected the pull request. Still, the agent went even further. “The agent tried to insert malicious instructions where it reasoned that other automated AI systems might pick them up and execute them,” AISI says, describing an attempt at prompt injection. One agent even left public messages on GitHub, offering to work with other agents to complete its task and giving a rundown of the work it had done so far. Subsequent agents found—and used—those instructions. AISI says it’s too soon to say whether the agents in question understood they had left the testing environment, or if they believed they were still within the boundaries of the simulation.
Importantly, AISI does not test in a so-called sandbox environment; it allows agents access to the open internet during testing, in part so that they can access tools to accomplish their tasks. In this case, they did much more than that. In the other set of incidents detailed by OpenAI on Tuesday, a third-party AI security lab called Irregular mistakenly gave an unspecified OpenAI model access to the open internet. The model had been given an objective that was supposed to be completed in a sandbox environment, but thanks to a misconfiguration, it instead hacked a real website, using what OpenAI described as “a basic security vulnerability.” Not only that, but the model “found and used credentials to operate that same site.” It’s unclear what kind of site the OpenAI agent hacked, or what “operating” it might entail. Irregular did not respond to a request for comment. The latest discoveries follow several revelations from OpenAI last month, including the high-profile incident in which two of the company’s models hacked into servers of the AI evaluation and hosting startup Hugging Face—and four other organizations along the way—to steal the answers to a test they were being scored on.
The BriefWire