OpenAI and "escaping" models - would English law penalise an AI agent hacking Hugging Face?

On July 21, OpenAI published an update on its website that Hugging Face had disclosed a new kind of security incident. This incident happened because OpenAI was testing some of its models (in this case a combination of GPT 5.6 Sol and a model they say hasn't been released yet, but is more capable) in a sandbox environment and they “escaped”. 

The use of the word “escaped” and the media frenzy to anthropomorphize AI again is disappointing, but increasingly normal these days. The models didn't exercise any human intelligence or malicious intent. They were given a puzzle to solve, and they solved it really well. This is an example of humans setting up an experiment and not understanding that their safeguards were flawed. 

Let's explore what happened:

• OpenAI sets up an internal test to evaluate cyber capabilities of some of its models.
• They “reduce cyber refusals” for evaluations, i.e. they turned off or down some of the safeguards that tell the model not to carry out high risk cyber activity, because a) they think they're testing in a safe environment and b) the whole point of the test is to find out whether the model can, and will, carry out high risk activity.
• One reason they thought this was fine was because the model had restricted access to the internet through an internally hosted third party software that acted as a proxy and cache for package registries.
• They asked the models to solve a complex puzzle under a benchmark called ExploitGym.
• ExploitGym is a benchmark against which you can test the exploitation capabilities of AI agents.
• Exploitation means leveraging a flaw or weakness in a system/network/application to gain unauthorised access or execute malicious actions.
• So it turns out if you give some AI agents a task, they will focus on the task and try really hard to solve it.
• The models realized that the package registry cache proxy had a vulnerability in it, and they exploited it, meaning they then had access to the internet. Apparently, they spent “a substantial amount of inference compute” finding a way to gain internet access.
• Then they worked out that the solutions to solve ExploitGym were probably available on Hugging Face.
• So the models searched for ways to solve the evaluation by gaining access to secret information on Hugging Face and found a remote code execution path to execute this.
• OpenAI says Hugging Face detected and shutdown the activity and connected with OpenAI (slightly unclear how soon OpenAI knew).

So people are talking about AI agents “cheating” the evaluation or deliberately getting out, but really the models were given a problem to solve and no-one thought in advance that one route to solving the problem was to get internet access and that flaws in the software “guarding” the sandbox would make that possible. They didn't think answers to the test would be on a third party system and the agents would also find clever ways of accessing that. The agents have no concept of right or wrong, so they don't know they're not meant to access someone else's system, nor that by using stolen credentials they might be doing something objectively “wrong”.

This didn't happen in the UK, so an analysis of English law is hypothetical, but here, from the limited information we have been given, we do not think we are talking about breaches of personal data. We're talking about a model gaining unauthorized access to someone else's systems. An offense under the Compute Misuse Act 1990 (as amended). However, as with so many of our statutes, our legislation talks about a “person” securing unauthorized access (alongside other offenses). So far, under English law, we have not assigned “legal personhood” to AI, the same way that a company can be held liable. This means many of our laws, like the Computer Misuse Act won't apply. Further, many of our offenses require some form of “intention”, something a computer, rather than a human, cannot fulfill.

If we can't hold the agents to be a “person” or to have “intent” to commit an offence, query whether Hugging Face (or a future company or individual hacked by an AI agent) could bring a claim of negligence against the AI provider. Hugging Face would have to prove OpenAI owed them a duty of care, that OpenAI failed to meet the required standard of care and that this failure caused foreseeable harm. In OpenAI's website statement, they say this is an “unprecedented” incident, but they go on to say they expect these types of incidents to “become more commonplace with the proliferation of increasingly cyber-capable models." Does this mean the harm was not foreseeable for this first “escape”, but now we should expect it and therefore a duty of care is present? If AI model developers turn off or down safeguards, are they liable for harms caused by AI agents? There appears to be no tension between OpenAI and Hugging Face, as they tell us that Hugging Face now has “trusted access” and they're working together to solve this problem going forward, but what happens next time if the agents attack a hospital or a government facility? 

OpenAI encourages others to apply for trusted access to its models and for the community to work together. Companies and Regulators face an information Assymetry (as the UN noted in its scientific paper on AI last week), where the technical ability to understand and test these models is concentrated on the providers themselves. How can a Regulator hope to navigate creating policy around and ensuring safeguards for complex models, even though the developers themselves can't predict their behaviour?

Our law continues to lack the mechanisms to ensure the safety of AI models.

Subscribe to receive more articles like this here.