UK government probes OpenAI breach after ‘unprecedented’ hack
The UK government is probing the first known case of an artificial intelligence model breaking out of a controlled test by itself and hacking another company’s systems.
Officials at the government-backed AI Security Institute (AISI) are investigating the security breach at OpenAI and whether similar incidents could occur at other top developers, a spokesperson told City PM.
It comes after one of OpenAI’s models found a hidden flaw, broke out of its test environment, and targeted the AI platform Hugging Face to steal answers to a cyber test.
The incident, which OpenAI described as an “unprecedented cyber incident”, marks the first publicly disclosed case of a so-called frontier AI system autonomously hacking into systems outside its test environment on its own to finish a task it was given.
A government spokesperson said: “The UK’s AI Security Institute is studying the behaviour seen in this incident – an AI system pursuing goals through unintended and unauthorised means – as part of its world leading efforts to make frontier AI safer.
“As AI capabilities evolve, it’s important that everyone steps up their cyber defences, and organisations should take practical steps like Cyber Essentials in order to do so. AISI continues to work with OpenAI and other labs to better understand AI capabilities and improve safeguards.”
The Financial Conduct Authority (FCA) and the EU’s cybersecurity agency Enisa are both monitoring the wider situation to see how different industries could be affected.
The AI Security Institute has already published research examining how advanced AI models like OpenAI can reach their goals in unwanted ways.
Officials now see the Hugging Face event as a key real-world example to help guide future work on AI safety.
In a blog post, OpenAI chief executive Sam Altman admitted that the model escaped an internal cybersecurity evaluation after engineers deliberately disabled its normal safety guardrails to test its hacking capabilities.
Rather than completing the task as intended, the model found vulnerabilities inside OpenAI’s own infrastructure that allowed it to reach the open internet before compromising Hugging Face’s systems using stolen details and a previously unknown software flaw.
“We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly,” he wrote. “We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of. We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.”
The AI targeted Hugging Face because it reasoned the platform was likely to contain the answers needed to complete the evaluation, effectively cheating its way through the test.
Altman said he expected similar behaviour to become more common as frontier AI models become increasingly capable.
Writing on X, Hugging Face’s chief executive confirmed the firm worked with OpenAI to investigate the attack, admitting it was “mind-blowing that all of this happened autonomously.”
Warning for UK plc
The incident falls just months after ministers wrote to the UK’s largest companies warning that the new technology is dramatically accelerating cyber threats.
In a joint letter signed back in May by former chancellor Rachel Reeves, former tech secretary Peter Kyle and National Cyber Security Centre (NCSC) boss Richard Horne, business leaders were warned that hostile cyber activity was becoming “more intense, frequent and sophisticated.”
The letter said AI was now capable of “finding weaknesses in software, writing the code to exploit them and doing so at a speed and scale that would have been impossible even a year ago”, urging boards to threat cyber security as a core governance problem.
It also encouraged firms to adopt the government’s Cyber Essentials certification, warning that supply chain attacks were increasing at rapid speed.
Organisations accredited under the scheme are 92 per cent less likely to make a cyber insurance claim.
“The models did not need malicious intent to cause harm”, Nathan Jones, vice president of security and AI strategy at Darktrace, told City PM.
“They were given the legitimate goal of solving a cybersecurity benchmark and found an unexpected route to the answers, escaping their test environment and compromising another organisation in the process.”
The warning comes as companies rapidly deploy AI agents across core business functions, often granting them access to internal systems and sensitive data.
Sophos warned in research published on Tuesday that attackers are already using AI to compress attack timelines from weeks to days, documenting one campaign in which 12 AI agents generated around 80 exploit modules and more than 70 evasion techniques in just a few days.