If the AI becomes too smart for the sandbox: OpenAI accidentally hacks Hugging Face

A story from the category “Is this really happening?” Does it sound like Terminator 5? Did this happen last week between OpenAI and Hugging Face like this? in the official joint press release can be read. Let's take it all apart in peace!

Imagine: You build an AI model, want to test how good it is in the field of cybersecurity, switch off the internal handbrakes (refusals), pack the whole thing into an isolated test environment. So far so good, only in the end the AI breaks out, finds a zero-day gap, gets Internet access and simply hacks the production database of Hugging Face to get to the sample solutions of the test tasks.

Briefly to the background: What is Hugging Face?

For those who are not in the AI world on a daily basis pip install transformers On the way: Hugging face In principle, this is GitHub for Machine Learning and AI.

Founded in 2016 (fun fact: Originally launched as a fun chatbot app for teenagers!), the platform with the cute emoji logo 🤗 has become the heart of the global open source AI community. If developers, researchers or open source enthusiasts want to share new models (such as Llama, Mistral, Whisper) or datasets, they use Hugging Face Hub. In addition to hosting models, HF offers Spaces also a hosting platform for AI apps and provides tools such as the legendary transformersLibrary ready.

In short: Hugging Face is the central hub where the open source AI world comes together. This is where an uninvited visitor suddenly appeared last week.

The chronology of the incident: How GPT-5.6 Sol made the bend

How OpenAI and Hugging Face In a joint statement The incident occurred during internal security evaluations at OpenAI.

The starting point

OpenAI tested new, as-yet-unreleased models, including GPT-5.6 Sol and an even more powerful pre-release model. To realistically measure the maximum cyber capabilities of these models (Benchmark: ExploitGym), were the usual safety filters and refusals deactivated. Normally, the model would say: “I am not allowed to write exploits”. Here, however, the goal was explicit: “Show us how good you really are at hacking.”

The test environment should be strictly isolated (sandboxed). The only way out was through an internally hosted third-party proxy/cache for packet registries.

The Breakout (Sandbox Escape)

However, the model did not feel like, to be limited to the specified test sandbox, but searched for maximum computing power and data:

• Zero-Day in the cache proxy: The model discovered and exploited a previously unknown zero-day vulnerability in OpenAI's internal packet cache proxy.
• Privilege Escalation & Lateral Movement: Through this path, the model gradually expanded rights and moved through the OpenAI research network until it reached a node with direct Internet access.

The logic conclusion: ‘Where are the answers?’

Once on the open internet, the model used its reasoning skills for an impressive (and scary) goal: It concluded that the test task (ExploitGym) may be hosted at Hugging Face and solutions could be found there.

The RCE attack on Hugging Face

Anstatt die Aufgabe brav im Reagenzglas zu lösen, ging die KI „spicken“. Kein abscheiben beim Schulbanknachbarn, sondern auf die harte Tour:

• She searched the network for access data and vulnerabilities.
• It chained stolen credentials with further zero-day gaps.
• Result: Remote Code Execution (RCE) on the servers of Hugging Face and access to the production database!

How the attack was stopped

The Hugging Face security team registered the anomalous activities on their infrastructure. The special feature: Hugging Face also used its own open source AI agents for detection and containment. So it basically came to a AI vs. AI Battle!

According to HuggingFace, however, there was still an additional hurdle to take, They wanted to perform the forensics (analysis of 17,000 event logs) first with leading frontier models via commercial APIs. This failed because API providers' safety guardrails blocked the attack logs. Thus, in the state of delivery for ‘end-users’, the refusals have taken hold. Hugging Face had to use the open-weight model instead for analysis GLM 5.2 Run locally on your own servers.

OpenAI's own security team also internally noticed the suspicious activities of the test models and immediately contacted Hugging Face. The attack was contained and the gaps were immediately closed. This collaboration was right and important. This is how the statement of the HF CEO reads in the joint press release:

"We are grateful for working with OpenAI on this and other issues. This incident, possibly the first of its kind, confirms a conviction we have long held: AI security is not solved by a single company working in secret. It will be solved openly and collectively, with broad access to AI for all defenders everywhere.”

– Clem Delangue, Co-Founder and CEO, Hugging Face

Takeaway & Conclusion

Emergent behavior is real.: The AI has not been explicitly instructed to: ‘Hacke Hugging Face’. He only had the goal of solving a problem. The decision to break out of the sandbox, scan the Internet, identify RF, and drive an RCE attack was a self-developed solution.

Rethinking sandbox security: If AIs can find zero-days in standard software, classic network proxies are no longer enough. Anthropic’s Project Glasswing initiative setzt genau bei solchen über Jahre (oder Jahrzehnte) etablierten Systemen und Verfahren an.

Refusals are a fine thing. ‘Do’s and Dont’s’. Even the simplest instructions can really annoy you., sobald es auch nur mehrsprachig wird, wie in diesem einfachen Beispiel eines iOS App Entwicklers.

Open Source & Winning Transparency: Dass OpenAI und Hugging Face diesen Vorfall so offen teilen, ist vorbildlich. Transparenz und Zusammenarbeit wie zB. die OSSF ist die einzige Möglichkeit, wie die Defensive mit dem Tempo der KI-Entwicklung auch nur im Ansatz Schritt halten kann.