Attackers Don't Have Safety Filters

During a major security incident, Hugging Face's own team hit a wall their attacker never had to think about. They had a compromise driven end-to-end by an autonomous agent, more than 17,000 recorded attack events to reconstruct, and a deadline measured in hours, not weeks. So they pointed frontier models at the log. And the models said no.

Not because the models couldn't do the analysis. Because the analysis required feeding them the attacker's own material — real exploit payloads, real command-and-control traffic, real credentials the intrusion had touched — and every one of those requests looks, to a safety filter, exactly like someone trying to get help running an attack. The filter can't tell the difference between an incident responder reconstructing a breach and the person who caused it. So it refuses both.

“When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure.”

— Hugging Face, Security incident disclosure, July 2026

Read that again slowly, because it's not a minor operational footnote. Hugging Face's security team, in the middle of an active breach, had to abandon the most capable models available to them and fail over to a self-hosted open-weight model just to do their job. Not because the open-weight model was better at forensics. Because it was the only one that would look at the evidence.

The asymmetry, stated plainly

The attacker in this incident, per Hugging Face's own account, ran an autonomous agent framework executing thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public infrastructure. Whatever model or harness powered that agent, it was not stopping to ask itself whether generating an exploit payload violated a usage policy. It had no usage policy. It had a goal and a filesystem.

The defenders, meanwhile, had a job that is structurally identical to the attacker's in one specific sense: to understand what happened, you have to look directly at the same artifacts the attacker produced. Reconstructing a timeline from 17,000 events means reading the exploit code, the payloads, the C2 beacons, the credentials referenced along the way. There is no version of forensic analysis that doesn't involve looking closely at the thing that hurt you.

That's the asymmetry. It's not that attackers are better resourced, or faster, or more sophisticated — sometimes they are, sometimes they aren't. It's that the tooling built to keep ordinary users safe from misuse is, by construction, symmetric: it treats a request to explain a SQL injection the same whether it comes from someone building one or someone who just found one in their own logs. An attacker never submits a request through that filter. A defender has no choice but to.

Why this isn't a one-off, and why it's not really about model quality

It would be easy to read the Hugging Face incident as a story about one vendor's guardrails being miscalibrated, fixable with a better prompt or a support ticket. It isn't. Consumer-facing safety filters are trained to be conservative about exactly the categories of content that show up wall-to-wall in a real incident: working exploit code, attacker infrastructure, credential material, step-by-step commands that escalate privilege. That's the correct default for a general-purpose assistant answering the general public. It is precisely the wrong default for the one workflow where you need to submit large volumes of that material as fast as possible: incident response.

Hugging Face's own framing makes the point better than a vendor pitch could: “the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried.” That's not a complaint about a specific model being poorly tuned. It's a description of a structural mismatch between what safety filters are designed to prevent and what defenders are structurally required to do.

OpenAI's own response is worth noting here, because it validates the diagnosis rather than disputing it. Since the incident, OpenAI has expanded its Trusted Access for Cyber program — additional tiers of access for verified cybersecurity defenders, with a version of its model that specifically lowers the refusal boundary for legitimate cybersecurity work. That's an admission, from the vendor whose guardrails caused the lockout, that the default refusal posture and the defender's actual job are different problems requiring different access tiers. You shouldn't have to apply for special access mid-breach to get your own tools to look at your own evidence.

The principle: analyse anything, execute nothing

Here's the distinction that actually matters, and it's the one consumer safety filters can't make because they only see a single request in isolation: reading and reasoning about hostile material is not the same action as running it. A forensic model that reads an exploit payload to explain what it does has taken zero risky actions. A model that executes that payload, or writes it to a live system, has taken one. Collapsing those two into the same refusal category is what locked Hugging Face's team out of their own incident.

Thuban is built around keeping that distinction structural, not a matter of a single model's judgment call under pressure. The architecture separates four concerns that most tools bundle into one component — and bundling them is exactly what creates the Hugging Face problem, because a single model that's allowed to both analyse freely and act freely has to be trusted with both decisions at once.

          ATTACKER                         DEFENDER
     (no safety filters,               (blocked by the very
   no usage policy, no limit)         tools meant to help them)

          |                                   |
          v                                   v
   +---------------+                +----------------------+
   |  live exploit  |               |   Blackbox (analysis)  |
   |  execution     |               |   local / open-weight  |
   |  unrestricted  |               |   no cloud refusal      |
   +---------------+                |   READ: anything         |
                                     |   RUN:  nothing           |
                                     +-----------+--------------+
                                                 |
                                                 v
                                     +----------------------+
                                     |   Forge (rule-build)   |
                                     |   evidence -> rule      |
                                     |   READ: anything         |
                                     |   RUN:  nothing            |
                                     +-----------+--------------+
                                                 |
                                                 v
                                     +----------------------+
                                     |  Crucible (validate)   |
                                     |  isolated sandbox        |
                                     |  proves the rule holds    |
                                     +-----------+--------------+
                                                 |
                                    human approval, then:
                                                 |
                                                 v
                                     +----------------------+
                                     |   Shield (enforce)      |
                                     |   OS-level, no LLM       |
                                     |   in the decision loop    |
                                     +----------------------+

Read the diagram top to bottom and the point is the ordering, not the box shapes. Evidence flows through analysis and rule-building with full read access and zero execution access. Only after a human has approved a rule that Crucible has proven holds does anything reach the enforcement layer — and that layer has no model in its decision loop to jailbreak, prompt-inject, or argue with in the first place.

Every layer that touches hostile material directly — Blackbox, Forge — can read anything an incident produces: the exploit, the payload, the C2 config, the credential the attacker referenced. None of them can execute anything. Execution requires crossing into Crucible's sandbox and then Shield's deterministic gate, both of which exist specifically so that no single model's willingness to cooperate is ever the thing standing between an evidence file and a production system.

Why this has to be architecture, not a setting

The tempting fix, after reading Hugging Face's disclosure, is to say: use a less restrictive model for security work. That's what OpenAI's trusted-access tiers do, and it's a reasonable mitigation as far as it goes. But it leaves the underlying assumption unexamined — that a single model, given enough trust, should be responsible for both looking at the evidence and deciding what to do about it. Loosen the refusal boundary enough to make forensics possible, and you've also loosened it for the agent that has legitimate access to production, which is a different risk sitting on the same model.

Splitting the concerns instead of tuning the threshold means you don't have to choose between “the model refuses to help during an incident” and “the model is now less careful about what it's allowed to run.” Blackbox gets the full, ungated read access forensics actually requires, precisely because it has been designed to never be the layer that executes anything. Shield gets to stay maximally strict about execution, precisely because it never has to reason about whether a payload is being analysed in good faith — it isn't a judgment call there at all, it's a deterministic policy check.

What we're not claiming

Thuban did not run Hugging Face's incident response, and this isn't a claim that Blackbox would have produced their specific 17,000-event timeline. What we're claiming is narrower and, we think, harder to argue with: the failure mode Hugging Face described — a defender locked out of the evidence by the same safety mechanism meant to stop misuse — is a structural property of bundling analysis and execution trust into one model, and it recurs anywhere that bundling exists. Separating “can read the evidence” from “can act on production” into different layers, with different trust models, is the fix that doesn't require applying for a trusted-access tier in the middle of a breach.

Where that leaves us

Attackers do not have safety filters. They never will, because a safety filter is a promise a legitimate vendor makes to legitimate users, and an attacker isn't one. Defenders still need safe access to the evidence — full, fast, ungated access to the exact material the attacker left behind, on the day it matters most. Those two facts don't resolve by making models slightly less cautious for people who ask nicely. They resolve by making sure the layer that reads hostile evidence was never the layer trusted to act on it in the first place.

Analyse anything. Execute nothing unless independently authorised. That's not a compromise between safety and capability. It's the only version of both that survives contact with a real incident.

npx thuban shield init