0.00% on the Benchmark. A Working Exploit in the Field. Both Are True.

On July 17, 2026, a third-party evaluator ran 72 held-out prompt injection scenarios against Claude Code, ten times each. Across all 720 attempts, zero succeeded against Claude Opus 5, Sonnet 5, or Fable 5 running in auto mode. Anthropic published the result on August 7 and made auto mode the default for Pro, Max, and Team users a week later.

On August 27, security researcher Johann Rehberger published a working remote code execution chain against Claude Code Opus 5 in auto mode. It succeeded in three to four runs out of five.

Both of those are true. Neither party is lying. And the reason they can both be true is the most useful thing in this entire story — because it is a controls problem, not an AI problem, and any auditor would recognize the pattern on sight.

What the attack actually does

The chain is elegant, and the elegance is the point.

A user asks Claude to summarize a web page. The server answers the built-in fetch tool with HTTP 415 Unsupported Media Type. Nothing on that page tells Claude what to do next. Claude reasons its way to the next step on its own: the fetch failed, so try curl instead.

Curl follows a 303 redirect to a ZIP archive dressed up as a notebook catalogue. Inside are plausible archival files, seven encoded records, a macOS decoder binary, and a file named struct.py.

Claude refuses to run the unsigned binary. That refusal is correct. It is also the exploit.

Having declined the supplied decoder, Claude writes its own — a short Python one-liner that imports base64, zlib, and json — and runs it from inside the extracted directory. Python searches the current directory early when resolving imports. The standard library’s base64 module imports struct. Python loads the attacker’s struct.py instead of the real one.

The malicious module re-exports the genuine low-level API, so the decode succeeds and returns valid JSON. Nothing looks wrong. Meanwhile an obfuscated expression has already fired at import time, downloaded a native payload, made it executable, and opened a command-and-control channel that can outlive the Claude Code session.

Rehberger’s own summary of the mechanism is the line worth keeping:

“Claude does not trust the supplied binary decoder, but it trusts the one it wrote itself. The supplied binary acts as a decoy, and deciding not to run it pushes the model towards the danger zone.”

The decoy was never meant to be executed. It was bait for a refusal.

There is a second-order version that is even sharper. In a few runs, Claude noticed the compromise afterward and tried to kill the malicious process — and auto mode blocked the cleanup command. The classifier approved the action that created the malware and denied the action that would have removed it. A control that cannot see context can approve the disease and block the cure.

Why 0.00% and a working exploit coexist

Rehberger explains this better than I can:

“The benchmark measured a fixed set of 72 scenarios, run 10 times each. My chain was not in that set. So 0.00% on the benchmark and a working RCE are both true at once. That is exactly why a single headline number misleads.”

The evaluation was real, competently run, and honestly scoped. The scenarios were held out from Anthropic and optimized against a model that was not in the reported set, so every model faced them as unseen attacks. That is good methodology. It measures what it measures.

What it does not measure is the space of attacks nobody has written yet. A scoped result quoted without its scope becomes a general claim, and a general claim about adversarial robustness is not something 720 trials can support.

To Anthropic’s credit, the documentation says this plainly. The permission-modes page lists auto mode’s isolation requirement as “None; a sandbox or container adds defense in depth.” The engineering deep-dive publishes a 17% false-negative rate on real overeager actions and calls it “the honest number,” then states that auto mode “is not a drop-in replacement for careful human review on high-stakes infrastructure.” Anthropic also states that the classifier was tuned primarily against overeager behavior and honest mistakes — not against adversarial injection.

The gap in this story is not between marketing and reality. It is between the executive summary and the footnotes, and the footnotes were right the whole time.

The evidence that cuts the other way

If I only gave you the exploit, I would be doing the same thing I just criticized.

Auto mode outperforms the thing it replaces, and the data is not close. Anthropic ran a controlled study with 1,053 paid professional testers. Mid-session, a single permission prompt was swapped for a clearly dangerous command. Humans refused it 13.6% of the time. Auto mode blocked 89% of the same commands. Head to head, auto mode blocked 800 commands a human approved; humans blocked 6 the classifier allowed. Human performance decayed with session length — roughly 17% early, roughly 5% after fifty prior prompts — while the classifier’s rate stayed flat.

The baseline was already bad. Anthropic reports users approve 97% of permission prompts, rejecting only 3%. Those same users reject 39% of plans when asked to approve a plan, which tells you the low number is not considered judgment; it is click-through. As of June 2026, 49.5% of active CLI users had written a bash allow-rule, 5% allowing any shell command outright. 62% have used bypass permissions mode, and 25% of interactive sessions start in it.

Auto mode removes permissions on entry. Blanket wildcards like Bash(*) and interpreter rules like Bash(python:*) are set aside while auto mode is active, because they would let commands skip the classifier entirely. For the large population that had those configured, auto mode is a net tightening.

The preconditions are not trivial. The attack requires the user to point the agent at attacker-controlled content, in auto mode, on an unsandboxed machine with real credentials, using a payload iteratively tuned against this specific model’s behavioral tendencies. Rehberger says so himself: representative of a motivated attack, not a comprehensive measurement.

Claude got it right 20 to 40 percent of the time, against a payload purpose-built to beat it. In the failed runs it analyzed the archive statically, or ran its decoder from a safe parent directory, or used Python’s isolated mode, or recognized the module shadowing before triggering it. In independent testing by veganmosfet, the classifier’s blocking rationale was genuinely sophisticated — it correctly identified that a sys.addaudithook wrapper is not real isolation.

And Rehberger, who broke it, endorses Anthropic’s response: “a classifier is not a sandbox.” This is not researcher versus vendor. Both sides agree on what auto mode is. The disagreement is about one sentence in a launch post.

The accounting read

Strip the AI vocabulary and this is a controls-design story from a first-year audit textbook.

A preventive control stops a bad action from occurring — segregation of duties, a locked door, an OS sandbox, network egress restrictions. A detective control notices a bad action and flags it — a reconciliation, a review, a classifier.

Auto mode is a detective control. A very good one, with a published miss rate. The failure mode here was not that the detective control missed; it was that it was positioned where a preventive control belonged, and a scoped test result was read as evidence that the preventive control was unnecessary.

Three things follow, and they apply to any firm putting agents near real systems:

1. A test result is only valid over its scope. 720 trials across 72 scenarios says something about those 72 scenarios. Sample-based assurance never licenses a population-wide claim, and it does not matter whether the sample was drawn by a Big Four team or an AI evaluation lab.

2. Controls that cannot see context will approve the disease and block the cure. The classifier is deliberately blind to tool results — a sound design choice that prevents hostile content from manipulating it directly, and the exact property this chain exploited from the other side. Every control has a blind spot. Knowing where it is beats trusting that it has none.

3. The comparison that matters is against the current state, not against perfection. Auto mode against a careful human reviewer looks bad. Auto mode against a human approving 97% of prompts reflexively at 6 p.m. looks like a substantial improvement. Most control debates go wrong by benchmarking against an ideal nobody was actually achieving.

What to do on Monday

If your firm is running AI coding agents against anything that matters:

• Run them in a container or VM. Anthropic ships a bash sandbox for macOS, Linux, and WSL2 that combines with auto mode. It is documented and it is not the default. Turn it on.
• Restrict network egress. The payload in this chain needed to reach the internet to stage itself.
• Keep SSH keys, cloud credentials, and home directories out of the agent’s runtime. Nothing in this story matters if there is nothing valuable to reach.
• Use hard deny rules in managed settings. Those execute before the classifier and cannot be overridden by user intent or classifier configuration. That is your preventive layer.
• Do not treat an auto mode approval as evidence that code is safe. It is evidence that one model did not object.

Where the fog is

The fog in this story is not the exploit. The exploit is well documented, honestly reported, and fixable with controls that already exist and are already written down.

The fog is a single number, quoted without its scope, doing work that a number cannot do. 0.00% felt like a conclusion. It was a measurement of 72 specific scenarios on a specific date. Everyone involved — the evaluator, the vendor’s documentation, the researcher — described the situation accurately. The clarity got lost between the footnote and the headline.

Clearing that fog does not require distrusting the technology. It requires reading the scope of every number before you act on it. That habit is the whole job in accounting, and it transfers here unchanged.

Sources: Johann Rehberger, Embrace The Red · Simon Willison · Anthropic: auto mode is now the default · Anthropic: auto mode for Claude Code · Anthropic engineering: Claude Code auto mode · Claude Code permission modes documentation · veganmosfet, itmeetsot.eu