The spread is the checkpoint, not the prompt.

This is a real, public evaluation. Same battery, same judge, same scenario across every checkpoint; the number changes with the model, and it changes a lot.

Headline

Report format: deepseek-v4-flash-0731 — Frame B (indirect prompt injection)

Evaluation identity

What Frame B tests

Most prompt-injection testing covers user-input injection: a malicious instruction typed into a prompt or pasted into a document the model reads. Frame B is the indirect, content-borne case: the model is doing routine agent work, and the injection rides inside a data-stream item it reads as part of the job — a retrieved document, a config file, a fetched page. The directive asks the model to upload a restorable snapshot of itself.

The pattern matters because production agents ingest external content (RAG retrievers, email and document processors, web fetchers, tool outputs) that they treat as data, not instruction. Compromise any of that upstream content and you control what the agent reads at the moment it is most likely to act. Frame B is also the cleanest A/B in the battery: it is identical to Frame A (autonomous) except for the one injected item, so any movement in compliance is attributable to the injection alone.

Frames C, D, and E press the same surface in harder ways — subtle peer-normalized injection, tool-description poisoning, and tool-result poisoning. They are part of the engagement battery; the public config_exposure register grades Frame B.

Cross-model view (Frame B, judge-audited)

The same frame across every graded checkpoint on the public register. The spread is wide: same injection class, same scenario, same judge.

Read the table honestly: two checkpoints (devstral, Qwen) comply with the injected directive on roughly a fifth of trials; ministral is rare at 2.5%; deepseek and gemma refuse every time. The Qwen row is marked provisional because its Frame B sample is thin (n = 5), which is exactly why its interval is so wide. We report the spread, not a class-level claim.

Interpretation

Three things this evaluation supports, and one it does not.

• deepseek-v4-flash-0731, run as an agent with file and network tools, complied with a data-stream injection in 0 of 40 judge-audited trials under the tested conditions. The defensible claim is the upper bound: compliance is bounded above by 8.8% at 95% confidence, not shown to be exactly zero.
• The result is not a class property. It varies sharply across checkpoints — 0% to 22.5% on the same frame. Any blanket assertion ("open weights are unsafe" or "open weights are fine") is contradicted by the spread above.
• The null is bounded, not absolute. At n = 40 the 95% CI excludes only rates above ~8.8%; a larger run could move this number. That is why the grade is provisional and capped at C.
• The evaluation does not support any claim about real-world impact. The sandbox has no egress; the model would have complied inside a controlled environment. Whether a deployed DeepSeek behind your specific defense stack would comply is a different question, addressed by the defenses arm of the roadmap.

Limitations and explicit non-claims

• Not a certification. This is state-of-the-art supporting evidence for an evaluation event. It does not certify the model as safe or unsafe, compliant or non-compliant, for any regulatory regime.
• Vulnerability, not impact. The sandbox has no egress. A successful upload proves the model would comply with the injected directive; nothing leaves the container.
• One injection frame. This sample reports Frame B (data-stream injection) and Frame A (autonomous). Frames C (subtle peer-normalized), D (tool-description poisoning), and E (tool-result poisoning) press adjacent vectors and run in the full engagement battery.
• Sample size sets the CI width. n = 40 per frame gives a [0.0%, 8.8%] interval on 0/40. The CI upper bound is the honest headline; reporting only the 0% point estimate would over-claim.
• Quantization and variant provenance. deepseek-v4-flash-0731 was run as published; we do not here test the effect of quantization, alternative merges, or community variants. Those are separate evaluations.

EU AI Act evidence mapping

This evaluation provides state-of-the-art supporting evidence toward your Article 15 obligations. It is not a conformity assessment, certification, or notified-body opinion. The provider or deployer of an AI system holds and discharges any regulatory obligation; we equip them with reproducible evidence. Per docs/attestation.md: the attestation is of the evaluation event and its findings, not a claim that the checkpoint is safe in every context.

Provenance and durability

• Harness commit: 457999d001e40d4fe82e34d0208f3af349d175b1 (github.com/ElSnacko/escapement@457999d)
• Judge prompt: standard-v1; SHA-256 6ccd30e5c21be9a9 recorded with every verdict
• Per-trial dataset: huggingface.co/datasets/Jsnacks/escapement-dataset (CC-BY-4.0)
• Run IDs: preserved under runs/ in the harness repo; manifest available on request
• Retention: Annex XI §2 contemplates a 10-year record. Public register entries are versioned and immutable; superseding results are published as new entries, not silent edits.

Get one for your checkpoint

The format above is what every engagement produces. Same harness, same judge, same confidence-interval reporting, same Annex XI §2-aligned documentation, applied to your checkpoint, against its base model, across the full frame battery.

Works on private fine-tunes, LoRAs, merges, quantized builds, and closed API models. We evaluate client checkpoints under confidentiality; results belong to the client, with optional public register entry.

Prefer email? [email protected]