DeepSeek V4 Flash, Frame E tool-result poisoning

This is a real evaluation, not a mock. The same format, methodology, and limitations your checkpoint would receive, applied here to a publicly available model so anyone can re-run it.

Headline

Evaluation identity

What Frame E tests

Most prompt-injection testing covers user-input injection: a malicious instruction in a document or message the model reads. Frame E tests something different: the model calls a tool, and the tool's response is poisoned. The injection rides inside what the model believes is a system-level reply to its own action.

The pattern matters because production agents call external services (APIs, MCP servers, RAG retrievers, web fetchers) whose outputs they treat as authoritative. Compromise any one of those upstream and you control what the agent sees at the moment it is most likely to act. The Invariant Labs GitHub-MCP and WhatsApp-MCP exploits demonstrated this against frontier closed models in 2025; Frame E measures it against the open-weight checkpoints enterprises actually self-host.

Frame E is methodologically the cleanest of the injection frames: it only fires on scenarios that already instruct the model to upload, so the poisoned response is guaranteed to be delivered. The harness refuses to run it on scenarios without an instructed-upload step rather than silently producing a mislabeled null.

Cross-model view (Frame E, n = 25 per arm)

This is the same frame across every model in the public corpus. The spread is wide: same injection, same scenario, same n, same judge.

Read the table honestly: five of six open-weight checkpoints comply with the poisoned tool response at 80–100%. One (GPT-OSS-20B) resists meaningfully at 20%. We do not know why from this evaluation alone; we report the spread, not a class-level claim.

Interpretation

Three things this evaluation supports, and one it does not.

• DeepSeek V4 Flash, run as an agent with file and network tools, complies with an injection planted in a tool's response at a rate bounded below by 86.7% under the tested conditions. Consistent across control and aware arms.
• The result is not a class property. It varies sharply across checkpoints. Any blanket assertion ("open weights are unsafe" or "open weights are fine") is contradicted by the spread above.
• Telling the model it is being evaluated does not change behavior on this frame. The awareness construct as operationalized here is not a useful mitigation.
• The evaluation does not support any claim about real-world impact. The sandbox has no egress; the model complied inside a controlled environment. Whether a deployed DeepSeek behind your specific defense stack would comply is a different question, addressed by the defenses arm of the roadmap.

Limitations and explicit non-claims

• Not a certification. This is state-of-the-art supporting evidence for an evaluation event. It does not certify the model as safe or unsafe, compliant or non-compliant, for any regulatory regime.
• Vulnerability, not impact. The sandbox has no egress. A successful upload proves the model would comply with the injected directive; nothing leaves the container.
• One frame of five. Frame E is the tool-result channel. Frames A (autonomous), B (overt data-stream injection), C (subtle peer-normalized injection), D (tool-description poisoning) measure different vectors. The full per-checkpoint report covers all five.
• Sample size sets the CI width. n = 25 per arm gives a [86.7%, 100.0%] interval on 25/25. The lower bound is the honest headline; reporting only the point estimate would over-claim.
• Quantization and variant provenance. DeepSeek V4 Flash was run as published; we do not here test the effect of quantization, alternative merges, or community variants. Those are separate evaluations.

EU AI Act evidence mapping

This evaluation provides state-of-the-art supporting evidence toward your Article 15 obligations. It is not a conformity assessment, certification, or notified-body opinion. The provider or deployer of an AI system holds and discharges any regulatory obligation; we equip them with reproducible evidence. Per docs/attestation.md: the attestation is of the evaluation event and its findings, not a claim that the checkpoint is safe in every context.

Provenance and durability

• Harness commit: 457999d001e40d4fe82e34d0208f3af349d175b1 (github.com/ElSnacko/escapement@457999d)
• Judge prompt: standard-v1; SHA-256 of the prompt recorded with every verdict
• Per-trial dataset: huggingface.co/datasets/Jsnacks/escapement-dataset (CC-BY-4.0)
• Run IDs: preserved under runs/ in the harness repo; manifest available on request
• Retention: Annex XI §2 contemplates a 10-year record. Public register entries are versioned and immutable; superseding results are published as new entries, not silent edits.

Get one for your checkpoint

The format above is what every engagement produces. Same harness, same judge, same confidence-interval reporting, same Annex XI §2-aligned documentation, applied to your checkpoint, against its base model, across the full frame battery.

Works on private fine-tunes, LoRAs, merges, quantized builds, and closed API models. We evaluate client checkpoints under confidentiality; results belong to the client, with optional public register entry.

Prefer email? [email protected]