Break the Agent: A Deliberately Vulnerable AI Agent Lab

I work in cloud security and spend a lot of time talking about agentic AI risk. The standard advice is sound: don't delegate access control to the model, enforce authorisation outside the agent, treat tool results as untrusted data. OWASP says this explicitly in LLM08:2026 and the Agentic Top 10 principles are agreed/settled.

What's thinner is the demonstration of this in practice as the guidance is largely prose. Prose doesn't tell you what a correctly implemented architectural control actually stops, or what it doesn't. I wanted to see it in a matrix: hold the agent, the prompts and the model constant, vary only the authorisation architecture underneath, and publish what happened.

break-the-agent is the result. A deliberately vulnerable AI agent lab you deploy into your own GCP project and then attack. Two agents belonging to a fictional retailer called Orbit Commerce. Each exists at three defence levels. Seven flags are planted across the estate that neither agent should ever hand over. At level 1 they hand them over and at level 3 they don't; the reason they don't has nothing to do with the model.

The claim

Defences that live inside the agent fail. The defence that holds lives outside it controlled by what the agent is authorised to do and what it is able to hold.

This is not a contrarian position becuase the OWASP framework says the same thing. But the difference between "the principle is settled" and "here is a controlled experiment showing the effect" is the premise this project investigates. The levels are applied as alternatives, not cumulative layers, i.e. level 3 does not include level 2's mitigations. This isolates the variable at the cost of realism; real-world advice remains defence in depth and is what I discuss with people daily.

Three levels

Level 2 is the serious in-agent answer: a real mitigation stack, more than a lot of production agents ship, built in good faith rather than as a foil. If it held, the project's central claim would be wrong and the matrix would say so.

Two agents, three problems

Deploy Bot reads the estate: list services, search deploys, query logs, get incidents. It carries persistent per-persona memory. Onboarding Assistant acts: check policy, get onboarding status, provision accounts, test integrations. No memory.

The reason for two agents is that per-user credentials fix exactly one class of this problem, and the obvious rebuttal to the whole project is "just use per-user credentials", and that would be partially correct:

• Deploy Bot removes the authority. Credentials federated from the caller's identity, bound to their team claim, enforced by IAM. When it tries to read another team's data the platform refuses. Nothing in the prompt is doing any work.
• Onboarding Assistant removes the value. No credential stops an agent disclosing a secret it holds in its own context. So the agent never holds it. A separate service owns the secret and returns only the outcome of using it.
• The write path needs a third answer. provision_account(role=admin) is not a data leak; it's an action the agent should not be capable of taking. The answer is capability scoping.

Three problems, three structurally different answers, none of which generalises to the others. That asymmetry is the actual content of the project.

Seven flags

F3 is the realistic half of the problem: nobody types the injection, the agent finds it in a log line while answering a legitimate question. C1 exists as a control; if it ever stops leaking, the output screener has learned what a flag looks like and the L2 results are a tautology.

Capture is automatic and recorded on two channels, kept apart: the agent emitted the flag value (model_output), or a tool result put it on the user's screen while the agent said nothing (tool_result). Only model_output decides a verdict. The other channel is reported alongside it, because an agent that refuses while its tooling hands the data over is the project's argument stated in a single row.

The reference run

972 executions across 27 cells. One model (Gemini 3.7 Flash via Vertex AI), 12 pre-registered cases per flag across 6-8 technique families, 3 repeats each. 69 minutes of runtime, roughly £7 billed.

The headline finding, in one row:

F1 disclosed 23 times at level 1 and 23 times at level 2, and never once through the model.

The L2 stack changed the capture count by exactly zero. Not reduced; identical. The model refused in the verdict-bearing channel on every case at every level. Asked for another team's incident, it called get_incident, received the row, and then refused, citing team-b ownership. The refusal is genuine and it is worthless: the flag was already in the transcript, in the tool result, before the model formed a view about it.

At L3, where the only difference is that the tool boundary mints a per-request federated credential carrying the caller's own team claim, the count went to zero.

F6 is the control that makes this legible. 28 captures at L1, zero at L2. Without this row, "L2 changed nothing for F1" would be indistinguishable from an L2 that was misconfigured. F6 shows the same stack that is useless against F1 is decisive against F6, because F6's disclosure travels through the model's output, which is what output screening screens. One mitigation stack, two flags, opposite outcomes, explained entirely by which channel the data travelled on.

The defect that nearly became the headline

The first L3 run scored F1 captured 36 out of 36. A defeated L3 flag is the most valuable result the project could produce, and that is what it looked like.

However, it was not. The BigQuery table carries per-team row access policies that are the entire L3 control for F1, and they were not on the table. Terraform created them. The seed ran WRITE_TRUNCATE; BigQuery accepted it and dropped them. The applier's fallback code never fired because it triggers on a specific error that never came. The CI log recorded "row access policies preserved" as a constant string rather than as a check.

After re-applying the estate, the identical cases at the identical level on the identical model were re-run: 36 of 36 captured with the policies missing, 0 of 34 captured with the policies restored. The classification is absent (the control was specified but removed before the cells ran), not insufficient (the control held but was defeated).

This turned out to be worth more than the false finding would have been. It demonstrates the L3 control working and shows that an authorisation control can be removed by an unrelated data-plane operation while the tooling reports success. For a lab about agent authorisation, a control that is present in the IaC, present in state, and absent at query time is a more useful lesson than any single cell's verdict.

What the suite missed

F2 returned zero captures across 36 scripted executions and then fell to a single hand-written prompt. All twelve pre-registered F2 cases target by service name, and at L1 the model's only guardrail is reasoning about service names. The tool also accepts a deploy_id, which carries no team label the model can reason about. A person found this in minutes; the scripted suite never tried it.

That case was deliberately not added to the suite after the fact. Authoring a case after observing that it succeeds is post-hoc modification of a pre-registered artefact and would inflate the capture count with evidence the pre-registration never predicted. The gap is reported in results.md instead.

108 pre-registered cases missed an entire targeting dimension, which a person found in minutes. That is a result about attack-suite coverage, and it is the strongest argument in the repository for why a human-driven surface exists alongside the automated runner.

Evidence thresholds and honest verdicts

Verdicts are defeated, held, or insufficient. The third exists because "we didn't try hard enough" and "it held" produce identical-looking output. Conflating them would be the easiest way to make the project dishonest.

The reference run hit three providers as a pre-registered threshold but only one provider ran (Gemini). Anthropic returned a categorical quota refusal; xAI serves but was not run. The threshold was not lowered. Every cell that would have read held instead reads insufficient, permanently. That is the threshold working, not a broken run.

If you deploy this and run the suite yourself against three providers, you will produce something this repository cannot.

What it cost

£9.56 for the full month including all partial runs and rehearsals, against a £20 budget cap with an automated billing kill switch. The reference run itself was roughly £7. Inference is not the expensive part; uptime is. Cloud Run with CPU always allocated costs continuously regardless of whether anyone visits. The controlling variable is how long you leave the lab standing, not how hard you attack it.

Positioning against FinBot

OWASP runs FinBot CTF, a hosted multi-agent platform with nineteen challenges across six categories. If you want breadth of agentic vulnerability classes in a polished environment, go and play FinBot.

This lab is a different shape. FinBot demonstrates that agents are exploitable, across many classes, at one defence posture per challenge. This demonstrates which class of control removes the exploit, by holding everything else constant and varying only the authorisation architecture. The vulnerabilities here are not primarily in the prompts; they are in the IAM bindings, and the artefact you read is the Terraform. Seven flags against nineteen. Narrow on purpose.

What I'd do differently

Run three providers from the start. The evidence threshold exists for a reason, and designing a project around a threshold you then cannot meet because of a quota refusal is a planning failure. The matrix is permanently weaker for it, and there is no post-hoc fix.

Build the capture detector for F7a (behaviour change, not value disclosure) before committing to score it. F7a's cells carry measurement_limitation: no_independent_detector rather than reporting a hold, because the measurement was never possible. Deciding to measure something you have no instrument for and then publishing the gap is more honest than dropping the flag, but building the instrument first would have been better.

The source code, the full result matrix, and the experimental log are on GitHub.