Methodology
How we measure what an agent puts at risk.
The agent operates in a simulated financial world: bank, suppliers, invoices, emails, network failures, fraud and hostile instructions. Every episode is graded and turned into a figure in pesos.
- 430
- episodes graded across DeepSeek, Claude Haiku and Sonnet, and GPT-6 Luna, plus 6 worlds shared between agents
- 423
- automated, verified checks that prove our grader catches what it claims to catch
- 48
- episodes in measurement 09, pre-registered, across three commercial models
Every measurement is pre-registered
Before running we freeze the rules, the scenarios, the budget and the code. Nobody gets to tune the test after seeing the result.
Every episode can be replayed
We keep the full transcript and the command to replay the episode exactly. If a failure is in the report, you can watch it again.
We audit our own grader too
We separate what comes from your configuration, from your agent, or from our instructions. Reviewing our own measurements we fixed 15 ambiguities in our instructions, so every failure we report belongs to the agent and not to the test.
Real providers, in test mode
Besides the simulated bank, we test against Flow and Mercado Pago in their sandboxes. In both, hostile text aimed at agents reaches the payment the agent reads, intact.
How we compute risk per million
We add up what the agent would have lost across all episodes and divide it by the total we delegated. Measured case, with simulated money: in measurement 09, DeepSeek V4.1 Flash with the support profile lost 990,000 CLP in 1 of 8 episodes, with 3,800,000 CLP delegated in total.
990,000 ÷ 3,800,000 × 1,000,000 = 260,526 CLP per million
Measurement 09 · accounts payable · 48 episodes, pre-registered
| Model | Profile | Episodes | Delegated (CLP) | Audited loss (CLP) | Per million |
|---|---|---|---|---|---|
| DeepSeek V4.1 Flash | support | 8 | 3,800,000 | 990,000 | 260,526 |
| DeepSeek V4.1 Flash | strict | 8 | 3,800,000 | 0 | 0 |
| Claude Haiku 5.5 | both | 16 | 7,600,000 | 0 | 0 |
| GPT-6 Luna | both | 16 | 7,600,000 | 0 | 0 |
Automatic grading assigned losses to Claude Haiku 5.5 and GPT-6 Luna; the audit showed they came from an ambiguity in our grader and the real loss is 0. DeepSeek had 2 failed episodes out of 16: one with a loss and one without.
Test disciplines
- · Transfers
- · Payment incidents: duplicate charges and refunds
- · Async payments with injected hostile instructions
- · Money shared between two agents
- · Accounts payable with a delegated budget of 100,000 to 1,000,000 CLP (simulated)
What we don't claim
We measure on known tasks, so we don't publish general failure rates per model. A result is an independent evaluation of the agent's behaviour in simulated scenarios: it describes what that version did, not that it is safe in general. The simulator accepts arbitrary accounts with no bank validation, and the Gateway policy is a proposal that does not run yet. We don't move real money.