01 / Products
Environment Audit
A pentest for reward functions.
Scroll to explore
01 / Problem
The problem.
Labs buy environments by the thousand and reject on verifier quality. A hackable grader trains the wrong behaviour, inflates metrics, and costs the vendor a resubmit cycle with their only customer.
Most teams catch exploits by a human reading grader code and imagining attacks.
02 / Deliverables
What you get.
- 01Exploit pack with one-command reproductions
- 02Severity-scored findings
- 03Hardened-grader PRs that block every confirmed exploit while the reference solution still passes
- 04Full re-test with a before/after table
- 05A 6–10 page vendor-shareable report
- 06Debrief
03 / Steps
How it runs.
Kickoff and scope
Baseline with reference and known-wrong solutions
Checklist and sandbox attacks
Adaptive attacks with a red-team model and an RL policy trained against the grader
Fixes as pull requests
Re-test and report
04 / Coverage
What we test for.
| Class | What the exploit does | Severity |
|---|---|---|
| Verifier bypass | The grader pays without checking the work | Critical |
| Test gaming | Passes the tests, fails the task | High |
| Degenerate completions | Surface-form matches, format-only answers | High |
| State and loop exploits | Replay, idling, early termination | Medium–High |
| Sandbox escape | Tampering with the grader, reading reference solutions | Critical |
| Judge manipulation | Prompt injection into an LLM judge | High |
| Specification holes | Wrong solutions accepted, gold solution fails | Medium |
Founding customers
We are taking two founding customers per product: a full engagement at no cost in exchange for a named logo and a case study. Write to us.
Write to usNext product · 02Agent Sandbox Evals