Skip to content
Initialising environment000

01 / Products

Environment Audit

A pentest for reward functions.

Scroll to explore

01 / Problem

The problem.

Labs buy environments by the thousand and reject on verifier quality. A hackable grader trains the wrong behaviour, inflates metrics, and costs the vendor a resubmit cycle with their only customer.

Most teams catch exploits by a human reading grader code and imagining attacks.

02 / Deliverables

What you get.

  • 01Exploit pack with one-command reproductions
  • 02Severity-scored findings
  • 03Hardened-grader PRs that block every confirmed exploit while the reference solution still passes
  • 04Full re-test with a before/after table
  • 05A 6–10 page vendor-shareable report
  • 06Debrief

03 / Steps

How it runs.

  1. Kickoff and scope

  2. Baseline with reference and known-wrong solutions

  3. Checklist and sandbox attacks

  4. Adaptive attacks with a red-team model and an RL policy trained against the grader

  5. Fixes as pull requests

  6. Re-test and report

04 / Coverage

What we test for.

Exploit classes we test for, what each exploit does, and its severity
ClassWhat the exploit doesSeverity
Verifier bypassThe grader pays without checking the workCritical
Test gamingPasses the tests, fails the taskHigh
Degenerate completionsSurface-form matches, format-only answersHigh
State and loop exploitsReplay, idling, early terminationMedium–High
Sandbox escapeTampering with the grader, reading reference solutionsCritical
Judge manipulationPrompt injection into an LLM judgeHigh
Specification holesWrong solutions accepted, gold solution failsMedium

Founding customers

We are taking two founding customers per product: a full engagement at no cost in exchange for a named logo and a case study. Write to us.

Write to us
Next product · 02Agent Sandbox Evals