A demo airline support agent built on Sierra's open τ-bench domain, with a workbench that shows every tool call. An experiment in making agents dependable.
Scored 0.800 pass^1 on a five-task slice. The methodology is in the repo.
Refuses off-policy requests and cites the policy when it does.
A demo agent for a made-up New Jersey insurer. It quotes new customers and services existing ones, using the same pricing engine as a voice quote line I made.
Passed 36 of 36 trials across 12 tasks (pass^3 of 1.000), graded on the final record.
Verification and explicit consent are enforced in code.
An open source harness that runs state-verified tasks against any chat model and reports pass@k and pass^k. Tasks are graded on the final database state.
Claude Haiku passed 20 of 20 trials. Llama 3.1 8B passed 12 of 20.
Caught a small model reporting a cancellation it never made.