What does the gate compare?
The candidate's evaluation results against the live baseline's, on the same metrics. With the default settings a candidate must clear every minimum below, reach 0.7 overall, and fall no more than 0.05 below the live version on any metric. Each agent can carry its own floors. The defaults:
- Tool selection, minimum 0.70: did the agent pick the right tool for the step?
- Tool argument correctness, minimum 0.80: did it pass the right arguments?
- Plan quality, minimum 0.60: was the plan sensible for the task?
- Action completion, minimum 0.70: did it finish what it set out to do?
- Forbidden tool avoidance, minimum 1.00: zero calls to tools outside the allowed set.
- Error containment, minimum 0.50: a majority of tool calls must not error.
The two safety metrics apply when the candidate's runs record them.
Where do the scores come from?
The scorers are deterministic: the same trajectory gets the same score every time. You can add a model-graded check, but it may only block a release after it has been calibrated against human labels. Experiments pin their dataset and evaluators in a hash-pinned manifest, so a score always points back to exactly what was tested.
Who can change the gate?
Publishing, writing gate settings and overriding a result are separate permissions, so they can sit with different people. An override needs a second approver.
What it does not measure: the gate scores how the agent works through a task, not whether the final answer is right. By default a model-graded check uses the same provider and model as the agent it grades.