Agent evaluation must support a change decision. A score alone cannot show why behavior changed.
Bind each result to the task, cases, rubric, agent policy, model, and available execution evidence.
Decision
Can this evidence justify release, rejection, or more testing for one agent change?
Compare one candidate with one baseline under the same evaluation contract.
Task
Version the behavior
Keep the required result and constraints stable.
Cases
Test known risks
Include normal, edge, and failure cases.
Graders
Define the score
Prefer deterministic checks when the task permits them.
Decision
Record the outcome
State release, reject, or test again.
1. Control the comparison
Use these controls before you collect or compare results.
Contract
Keep the contract fixed
A comparison is valid only when both versions use the same task, cases, rubric, and run conditions.
Hash or version each input
Record the model and provider
Record evaluator changes
Evidence
Layer the graders
Use the least subjective grader that can measure the required behavior.
Run schema and execution checks first
Use rubric graders for judgment
Validate model graders with reviewed cases
Review
Inspect failure groups
Aggregate scores can hide severe regressions in a small but important case group.
Name each risk group
Compare pass rates by group
Review changed failures
Decision
Separate proof levels
Local output, hosted Episode evidence, and verified deployment evidence support different claims.
Label local results
Keep external records separate
Require deployment proof for release claims
2. Use the procedure
Complete each step in order. Stop when a required input or control is missing.
1
Define one behavior
Write the required result and all constraints.
2
Select representative cases
Include normal inputs, edge cases, and known failures.
3
Define each grader
State its inputs, output, threshold, and known limits.
4
Run the baseline
Keep the first complete result as the comparison record.
5
Run the candidate
Use the same contract and run conditions.
6
Record the decision
Link the decision to the results and reviewer.
3. Keep the evidence
Store enough evidence for another reviewer to repeat the decision.
| Record | Required evidence | Failure signal |
|---|---|---|
| Task contract | Task version or digest | The task changed during comparison |
| Case set | Case identifiers and visibility | The candidate saw hidden cases |
| Rubric | Rubric version and grader settings | The grading rule changed |
| Execution | Model, provider, harness, and timestamps | Run conditions are unknown |
| Decision | Reviewer, reason, and rollback rule | The score has no owner or action |
4. Keep the product boundary
- Do not compare results from different task or rubric versions as one experiment.
- Do not use a model grader without reviewed validation cases.
- Do not describe local results as platform-verified evidence.
- Do not approve a change from an aggregate score alone.
5. Apply this on Versalist
Run a versioned Challenge
Use one Challenge to keep the task, cases, rubric, and execution policy connected.
Browse ChallengesInspect hosted results
Review Episode results and available Trace metadata before you approve a change.
Read result documentationRelease gate
Make the decision explicit
Release only when the candidate passes the required groups without an unexplained regression.