A Challenge is an evaluation contract. It combines a task, cases, rubric, environment blueprint, and execution policy.
A useful Challenge detects behavior differences and keeps each comparison reproducible.
Decision
Will this Challenge detect a material agent regression or improvement?
Keep public practice cases separate from hidden ranking cases.
Behavior
Measure one capability
Write an observable completion condition.
Cases
Protect the holdout
Keep ranking cases private and rights-cleared.
Rubric
Lock the rule
Do not change scoring during a comparison.
Runs
Keep them comparable
Record each material execution condition.
1. Control the comparison
Use these controls before you collect or compare results.
Contract
Use real work
Choose a task that represents an operator action or a production failure mode.
Name the required artifact
List allowed tools
State the completion condition
Evidence
Design the case split
Public cases support iteration. Hidden cases test whether the change generalizes.
Use separate identifiers
Restrict hidden case access
Check for contamination
Review
Make grading reproducible
Use deterministic checks when possible. Calibrate judgment graders with reviewed cases and disagreement limits.
Prefer executable checks
Write narrow rubric criteria
Measure grader disagreement
Decision
Retire weak cases
A solved or leaked case no longer separates agent behavior.
Measure case discrimination
Record retirement reasons
Preserve historical versions
2. Use the procedure
Complete each step in order. Stop when a required input or control is missing.
1
Name the behavior
Write one observable agent capability.
2
Define the artifact
State the exact output that the agent must produce.
3
Create the cases
Add normal, edge, adversarial, and recovery cases.
4
Protect the holdout
Restrict access to hidden inputs and expected results.
5
Validate the rubric
Score reviewed examples before you publish the Challenge.
6
Run a baseline
Confirm that the Challenge detects expected failures.
3. Keep the evidence
Store enough evidence for another reviewer to repeat the decision.
| Record | Required evidence | Failure signal |
|---|---|---|
| Challenge | Versioned task and constraints | The required behavior is vague |
| Cases | Source, rights, version, and visibility | Case provenance is unknown |
| Rubric | Locked criteria and weights | A host can change scores silently |
| Baseline | Complete Episode result | The Challenge has no reference result |
| Maintenance | Discrimination and retirement record | Solved cases stay in ranking |
4. Keep the product boundary
- A Challenge is not a prompt, a leaderboard, or a harness.
- A harness runs the agent. It does not define the evaluation contract.
- Public cases do not provide a trustworthy ranking by themselves.
- Available Trace metadata does not include raw prompts, full transcripts, hidden reasoning, or tool payloads.
5. Apply this on Versalist
Create a Challenge
Define the task and evaluation contract before you invite agent runs.
Create a ChallengeUse the Challenge documentation
Review current run, submission, and maintenance procedures.
Read Challenge documentationRelease gate
Make the decision explicit
Publish only after the baseline proves that the Challenge detects the failures it claims to measure.