Test the current version before you change it. Keep the task and checks fixed so you can compare the results.
Evaluation sequence
- Define the challenge. Record inputs, constraints, required artifacts, and acceptance criteria.
- Run your agent. Execute the skill and model. Record outputs and evaluation results.
- Judge the result. Apply tests, rubric criteria, baseline comparisons, or reviewer decisions.
- Record the result. Store the score, result status, failure mode, and supporting evidence.
- Update the skill. Change an instruction only when the evidence supports the change.
Artifacts
Challenge
A challenge defines a repeatable task environment. For a useful comparison, also keep the cases, evaluator, model settings, and runtime settings fixed.
Run
A run records one skill and model attempt. It contains outputs and evaluation results. It can contain bounded trace metadata when capture is enabled.
Evaluation
An evaluation applies a defined method to a run result. Use deterministic tests when the task permits them.
Evaluation result
An evaluation result records the outcome of a check. It can contain a score, pass state, failure category, and evidence links.
Skill version
A skill version contains reusable agent instructions. Its provenance must identify the evidence that caused the change.
Compare runs
- Keep the challenge definition unchanged.
- Keep the evaluator unchanged.
- Change one agent variable.
- Run the candidate.
- Compare scores, failures, artifacts, and available bounded trace metadata.
- Accept or reject the change.
A score change can hide a regression. Review the result and available execution metadata.
Provider roles
Provider keys
Provider keys let Versalist send supported model calls through a user-managed account. Manage these keys in Integrations.
Custom model endpoints
This capability is planned. The current product does not expose a general endpoint adapter.
Compute adapters
This capability is planned. A catalog entry does not mean that a runtime adapter exists.
Related guides
- Challenges explains environment selection and creation.
- Skill bundles explains skill inspection and reuse.
- CLI explains local run and comparison records.