Test a change before you release it

A skill is a versioned set of instructions your agent follows. Improving an agent on Versalist means changing a skill and evaluating the change against the current version. This guide explains how to evaluate one skill change.

Two terms carry the whole page. The baseline is the version you run today. The candidate is the proposed version. Both are evaluated on the same test cases with the same evaluator, and the candidate is released only when the comparison supports it.

The example

A support-triage skill labels each incoming request with a category and a priority. On the run from Understand your results, the illustrative case-1 (a refund not received after ten days) came back with priority medium; the expected value is high. Category accuracy was fine.

1. Find the failure

1. Open the run from run history.
2. In Task-by-task, open case-1. The step completed with a partial score, so this is a wrong answer, not an execution error. If the step had failed, fix the execution problem first; a skill change would not address it.
3. In Execution trace, open the judge call for that case. Trace metadata identifies the call. Read available judge reasoning in the public step result, rather than expecting reasoning text in the trace.

2. Save the failure as a test case

A saved case is a finding plus a private test case built from the exact original input. Include that saved case explicitly when you register the comparison.

On a failed trace event, expand Review failure as an evaluation case, paste the exact Original case input, write a Review note that says what was wrong, and select Save reviewed case. The confirmation reads Reviewed finding and evaluation case saved.

The same action over HTTP:

curl https://versalist.com/api/v1/governance/findings \
  --header "x-api-key: $VERSALIST_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "trace_event_id": "9d4e…",
    "case_input": "Subject: Refund not received\n\nI requested a refund 10 days ago and …",
    "review_note": "Priority should be high: money owed for more than a week."
  }'
# 201 { "finding_id": "…" }

The pasted input is kept as private evaluation content for your workspace. It is separate from trace payload retention and does not expire with it.

3. Propose the change

Make the smallest instruction change that addresses the finding. For the example, add one rule to the priority section of the skill: unresolved money owed for more than seven days is high. A proposed change is recorded as an amendment: the skill it changes, the version it starts from, the new content, and the rationale.

curl https://versalist.com/api/v1/governance/amendments \
  --header "x-api-key: $VERSALIST_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "finding_id": "…",
    "parent_id": null,
    "resource_type": "skill",
    "resource_id": "<skill-bundle-uuid>",
    "base_version_id": "<current-skill-version-uuid>",
    "proposed_content": { "skill_md": "<complete updated skill instructions>" },
    "rationale": "Refund delays over 7 days were labelled medium; the policy says high."
  }'
# 201 { "amendment_id": "…" }

Record one change per amendment. A comparison can only tell you whether the whole candidate is better; if you bundle three edits, you will not know which one helped.

4. Evaluate baseline and candidate

Hold everything constant except the skill:

• the same test cases, including the one you just saved,
• the same environment version and evaluator,
• the same model, pinned to a dated snapshot, and the same seed,
• the same token limits.

In the web app, run the challenge twice from the same challenge page, once with the baseline skill version and once with the candidate, and open both runs side by side. This is enough for a first look, but it is two hosted runs on the public and private cases, not a registered comparison.

For a registered comparison, create one execution sweep per version with defer_execution: true, then register the comparison before either sweep runs. Registering first is what lets Versalist attest that the decision policy was fixed before the results existed.

# Illustrative one-case request. Use at least five paired cases for promotion; repeat with a distinct idempotency key for the candidate
curl https://versalist.com/api/v1/corpus/sweeps \
  --header "x-api-key: $VERSALIST_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "idempotency_key": "triage-baseline-2026-09-07",
    "targets": [
      { "episode_id": "<baseline-episode-uuid>", "environment_version_id": "<env-version-uuid>",
        "task_key": "case-1", "input": "Subject: Refund not received …" }
    ],
    "models": ["gpt-4.1-2025-04-14"],
    "seed": 0,
    "defer_execution": true
  }'
# 202 { "sweep_id": "…" }

# Read each sweep to obtain its rollout IDs. Read findings to obtain case_id.
# Pair every saved case before either deferred sweep executes.
curl https://versalist.com/api/v1/governance/comparisons \
  --header "x-api-key: $VERSALIST_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "amendment_id": "…",
    "pairs": [
      { "case_id": "<case-uuid>", "baseline_rollout_id": "<uuid>", "candidate_rollout_id": "<uuid>" }
    ]
  }'
# 201 { "comparison_id": "…" }

Each execution record (the API calls it a rollout) moves through queued, claimed, running, verifying, and completed, or ends failed or cancelled. Poll GET /api/v1/corpus/sweeps/{id} for progress and GET /api/v1/corpus/rollouts/{id} for one record's rewards, cost, and trace summary.

5. Inspect the comparison

Request a decision. Versalist applies a pre-registered policy to the paired records and returns a verdict with the reasons that produced it.

curl -X POST https://versalist.com/api/v1/governance/comparisons/<comparison-id>/decision \
  --header "x-api-key: $VERSALIST_API_KEY"

{
  "decision_id": "…",
  "result": {
    "decision": "manual_review",
    "reasons": ["score_improvement_below_threshold", "cost_evidence_unavailable"]
  },
  "policy": { "id": "evaluation-decision-policy-v1", "version": "1.0.0" },
  "policy_digest": "…",
  "evidence_digest": "…"
}

Read the comparison in this order, and stop at the first problem:

Cost is reported when the environment has no model judge; with a judge component the cost field is null and the decision says cost_evidence_unavailable. Latency is completed_at minus started_at on each record.

The verdict is promote, reject, or manual_review. A reject names the regression (overall_score_regressed, dimension_score_regressed, worst_case_score_regressed, candidate_required_component_failed, or candidate_added_failed_cases). Manual review can mean missing evidence, insufficient improvement, or a cost or latency limit was exceeded. Read the reasons before proceeding.

6. Record the release decision

A release records that a decision was acted on. It points at the decision, at the release it supersedes, and at the action: promote the candidate or rollback to the previous release. The response includes a signed package you can store with your own change record.

curl https://versalist.com/api/v1/governance/releases \
  --header "x-api-key: $VERSALIST_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{ "decision_id": "…", "previous_release_id": null, "action": "promote" }'
# { "release_id": "…", "package": { "manifest": {…}, "manifest_digest": "…", "signature": "…", "public_key": "…" } }

# later: export the whole lineage of an amendment for an audit
curl "https://versalist.com/api/v1/governance/amendments/<amendment-id>/export" \
  --header "x-api-key: $VERSALIST_API_KEY" --output governance-lineage.json

For rollback, use the decision for a previously approved release of the same resource. Set previous_release_id to the current release. A release records authorization and returns a package. Deploy that package through your own deployment process.

The same loop from the terminal

If you are not using hosted runs, the CLI gives you the same discipline with local records: keep the failed case in your test set, run baseline and candidate with the same verifier, and let compare gate the change.

versalist run --command "python agent.py" --label baseline
versalist evaluate --run latest --command "python check_output.py"
# edit the skill
versalist run --command "python agent.py" --label candidate
versalist evaluate --run latest --command "python check_output.py"
versalist compare --baseline <baseline-run-id> --candidate <candidate-run-id> --min-delta 0

compare exits 1 on regressed or below_threshold, so it can block a merge. The decision is yours to record in your own change log; nothing is signed or uploaded. See the quickstart for the annotated output.

Skill bundle contents

A bundle can include:

• SKILL.md instructions
• Reference files, scripts, and examples
• Version and provenance metadata

Public bundles are in the skill catalog. To search, pull, or publish from a terminal, use vskill with the read:skills and write:skills scopes. Inspect every executable file before you install or run a bundle, run scripts with the minimum permissions, and never put credentials in bundle files.

Learning tracks and certificates

Learning tracks explain concepts. Skill bundles contain reusable instructions. These are different artifacts. Progress records completed activities and Certificates record completed track requirements.

Related guides

• Understand your results: how to tell a wrong answer from a broken run before you save it as a test case.
• API reference: comparisons and release decisions: every field, error code, and scope.
• Evaluation loop: the model behind this workflow.
• Vera Workbench can draft and review skills from completed run evidence.