Run your agent

Versalist helps developers test AI agents, inspect failures, and compare changes before release.

This guide requires an existing agent command. For a complete local example, use the quickstart. The integration below uses an illustrative support request example. Each case has an expected category and priority. Supply your own agent command and adapt the verifier to your chosen challenge. If you would rather start in the browser, skip to Use the web app.

1. Pick a challenge

You do not need an account to browse. A challenge is a fixed task with inputs, expected results, and a scoring rubric. For a useful comparison, keep the test cases, scoring rules, model settings, and runtime settings the same.

npx -y @versalist/cli list
npx -y @versalist/cli list --search "support"

Note the slug of the challenge you want. The same catalog is on the challenges page.

2. Create an API key

1. Sign in with work email, Google, or GitHub.
2. Open Developer API and create a key with read:challenges and submit:solutions. That is all this page needs.
3. Copy the value once. It starts with vk_live_.

export VERSALIST_API_KEY=vk_live_...

Other scopes are explained on the API keys page. You do not need them yet.

3. Load the challenge into your repository

This writes files. It does not run anything.

npm install -g @versalist/cli
versalist start <challenge-slug>

• CHALLENGE.md: the brief your agent should read.
• .versalist.json: the challenge id and local metadata.
• eval/examples.json: the public test cases with expected results, when the challenge publishes them. Private cases are never downloaded.

Point your agent at the working directory. Agents that read the repository pick up CHALLENGE.md without an extra call.

4. Run your agent and record it

Wrap the command you already use. Versalist records the command, exit status, duration, Git revision, and a hash of the challenge files, plus stdout.log and stderr.log.

versalist run --command "python agent.py" --label baseline

Replace python agent.py with your agent command. Versalist does not create this script. For the verifier below, your command must write output.json, an object keyed by case ID. Each value must contain the fields in that case’s expected_output object.

5. Evaluate the run

A local evaluation attaches a verifier result to a recorded run. The command reads files in your current working directory. The verifier exits 0 when every check passes and non-zero otherwise. Write it to print what it compared, because that output is the evidence you will read later.

Save the following script as check_output.py. It requires nonempty test cases with object-shaped expected results. It rejects missing results. For another output format, use a verifier that checks that format.

import json
import sys
from pathlib import Path

cases = json.loads(Path("eval/examples.json").read_text())
actual = json.loads(Path("output.json").read_text())
if not isinstance(cases, list) or not cases:
    sys.exit("FAIL: no test cases")
if not isinstance(actual, dict):
    sys.exit("FAIL: output must be an object keyed by case ID")

failures = 0
seen = set()
for case in cases:
    item_id = case.get("item_id") if isinstance(case, dict) else None
    want = case.get("expected_output") if isinstance(case, dict) else None
    if not isinstance(item_id, str) or not item_id or item_id in seen:
        sys.exit("FAIL: case IDs must be nonempty, unique strings")
    seen.add(item_id)
    if not isinstance(want, dict) or not want:
        sys.exit(f"FAIL: {item_id} has no expected fields")
    got = actual.get(item_id)
    if not isinstance(got, dict):
        sys.exit(f"FAIL: {item_id} has no result object")
    for field, value in want.items():
        ok = field in got and got[field] == value
        failures += int(not ok)
        print(f"{'ok  ' if ok else 'FAIL'} {item_id} {field}: expected {value!r}, actual {got.get(field)!r}")
print(f"{len(cases)} cases checked, {failures} failed field(s)")
sys.exit(1 if failures else 0)

versalist evaluate --run latest --command "python check_output.py"

Illustrative verifier output for three support cases:

ok   case-1 category: expected 'billing', actual 'billing'
FAIL case-1 priority: expected 'high', actual 'medium'
ok   case-2 category: expected 'access', actual 'access'
ok   case-2 priority: expected 'high', actual 'high'
ok   case-3 category: expected 'billing', actual 'billing'
ok   case-3 priority: expected 'low', actual 'low'
3 cases checked, 1 failed field(s)

The full verifier output is in .versalist/runs/<run-id>/evaluations/<evaluation-id>/stdout.log. This is the local counterpart of the failed check you would see on a hosted run. Understand your results covers both.

6. Fix one thing and compare

Change one thing, for example the instruction that decides priority. Then run and evaluate again with the same verifier. The first run is your baseline, the current version. The second is the candidate, the proposed version.

versalist run --command "python agent.py" --label candidate
versalist evaluate --run latest --command "python check_output.py"

versalist compare --baseline <baseline-run-id> --candidate <candidate-run-id>

Use the run IDs from your command output. If a baseline fails and a candidate passes, their default scores are 0 and 100. The comparison reports a 100-point improvement. This example describes the scoring rule. It is not evidence of a measured agent improvement.

decision is one of improved, unchanged, below_threshold, or regressed. The command exits 1 when the candidate fails its verifier or scores lower, so you can use it as a gate in CI. Add --min-delta 5 to require a minimum improvement.

Evidence stays in .versalist/. Run logs stay in your repository.

7. Optional: record a finer score

When your verifier computes a score, pass that value with --score. Create results.json before you attach it with --metrics. The values below are illustrative:

versalist evaluate --run latest --command "python check_output.py" \
  --score 83 --metrics results.json

8. Submit

versalist submit \
  --url https://github.com/you/solution \
  --title "Candidate run"

Submit sends the project URL and metadata, not the contents of .versalist/. One submission per challenge per account. A second submit returns 409.

Use the web app

A hosted run gives you more than a local evaluation: a score for each rubric dimension, results for public test cases, and aggregate progress that includes private cases. Traces require capture and access to be enabled.

1. Sign in. Company workspaces that require approval use enterprise sign-in.
2. Open a challenge on Challenges. Read the task, constraints, and rubric.
3. Select Run, then choose the skill and model. The card shows a Runtime block when the challenge needs a sandbox. If that runtime is unavailable on Versalist, the Run button is disabled and the card says why.
4. Open the result. Score by rubric dimension shows where points were lost, Task-by-task shows public test cases, and Execution trace shows recorded calls when available.
5. Change one thing, run again, and compare the two results.

Connect a coding agent

If you want the challenge tools inside Cursor, Claude Code, or another host, start the Model Context Protocol (MCP) server instead of, or in addition to, the command-line interface (CLI):

npx -y @versalist/cli mcp

Host configs are on coding agents. The tools are listed on MCP tools.

Next steps

• Understand your results: scores, failed checks, execution errors, timeouts, and traces.
• Test a change before you release it: save a failure as a test case, compare baseline and candidate, record the decision.
• Where your agent runs: hosted sandbox, your own machine, and local records.
• CLI reference: every flag, including VERSALIST_BASE_URL for staging.
• API reference: HTTP if you are not using the CLI.