An agent harness is the code around a model: prompts, tools, control flow, and memory. Teams build harnesses on open-source frameworks such as LangGraph, CrewAI, the OpenAI Agents SDK, Google ADK, and Pydantic AI, or write their own. If your harness starts from a shell command, the Versalist command-line interface (CLI) can record it, check its output, and compare two versions of it.
What Versalist supports
| Surface | What your harness does with it | Status |
|---|---|---|
versalist run | Runs your harness as a shell command and records its output, exit code, Git revision, and challenge file hashes under .versalist/. | Available in @versalist/cli 1.3.0 |
versalist evaluate, versalist compare | Runs your check against a recorded run, then gates the candidate against the baseline. compare exits 1 when the gate fails, so it works in CI. | Available |
versalist mcp | Gives a coding agent challenge context and submission tools over stdio. | Available for supported agents. Pi uses the CLI. See coding agent setup. |
/api/v1/runs | A remote runner claims one public case at a time, sends back one text answer, and Versalist judges it. | Off in production |
| Hosted runs on Versalist | Versalist runs its own runner with the skill and model you choose. It does not run your harness. | Not available for your harness |
Run your harness under the CLI
Load a challenge into your repository, run the current harness as the baseline, run the changed harness as the candidate, check both, and compare. Replace the commands with the ones that start your harness and check its output.
npm install -g @versalist/cli
export VERSALIST_API_KEY=vk_live_your_key
versalist start <challenge-slug>
# Baseline: your harness as it is today
versalist run --label baseline --command "python -m my_harness --cases eval/examples.json --out output.json"
versalist evaluate --command "python check.py output.json"
# Candidate: the same harness with one change to prompts, tools, or model
versalist run --label candidate --command "python -m my_harness --cases eval/examples.json --out output.json"
versalist evaluate --command "python check.py output.json"
versalist compare --baseline <baseline-run-id> --candidate <candidate-run-id>What the CLI expects from your harness
- The command runs from the current directory through your shell. Standard input is not forwarded.
- Exit code 0 means the run succeeded. The CLI records standard output and standard error.
- The default timeout is 900 seconds. Set
--timeoutup to 86,400 seconds. versalist startwritesCHALLENGE.mdand, when the challenge has them, public cases ineval/examples.json. Your harness reads them and writes output in any format your check understands. The CLI does not define the output file.versalist evaluatepasses when your check exits 0. The score is 100 or 0 unless you pass--score.versalist comparepasses when the candidate's check passed and its score changed by at least--min-delta, which defaults to 0.
Frameworks
Versalist has no adapter for any framework. A harness works with the CLI when it starts from a shell command, whichever framework it uses. The CLI does not inspect the framework, so there is nothing to install inside your harness.
versalist submit accepts --toolchain to record the framework you used. It is a label on the submission, not an integration.
Not supported yet
- Framework adapters or a Python SDK.
- Traces from your harness. Versalist does not accept traces, including OpenTelemetry. Traces exist only for runs on the platform; see Trace capture.
- Running your harness on Versalist's hosted runner.
- Platform-verified scores for local runs. Records under
.versalist/are evidence for you and your reviewers. Versalist does not verify them, andsubmitdoes not upload them.