Versalist operates the hosted sandbox. Customers do not configure the hosted worker. Execution is subject to the platform's availability and usage limits. When a challenge needs code execution and that runtime is not enabled on the deployment you are using, the run does not start and the interface says why.
The three options
| Platform execution | Your own machine | Local CLI records | |
|---|---|---|---|
| What runs where | Versalist calls the model, judges the output, and for sandbox challenges executes the generated Python program on a Versalist-operated sandbox. | Your computer runs the model (through Ollama) or your agent. Versalist judges the returned answers and stores the run. | Your computer runs everything. The CLI writes a record of the command, its output, and your verifier result to .versalist/. |
| Start it with | Run on a challenge page, or POST /api/episodes | versalist challenge run … --model ollama:<tag>, or the run protocol at /api/v1/runs | versalist run, versalist evaluate, versalist compare |
| Prerequisites | A signed-in account. For sandbox challenges, the runtime must be enabled on the deployment. | A Versalist API key, Ollama running with a model installed, and a challenge that allows local execution. | Node.js 18 or later and an API key for versalist start. |
| Cases evaluated | Public and private cases (evaluation_scope: full_suite) | Public cases only (evaluation_scope: public_suite) | Whatever your verifier checks. Public cases are in eval/examples.json. |
| Verification status | provenance_level: platform_verified. Versalist made the model call and can attest to the model used. | provenance_level: self_reported. The model identity is your claim; the run is labelled as local execution and stays off the trusted leaderboard. | None. The record is evidence for you and your reviewers, not a platform-verified result. |
| Where the result lives | Run history (/episodes) and the run page | Run history, marked as local execution | .versalist/runs/ and .versalist/comparisons/ in your repository |
Platform execution and the hosted sandbox
Model-only challenges evaluate model output directly. The run card shows no runtime block and execution_mode is llm_only.
Some challenges need the agent's answer to be a program that is executed against each test case. For those, the model is asked for one fenced program, the program runs on the sandbox with the case input on standard input, and the judge scores the observed output. The run records execution_mode: sandbox.
Supported runtimes
| Blueprint sandbox type | Label in the app | Status |
|---|---|---|
none | Model-only evaluation | Available. No sandbox involved. |
cloud_run, python, docker, local | Python sandbox | Runs on the Versalist Cloud Run sandbox worker when the runtime is enabled. All four types resolve to the same restricted Python profile; docker and local do not reproduce your image or your machine. |
e2b | E2B sandbox | Not operated by Versalist. Runs cannot start. |
browser | Browser sandbox | Not operated by Versalist. Runs cannot start. |
api_only | API-only sandbox | Not operated by Versalist. Runs cannot start. |
What the Python sandbox allows
- Python standard library only. No package installation.
- No network. The policy is
deny_alland cannot be changed per challenge. - No mounted files. The program reads the case input from standard input and writes to standard output.
- A fresh, ephemeral filesystem per execution. Nothing persists between cases.
- Program source and case input up to 200,000 bytes each. Standard output up to 200,000 bytes; the last 16,000 bytes of standard error are kept.
- A deadline of 1 to 300 seconds from the challenge's blueprint
timeout_seconds, default 120.
A program that crashes, times out, or exceeds the output cap is scored, not treated as a platform failure. The sandbox action records the status (completed, runtime_error, timeout, or output_limit), and the judge sees that outcome. Worker failures or invalid model or judge responses can fail the episode step.
How availability is shown
The run card has a Runtime block for sandbox challenges. When the runtime is available it reads Python sandbox runs on the Versalist Cloud Run sandbox worker. When it is not, the badge adds unavailable, the Run button is disabled and labelled Sandbox runtime unavailable, and the message explains which of these applies:
| Reason | Message |
|---|---|
runtime_disabled | Python sandbox is currently unavailable on Versalist. Runs for this challenge cannot start until the platform enables its sandbox runtime. |
worker_not_configured | Python sandbox is enabled but no sandbox worker is configured. Runs for this challenge cannot start until an operator sets the worker endpoint. |
unsupported_sandbox_type | E2B sandbox is not operated by Versalist yet. Runs for this challenge cannot start until the platform provides this runtime. |
The same status is in the runtime field of GET /api/challenges/{id}/run-options, and POST /api/episodes returns 409 SANDBOX_RUNTIME_UNAVAILABLE instead of creating a run. The challenge recipe lists the runtime as unavailable on Versalist.
Cancellation
- Cancel a running hosted run from the run page, or with
POST /api/episodes/{id}/cancel. The response is{ status: "cancelled" }, orEpisode is not runningif it had already finished. - Cancellation stops further steps. A sandbox execution that is already in flight is not interrupted immediately; it ends at its own deadline, and the worker then destroys the sandbox.
- A run stopped by a deadline or a lost connection is recorded as such. It does not prove the program was correct or incorrect.
Execution on your own machine
Use this to run an open-weight model on your own computer. Case responses leave your computer for platform evaluation. The CLI sends public cases to Ollama on your computer, returns the answers, and Versalist judges them. Setup and troubleshooting are on Local model runs.
- The platform records the run as local execution with a self-reported model.
- Only public cases are evaluated, so the score is not comparable to a full-suite hosted run.
- The challenge owner must allow local execution. The CLI cannot override that.
- The underlying run protocol at
/api/v1/runsalso acceptsopenai,anthropic,google-ai, andopenai-compatibleproviders for your own runner. Accepting a provider name does not mean Versalist hosts that provider.
Local CLI records
versalist run, evaluate, and compare never call Versalist. They write JSON records and logs under .versalist/ and exit non-zero when a check fails, which makes them usable as a CI gate. They are evidence for your own review; the platform does not verify them and submit does not upload them.
Fixed test environments
A hosted run with an enabled environment binding records an environment version: an immutable, digested description of the runtime, verifier, and evaluator the challenge was bound to when the run started. A shared environment version is one comparison requirement. Also match the cases, model, and generation settings.
Environment definitions and versions are managed through /api/v1/environments. Publishing a version and binding it to a challenge require a signed-in session. Executing a run against a published version through POST /api/v1/environments/runs is gated separately and returns 503 SECURITY_REVIEW_REQUIRED while hosted execution for that path is disabled.
Operating the sandbox yourself
Customers cannot bring their own sandbox worker today. If you run a self-hosted Versalist deployment, the worker's API, isolation controls, deployment steps, and configuration variables are in the operator guide in the repository at docs/sandbox-worker.md. That guide, not this page, is where the Cloud Run commands live.