Understand Your Results

A run is one recorded attempt by an agent against a challenge. The API calls it an Episode. A run records its status and available evaluation results. Failed or cancelled runs can have incomplete results. Authorized capture can add a trace. This page reads one run from top to bottom, then explains retention and access.

Two kinds of result

Result	Produced by	What you get
Hosted run	Run in the web app, or POST /api/episodes	Rubric score per dimension, public test-case results, aggregate scores, status, available duration and errors, plus authorized trace metadata.
Local evaluation record	versalist evaluate in your repository	passed (from the verifier exit code), a score of 100 or 0 unless you pass --score, duration, and the verifier logs. The CLI stores these records locally. Your command can still contact external services.

The rest of this page is about hosted runs. The local record is covered on the quickstart and the CLI reference.

One run, annotated

This illustrative response excerpt follows the shape of GET /api/episodes/{id} for a hypothetical support-request evaluation. It is not a captured customer run. The run page presents results in Score by rubric dimension, Task-by-task. Execution trace loads separate trace data when available.

{
  "id": "2f6c…",
  "status": "completed",                 // pending | running | completed | failed | cancelled
  "score_percentage": 66.7,              // 100 * total_score / max_possible_score
  "total_score": 4,
  "max_possible_score": 6,
  "dimension_scores": [
    { "dimension_id": "category_accuracy", "label": "Category accuracy", "type": "binary",  "score": 3, "max_score": 3 },
    { "dimension_id": "priority_accuracy", "label": "Priority accuracy", "type": "ordinal", "score": 1, "max_score": 3,
      "reasoning": "case-1 labelled medium; policy requires high for refunds older than 7 days" }
  ],
  "execution_mode": "llm_only",          // llm_only | sandbox
  "execution_origin": "platform",
  "provenance_level": "platform_verified",
  "evaluation_scope": "full_suite",      // public and private cases
  "total_duration_ms": 18420,
  "error_message": null,                 // set when the run itself failed
  "steps": [                             // public test cases only
    { "gold_item_id": "…", "status": "completed", "step_score": 1, "step_max_score": 2,
      "agent_output": "category: billing; priority: medium",
      "evaluator_output": "Judge verdict text for this public case" },
    { "gold_item_id": "…", "status": "completed", "step_score": 2, "step_max_score": 2 },
    { "gold_item_id": "…", "status": "completed", "step_score": 1, "step_max_score": 2 }
  ],
  "total_step_count": 3,
  "completed_step_count": 3
}

Field	How to read it
`status`	completed means every step finished and was scored. failed means the run could not finish. cancelled means you or the platform stopped it. pending and running are in progress.
`score_percentage`	The headline number. It hides where points were lost; read dimension_scores and steps before you act on it.
`dimension_scores`	One row per rubric dimension, with the judge reasoning when there is one. In the example the agent gets categories right and priorities wrong, which points at one instruction to fix.
`steps`	One row per public test case. A completed step with step_score below step_max_score is a wrong or partial answer; agent_output is what the agent produced and evaluator_output is the verdict. A failed step is a case where the agent could not produce a scorable answer. Private case payloads are excluded even for the owner. Owner progress totals include all cases.
`error_message`	Only on failed runs, and only visible to the owner. It describes why execution stopped, for example a provider timeout, not why an answer was wrong.
`provenance_level`	platform_verified means Versalist ran the model call. self_reported means the run came from your machine and the model identity is your claim.

A wrong answer is not a broken run

Keep these two failure kinds apart, because they call for different fixes. Changing an instruction does not fix a timeout, and raising a timeout does not fix a wrong answer.

Symptom	Where it shows	Recorded as	Usually fixed by
Wrong or partial answer	Step completed, step_score below step_max_score; dimension score below its max	`rubric_shortfall`	Changing the skill or instructions, then re-running the same cases
Agent could not produce a scorable answer	Step failed; the hosted episode fails when a step cannot be evaluated	`execution_error`	Checking the provider, model, or output format before touching the instructions
Sandbox check failed on the program output	Sandbox and verifier runs	`assertion_failed`	Fixing the program logic the agent produced
Program crashed	Sandbox and verifier runs	`runtime_error`	Reading stderr; often an import or an unhandled input
Program exceeded its deadline	Sandbox and verifier runs	`timeout`	Checking for loops or blocked I/O first; the sandbox has no network

Saved execution records (see below) also carry a classification from the failure-v1 taxonomy: assertion_failed, timeout, runtime_error, provider_error, policy_denied, invalid_output, or unknown. Use GET /api/v1/corpus/failures to list them by challenge, model, or environment version.

Timeouts you can hit

Limit	Value	Applies to
Sandbox program deadline	1 to 300 seconds, default 120	The challenge's environment blueprint timeout_seconds; enforced inside the sandbox.
Sandbox output	200,000 bytes stdout; last 16,000 bytes of stderr	Excess stdout stops execution with output_limit. Excess stderr retains its tail without changing a successful status.
Local agent command	900 seconds by default, max 86,400	versalist run --timeout
Local verifier command	900 seconds by default, max 86,400	versalist evaluate --timeout
Authenticated HTTP call from the CLI	20 seconds	Any CLI request to versalist.com

Traces

A trace is a best-effort, ordered record of what happened during a run, as metadata. When capture is enabled, it contains the agent turns, the agent model calls, and the judge calls. The run page summarises it as Wall time, Calls, Tokens, Est. cost, and Errors, with provenance rows for the skill version, challenge, rubric, cases, and model contract hashes.

For a sandbox run, the trace also has a sandbox_action event with the execution status, exit code, duration, byte counts, and a hash of the payload. The sandbox event excludes program source and output. Episode result storage follows separate rules.

What a trace does not contain

Hidden model reasoning.
Tool calls made by an agent running outside Versalist.
Prompt and completion text in trace metadata. Separate retained payloads can be available for eligible execution records.
Events that were dropped. Execution record summaries report dropped_event_count and observed_drop_rate against a target of 1 percent. Treat a record above that as incomplete evidence.

What is kept, and for how long

Two separate things are stored, with separate rules. Read both before you run anything containing customer data.

Run metadata

Scores, statuses, durations, model and skill identifiers, content hashes, and trace event metadata are stored with the run. They are what the run page and run history show, and what comparisons are built from. There is no user-facing setting to shorten this.

Raw content

The corpus payload policy controls an additional copy of execution content. Its default, metadata_only, retains digests without uploading raw payloads. This policy does not remove stored episode outputs, evaluator text, local logs, or saved test-case inputs. Public-case episode outputs and evaluator text can be stored independently of trace capture. Corpus execution also holds temporary input while a job runs. Completion or scheduled cleanup removes that temporary input. Eligible execution records can retain additional payloads under these modes:

Mode	What is stored	Retention
`metadata_only`	Digests only. Default.	Not applicable
`redacted`	The payload after credential, email, identifier, and sensitive-key redaction (corpus-redaction-v1).	1 to 30 days, default 7
`encrypted_raw`	The original payload, encrypted with AES-256-GCM.	1 to 30 days, default 7

Expired payloads are deleted by a scheduled job. The digest stays.
A payload can be read back only by its owner, while the policy that allowed it is still active, and only from a signed-in browser session. Reads return 403 with Payload consent is no longer active after you tighten the policy.
Redaction removes known credential and identifier patterns. It cannot guarantee that all personal or confidential information is removed.
This retention workflow does not automatically publish payloads or send them for training.

When you save a failure as a test case (see Test a change before you release it), the case input you paste is kept as private evaluation content. That is separate from payload retention and does not expire with it.

Availability

The payload policy is read and set through GET and PUT /api/v1/corpus/payload-policy from a signed-in session. There is no settings page for it yet. Both retention modes also require an environment manifest that permits retention and an active trace-capture authorization; without those, runs stay at metadata_only regardless of the policy.

Who can see a run

You, the owner, see public-case outputs, available error details, and run identifiers. Private-case payloads remain hidden. Progress totals include private cases.
Other viewers can read an accessible public run's scores, status, timing, labels, hashes, and public-step summaries. Outputs, evaluator text, and error messages are withheld.
Your company workspace scopes saved execution records, comparisons, and release decisions. Switching the active company changes which records the API returns.
Local CLI records in .versalist/ are visible to whoever can read your repository. Review them before sharing; command output can contain secrets.

Test a change before you release it: turn a failed case into a regression test and compare two versions.
Where your agent runs: why a run says execution_mode: sandbox and what that runtime allows.
API reference: runs: the full response shape and error codes.