Skip to main content

From a fixed task to a release decision

Define a task, evaluate agent behavior, inspect evidence, and compare changes before release.

Four tracks, one loop

Work moves through them in this order, and the last one feeds the first.

Challenges
Fix the task, the cases and the rubric. A challenge turns one real workflow into test cases and a locked rubric. Two runs can only be compared when both used the same ones.
Skills
Version what your agent is told. A skill is the instructions, scripts and references an agent works from. Each published revision is pinned, so you know which one produced a result.
Continuous Evals
Compare two revisions and record the decision. A release review puts the current and proposed revision side by side, case by case. The review owner records Ship, Hold or Needs work.
Continuous learning
Turn a failed case into the next revision. A failure becomes a proposed skill amendment. The amendment runs against the baseline on cases it has not seen, and a person promotes it only if the comparison supports it.

Four terms

Challenge
A fixed task. It defines the inputs, the expected behavior and how results are scored.
Episode
One scored run of a skill and a model against the challenge cases.
Evidence
Scores per rubric dimension and per case. With trace capture on, bounded call metadata too.
Comparison
Run the changed skill on the same cases. A person reads both results and decides what ships.

Trace events are best-effort; a missing event proves nothing. Capture limits · Where your agent runs

Continuous learning

The model behind your agent does not learn from yesterday’s failure. Its skill can, one tested amendment at a time.

  1. ObserveImport the execution receipts from a completed episode and read what failed.
  2. ProposeAn improvement cycle drafts an amendment: the skill version it starts from, the new content and the reason.
  3. TestBaseline and candidate run on a random sample of acceptance cases. Cases that produced the proposal are excluded.
  4. PromoteYou promote the amendment when the comparison supports it. An inconclusive result needs your explicit confirmation.

The skill changes, not the model weights. A proposal alone cannot authorize promotion: the comparison has to support it. The workflow runs on personal skills today. Read the improvement workflow

Challenges, docs and guides are public. You sign in when you want to run or save something.