From a fixed task to a release decision
Define a task, evaluate agent behavior, inspect evidence, and compare changes before release.
Four tracks, one loop
Work moves through them in this order, and the last one feeds the first.
- Challenges
- Fix the task, the cases and the rubric. A challenge turns one real workflow into test cases and a locked rubric. Two runs can only be compared when both used the same ones.
- Skills
- Version what your agent is told. A skill is the instructions, scripts and references an agent works from. Each published revision is pinned, so you know which one produced a result.
- Continuous Evals
- Compare two revisions and record the decision. A release review puts the current and proposed revision side by side, case by case. The review owner records Ship, Hold or Needs work.
- Continuous learning
- Turn a failed case into the next revision. A failure becomes a proposed skill amendment. The amendment runs against the baseline on cases it has not seen, and a person promotes it only if the comparison supports it.
Four terms
- Challenge
- A fixed task. It defines the inputs, the expected behavior and how results are scored.
- Episode
- One scored run of a skill and a model against the challenge cases.
- Evidence
- Scores per rubric dimension and per case. With trace capture on, bounded call metadata too.
- Comparison
- Run the changed skill on the same cases. A person reads both results and decides what ships.
Trace events are best-effort; a missing event proves nothing. Capture limits · Where your agent runs
Continuous learning
The model behind your agent does not learn from yesterday’s failure. Its skill can, one tested amendment at a time.
- ObserveImport the execution receipts from a completed episode and read what failed.
- ProposeAn improvement cycle drafts an amendment: the skill version it starts from, the new content and the reason.
- TestBaseline and candidate run on a random sample of acceptance cases. Cases that produced the proposal are excluded.
- PromoteYou promote the amendment when the comparison supports it. An inconclusive result needs your explicit confirmation.
The skill changes, not the model weights. A proposal alone cannot authorize promotion: the comparison has to support it. The workflow runs on personal skills today. Read the improvement workflow
Challenges, docs and guides are public. You sign in when you want to run or save something.