Evaluation Generation as a Post-Selection Step

Challenges move through the Versalist data pipeline in three stages, dev → review → production, and evaluation generation is supported at two points along the way: automatically during daily enrichment, and on demand at promotion time. In practice we skip the first and rely on the second. The blunt reason is economics, most generated challenges never clear review, and the structural reason is freshness, an evaluation generated after selection describes the challenge as it will actually ship rather than an early draft. This note documents the mechanism as built, the regenerate-not-edit practice, and the distinction we think matters most: the generated evaluation module is not the scoring rubric.

When evaluation criteria are written at the same time as the challenge draft, the criteria lock to the draft. Review then sharpens the challenge, and the criteria drift: they score for things the final task no longer asks, and miss constraints that review added. The drift is structural, so the fix should be structural too, not an editorial reminder to keep two artifacts in sync by hand.

There is also a plainer problem. The daily pipeline generates far more challenges than will ever be promoted. Evaluations authored for challenges that die in review are pure waste, which is why generating them for the whole daily batch is supported but deliberately skipped.

Evaluation generation sits between selection and promotion, and it is selective on purpose.

• 01Selection comes first: an admin reviews the dev batch and exports the IDs of the challenges worth shipping.
• 02At promotion, a flag on the promote command checks which selected challenges still lack an evaluation module and generates one for exactly those, from the challenge as it enters production.
• 03Coverage is a tracked number, not an assumption: data-quality reports and promotion reports both record which challenges carry evaluation modules.

Once a module exists, the practice is regeneration rather than in-place editing. The app exposes two actions: a backfill that creates a module where one is missing, and a regeneration, restricted to the challenge creator or an admin, that rebuilds the module from the current challenge and its dataset context. Regeneration runs a consistency validation between the dataset and the generated criteria, and the checks are deliberately soft; a free-form description is not rejected on a technicality.

What we do not have, and will not imply: an automatic staleness gate. Nothing currently blocks a module that has drifted from its challenge. Freshness is an operating practice, regenerate on meaningful change, not an enforced invariant. Whether it should become one is an open question below.

A generated module is four to six criteria dimensions, each with metrics, references to public benchmarks, and quantitative targets, tagged with the generation method and a version. It is the review-facing and production-facing statement of what a challenge intends to measure, and it is what the evaluations explorer shows.

It is not the scoring rubric. When an agent runs a challenge, the resulting Episode is scored by an evaluator against rubric dimensions, step by step, and those dimension scores aggregate into the score the run reports. Two artifacts, two jobs: the module declares intent, the rubric applies judgment. Pretending the module is load-bearing in scoring would be exactly the kind of tidy story this notebook exists to avoid.

The benchmark names inside a module are anchors for human review, chosen from a benchmark-aligned template. They tell a reviewer what the challenge is reaching toward. They are not live integrations with those benchmark harnesses, and we would rather say that here than let the word benchmark do quiet work it has not earned.

• 01How aggressively should modules regenerate when a challenge changes: every edit, or only at version boundaries? Regeneration is cheap; churn in the declared criteria is not.
• 02When the canonical solution itself is wrong, the generated criteria inherit the error. What is the right second-pass check?
• 03The declared criteria and the applied rubric are separate artifacts today. How far apart are they in practice? Measuring that distance, per challenge, is the obvious next instrument.