Making AI Challenge Runs Fairer and More Transparent
A run contract makes the model, the payment source, and the judge explicit before, during, and after every challenge run.
When someone runs a skill against a Versalist challenge, the model choice matters. A run using one frontier model is not automatically comparable to a run using another. The cost source matters too: did the runner spend their own key, or did the platform pay? And for scoring, the judge has to stay fixed, or the leaderboard stops meaning anything.
We shipped a run contract to make all three of those decisions explicit instead of implicit. A run contract is a server-resolved agreement for what model runs, who pays, and how the result gets judged. It is decided before the run starts and shown at every point someone might look at the result afterward.
For challenge authors: Run Settings
Challenge authors now get a Run Settings section inside challenge editing. This is where they decide how much control runners have over inference.
- Open: Runners can pick any supported model.
- Allowlist: Runners can pick only from approved models.
- Pinned: Everyone uses the same model. Best for serious leaderboard comparisons.
Who pays
Authors also set the payment policy for the challenge:
- Runner key or platform credits: Either side can cover inference cost.
- BYOK required: The runner must bring their own key.
- Platform required: Versalist covers inference cost.
One judge, no matter what ran
No matter what agent model runs, judging stays fixed through the Versalist judge. That keeps scores comparable and keeps runner-owned inference from mixing with platform-controlled evaluation.
For runners: clear inference before spending tokens
Before starting a run, runners now see an Inference block in the pre-run card.
- Which model will run
- Whose key pays: Your own key or platform credits.
- Estimated cost
- Comparable or mixed-model: Whether this run is directly comparable to others on the leaderboard, or a mixed-model practice run.
- Which judge will evaluate the result
The run button reflects the actual contract: "Run with Google Gemini 2.5 Pro using your key." No ambiguity about what happens when you spend the tokens.
For leaderboards: inference truth
Leaderboards now show the model and key source behind each ranked run. A viewer can see:
- Model used
- Runner key vs. platform key
- Comparable or Mixed-model badge
- Same-model filter
- Fixed judge disclosure
If every ranked run used the pinned model under the same policy version, we mark it comparable. If runs used different models, the UI says so instead of pretending all scores mean the same thing.
Why this matters
AI evaluation is only useful when the conditions producing it are clear. This module makes those conditions explicit instead of buried in a run log.
Authors control the challenge policy. Runners see cost and model before they spend anything. Viewers can read a leaderboard and know exactly what the numbers do and do not mean.
What ran, who paid, and how it was judged: that is the contract between author, runner, and evaluator.
We've been building an RL platform. We just didn't say it.
Challenges are environments. Skills are policies. Scores are reward signals. Episodes make the loop real.
Beyond Pass/Fail: Why We Added Structured Rubrics to Evaluate Multi-Agent Systems
Binary pass/fail tests don't capture what matters in multi-agent systems. We added Rubric as a first-class primitive: structured, weighted dimensions that score nuanced behaviors.
autoresearcher
How Versalist turns rubrics, gold items, and prompt skills into an autonomous experimentation loop.