A score you can't audit is decoration

A number on a leaderboard is a promise: this skill scored 71%. But 71% with which model? Paid for by whose key? Graded by which judge? On most eval platforms you cannot tell, and a score you cannot audit is decoration.

A run that used one frontier model is not automatically comparable to a run that used another. It matters whether the runner spent their own key or the platform paid. And if the judge can drift between runs, the leaderboard quietly stops meaning anything at all. Those three decisions used to live implicitly in a run log. We made them a contract.

A run contract is a server-resolved agreement covering what model runs, who pays for the inference, and how the result gets judged. It is settled before the run starts and shown again at every point someone might look at the result later: the pre-run card, the run button, the leaderboard row. Author policy, runner disclosure, server validation, leaderboard truth: one object, visible end to end.

Authors set the rules in Run Settings

Challenge authors decide how much control a runner has over inference. There are three modes, and they trade freedom for comparability.

• Open: Runners pick any supported model.
• Allowlist: Runners pick from an approved set.
• Pinned: Everyone runs the same model. This is the setting for a leaderboard you want to take seriously.

Who pays

Payment is a separate call. An author can allow either the runner's key or platform credits, require the runner to bring their own key, or have Versalist cover the cost. The point is that nobody discovers who is paying after the tokens are already spent.

One judge, whatever ran

The agent model is the runner's choice. The judge is not. No matter what model produces the output, evaluation always routes through the fixed Versalist judge. That is the line that keeps runner-owned inference from bleeding into platform-controlled scoring: the thing being measured can vary, the ruler cannot.

Runners see the whole deal before spending a token

Before a run starts, the pre-run card spells out exactly what is about to happen: which model will run, whose key pays, the estimated cost, whether the result will be directly comparable to others on the leaderboard or logged as a mixed-model practice run, and which judge will grade it.

The run button reads back the actual contract: "Run with Google Gemini 2.5 Pro using your key." No ambiguity about what you are buying when you click.

Leaderboards stop pretending

Every ranked run now carries its inference truth: the model used, whether it ran on a runner or platform key, and a Comparable or Mixed-model badge. You can filter to a single model, and the fixed judge is disclosed on the row.

Here is the part that matters. If every ranked run used the pinned model under the same policy version, we mark the board comparable. If runs used different models, the UI says so instead of stacking them into one ranking and pretending the scores are the same currency. An honest "these do not compare" beats a tidy leaderboard that lies.

What it does not do yet

A run contract makes the conditions of a score legible; it does not collapse them into a single universal number. Comparing a pinned-model board to an open one is still your judgment call. We give you the badges and the filter, not a verdict. That is deliberate. The goal is to show the conditions, not to hide them behind a rating.

What ran, who paid, and how it was judged. Answer those three before you trust the number. That is the whole contract between author, runner, and judge.