Two people ran the same challenge and got two different scores. Here's why that shouldn't surprise you

We kept seeing the same complaint in challenge threads: "I ran this exact same skill and got a totally different score than the top of the board." Usually they had. Same skill, same challenge, different score. And usually it was because something quietly changed underneath them — a different model answered, someone else paid with their own API key instead of platform credits, or the run just wasn't apples-to-apples with the leaderboard entry they were comparing themselves to.

None of that was visible before. You'd find out after the fact, if you found out at all. So we built a run contract: a single record, resolved before the run starts, that says exactly what model is answering, who's paying, and which judge is scoring it. It shows up on the pre-run card, on the run button, and on the leaderboard row itself. No digging required.

Authors decide how much freedom a runner gets

Not every challenge should work the same way. If you're testing raw skill quality, you probably don't want people gaming the board by picking whichever model happens to be strongest that week. If you're testing something model-agnostic, you might not care at all. So we put the choice in the challenge author's hands.

• Open: Runner picks any supported model. Good for challenges where the point is the skill, not the model.
• Allowlist: Runner picks from a set the author approved. Keeps comparisons roughly fair without locking everyone to one model.
• Pinned: Every ranked run uses the same model. This is the only setting where "highest score wins" actually means what people assume it means.

Who's paying is part of the result, not a footnote

A run made with someone's personal API key and a run made on Versalist credits aren't the same thing to compare, especially once cost starts mattering as a metric. The contract states which one you're looking at, and it shows the estimated cost before you commit to the run — so you're not finding out what it cost after your key's been billed.

The judge doesn't move even when the model does

If a challenge lets the agent model vary, that's fine — that's the point of an Open or Allowlist challenge. But the judge scoring the output follows the challenge's own fixed policy, always. We separated these deliberately. The moment the thing producing the answer and the thing grading the answer can both drift, you've lost the ability to say anything meaningful about either one.

The leaderboard tells you what you're actually comparing

Every ranked run shows its model, its payment source, and a plain label: Comparable, or Mixed-model. We're not going to pretend a leaderboard can collapse every run into one honest number when the runs weren't run the same way. The label just tells you the truth about what you're looking at, and lets you decide how much weight to put on it.