Challenges

A challenge combines a task, constraints, tools, source data, required artifacts, and an evaluation method. It is more than a prompt.

Select a challenge

1. Open the challenge catalog.
2. Filter by category, difficulty, or search term.
3. Open a challenge detail page.
4. Read the complete task and constraints.
5. Review the tools, datasets, rubric, and reference items.

Select a task that supports repeated runs. The task must also have an evaluation method that you can apply consistently.

Challenge detail

Task

The overview identifies the required result, constraints, and completion conditions.

Tools and data

The challenge identifies required tools, allowed tools, datasets, and supporting files.

Rubric and reference items

The rubric defines evaluation dimensions. Reference items provide expected examples when the task supports them.

Leaderboard

The leaderboard provides comparison data when the challenge enables public results.

Run a challenge

1. Keep the challenge definition unchanged.
2. Run the agent once.
3. Store the output and execution evidence.
4. Apply the documented evaluator.
5. Record the score and failure mode.
6. Use the first complete attempt as the baseline.

Use the CLI when you need challenge files and evidence in a local repository.

Submit a result

A submission can include a project URL, source repository, title, and implementation notes.

1. Confirm that the project URL is accessible to the intended reviewer.
2. Identify the challenge.
3. Add the model and toolchain names when they help comparison.
4. Submit the result.

Local CLI evidence remains local unless a later workflow explicitly accepts it.

Create a challenge

Use Create challenge for guided, manual, or AI-assisted creation. Use Studio for detailed editing.

1. Write one measurable task.
2. List all task constraints.
3. Define the required output artifact.
4. Add only the required tools and datasets.
5. Define rubric dimensions and weights.
6. Add reference items when they improve evaluation consistency.
7. Run a baseline before publication.

Maintain a challenge

Use Fix tools when a challenge has incorrect tool metadata. Edit the challenge when its task contract changes.

Do not compare runs across different challenge versions without recording the version change.

Evaluation rules

• Use deterministic checks when the result supports them.
• Keep rubric criteria separate from promotional claims.
• Use the same evaluator for baseline and candidate runs.
• Review available bounded trace metadata when the score changes.
• Record evaluator changes with the result.