Mathematical Proof Assistant
This challenge focuses on building an advanced AI system capable of understanding complex mathematical questions, retrieving relevant theorems and definitions from a specialized knowledge base, and constructing logical proofs or counter-examples. Participants will leverage LlamaIndex's advanced RAG capabilities to ensure contextual understanding and Gemini 2.5 Pro's strong reasoning for generating robust mathematical arguments. The emphasis will be on accurate grounding of facts, verifiable proof construction, and systematic evaluation of the AI's mathematical competence against novel problems. The project requires designing and populating a structured mathematical knowledge base using LlamaIndex data connectors, integrating a vector store like ChromaDB for efficient retrieval. Developers will orchestrate a multi-stage LlamaIndex agent workflow that can plan, execute, and verify proof steps. The final system should demonstrate robust reasoning by generating mathematically sound proofs and identifying valid counter-examples when applicable, similar to the objectives of the 'First Proof' experiment.
What you are building
The core problem, expected build, and operating context for this challenge.
This challenge focuses on building an advanced AI system capable of understanding complex mathematical questions, retrieving relevant theorems and definitions from a specialized knowledge base, and constructing logical proofs or counter-examples. Participants will leverage LlamaIndex's advanced RAG capabilities to ensure contextual understanding and Gemini 2.5 Pro's strong reasoning for generating robust mathematical arguments. The emphasis will be on accurate grounding of facts, verifiable proof construction, and systematic evaluation of the AI's mathematical competence against novel problems. The project requires designing and populating a structured mathematical knowledge base using LlamaIndex data connectors, integrating a vector store like ChromaDB for efficient retrieval. Developers will orchestrate a multi-stage LlamaIndex agent workflow that can plan, execute, and verify proof steps. The final system should demonstrate robust reasoning by generating mathematically sound proofs and identifying valid counter-examples when applicable, similar to the objectives of the 'First Proof' experiment.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
proof_structure_validity
Checks if the generated proof follows a logical, step-by-step structure and uses standard mathematical notation.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
correctness_of_conclusion
Verifies if the final conclusion or counter-example is mathematically consistent with the initial statement and intermediate steps.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
retrieval_accuracy
Assesses if relevant theorems and axioms were accurately retrieved from the knowledge base and cited (if applicable).
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
proof_length
Number of logical steps in the generated proof. • target: 8 • range: 3-15
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
relevance_score
Semantic similarity of retrieved context to the query, indicating effective RAG. • target: 0.9 • range: 0.7-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
reasoning_confidence
Model's self-assessed confidence level in the generated proof or counter-example. • target: 0.85 • range: 0.5-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master LlamaIndex's advanced RAG techniques, including recursive retrieval and query rewriting, for navigating complex mathematical knowledge graphs.
Implement custom LlamaIndex data connectors for ingesting academic papers, LaTeX documents, and structured theorem repositories into a ChromaDB vector store.
Orchestrate a multi-stage reasoning agent using LlamaIndex's agentic capabilities to plan mathematical proof steps and execute sub-tasks.
Integrate Gemini 2.5 Pro's advanced mathematical reasoning mode to generate proof steps, hypotheses, and formal arguments.
Design and implement an evaluation harness using Continue.dev to iteratively test the system's proof generation capabilities against a benchmark of unpublished mathematical problems.
Deploy the LlamaIndex query engine and specialized Gemini 2.5 Pro inference endpoint via Baseten for scalable and efficient proof generation services.
Develop a user interface or API wrapper that allows mathematicians to interact with the proof assistant and review generated proofs.
How this agent runs
Evaluate the system's ability to generate logically sound and verifiable mathematical proofs or counter-examples based on provided axioms and theorems from a knowledge base.
Challenge input
JSON object with a 'statement' string field.
Agent execution
The configured agent processes the input under the challenge policy.
Evaluated output
JSON object with 'proof' string, 'is_valid' boolean, and 'confidence' float fields.
- Checks if the generated proof follows a logical, step-by-step structure and uses standard mathematical nota...
- Verifies if the final conclusion or counter-example is mathematically consistent with the initial statement...
- Assesses if relevant theorems and axioms were accurately retrieved from the knowledge base and cited (if ap...
- Proof Length target: 8
- 2 public reference cases
- Python execution harness
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- proof_structure_validity · Weight 1
- correctness_of_conclusion · Weight 1
- retrieval_accuracy · Weight 1
- proof_length · Weight 1
- relevance_score · Weight 1
- reasoning_confidence · Weight 1
Recipe state
This is a preview. The configuration can change before the evaluation recipe is locked.
Run this agent on your dataset and AI stack
Bring your dataset, model providers, and success criteria. We will scope the right managed run for your team.
Scope a managed run[ok] Wrote CHALLENGE.md
[ok] Wrote .versalist.json
[ok] Wrote eval/examples.json
Requires VERSALIST_API_KEY. Works with any MCP-aware editor.
DocsFind another challenge
Jump to a random challenge when you want a fresh benchmark or a different problem space.