Mathematical Proof Assistant
This challenge focuses on building an advanced AI system capable of understanding complex mathematical questions, retrieving relevant theorems and definitions from a specialized knowledge base, and constructing logical proofs or counter-examples. Participants will leverage LlamaIndex's advanced RAG capabilities to ensure contextual understanding and Gemini 2.5 Pro's strong reasoning for generating robust mathematical arguments. The emphasis will be on accurate grounding of facts, verifiable proof construction, and systematic evaluation of the AI's mathematical competence against novel problems. The project requires designing and populating a structured mathematical knowledge base using LlamaIndex data connectors, integrating a vector store like ChromaDB for efficient retrieval. Developers will orchestrate a multi-stage LlamaIndex agent workflow that can plan, execute, and verify proof steps. The final system should demonstrate robust reasoning by generating mathematically sound proofs and identifying valid counter-examples when applicable, similar to the objectives of the 'First Proof' experiment.
What you are building
The core problem, expected build, and operating context for this challenge.
Build a LlamaIndex agent capable of mathematical reasoning and proof generation, leveraging advanced RAG with Gemini 2.5 Pro.
How work is evaluated
Evaluate the system's ability to generate logically sound and verifiable mathematical proofs or counter-examples based on provided axioms and theorems from a knowledge base.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
proof_structure_validity
Checks if the generated proof follows a logical, step-by-step structure and uses standard mathematical notation.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
correctness_of_conclusion
Verifies if the final conclusion or counter-example is mathematically consistent with the initial statement and intermediate steps.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
retrieval_accuracy
Assesses if relevant theorems and axioms were accurately retrieved from the knowledge base and cited (if applicable).
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
proof_length
Number of logical steps in the generated proof. • target: 8 • range: 3-15
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
relevance_score
Semantic similarity of retrieved context to the query, indicating effective RAG. • target: 0.9 • range: 0.7-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
reasoning_confidence
Model's self-assessed confidence level in the generated proof or counter-example. • target: 0.85 • range: 0.5-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master LlamaIndex's advanced RAG techniques, including recursive retrieval and query rewriting, for navigating complex mathematical knowledge graphs.
Implement custom LlamaIndex data connectors for ingesting academic papers, LaTeX documents, and structured theorem repositories into a ChromaDB vector store.
Orchestrate a multi-stage reasoning agent using LlamaIndex's agentic capabilities to plan mathematical proof steps and execute sub-tasks.
Integrate Gemini 2.5 Pro's advanced mathematical reasoning mode to generate proof steps, hypotheses, and formal arguments.
Design and implement an evaluation harness using Continue.dev to iteratively test the system's proof generation capabilities against a benchmark of unpublished mathematical problems.
Deploy the LlamaIndex query engine and specialized Gemini 2.5 Pro inference endpoint via Baseten for scalable and efficient proof generation services.
Develop a user interface or API wrapper that allows mathematicians to interact with the proof assistant and review generated proofs.
Reference links and supporting material
Sample data for 2 tasks
Challenge setup
Evaluate the system's ability to generate logically sound and verifiable mathematical proofs or counter-examples based on provided axioms and theorems from a knowledge base.
Challenge input
JSON object with a 'statement' string field.
Agent execution
No agent tool is listed for this challenge.
Evaluated output
JSON object with 'proof' string, 'is_valid' boolean, and 'confidence' float fields.
- Checks if the generated proof follows a logical, step-by-step structure and uses standard mathematical nota...
- Verifies if the final conclusion or counter-example is mathematically consistent with the initial statement...
- Assesses if relevant theorems and axioms were accurately retrieved from the knowledge base and cited (if ap...
- Proof Length target: 8
- 2 public reference cases
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- proof_structure_validity · Weight 1
- correctness_of_conclusion · Weight 1
- retrieval_accuracy · Weight 1
- proof_length · Weight 1
- relevance_score · Weight 1
- reasoning_confidence · Weight 1
Metrics
- proof_length · Target 8
- relevance_score · Target 0.9
- reasoning_confidence · Target 0.85
Reference cases
- 2 public reference cases
Workflow tasks
- Task 1
Input: JSON object with a 'statement' string field.
Output: JSON object with 'proof' string, 'is_valid' boolean, and 'confidence' float fields.
Execution harness: Python
- Task 2
Input: JSON object with a 'statement' string field.
Output: JSON object with 'proof_or_counterexample' string, 'is_valid' boolean, and 'confidence' float fields.
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.