'ARC-AGI' Research Agents for Complex Problem Solving
This challenge tasks you with building a collaborative multi-agent system using CrewAI. The goal is to tackle a complex, abstract research problem, simulating an 'ARC-AGI-like' scenario where agents must demonstrate advanced reasoning, synthesis, and problem-solving capabilities to produce an expert-level research report. Your CrewAI setup will include specialized agents such as a 'Problem Decomposer' (Gemini 2.5 Pro), a 'Knowledge Synthesizer' (Gemini 2.5 Pro), and a 'Critical Evaluator' (GPT-4o). These agents will collaborate, utilizing tools like a Pinecone vector database for persistent memory and context management, and leveraging Ray Serve for efficient deployment of custom analysis tools or specialized models. The challenge emphasizes orchestrating sophisticated agent workflows to go beyond simple information retrieval and truly engage in deep, structured reasoning to arrive at novel insights and solutions, with the final output evaluated for its comprehensiveness and conceptual depth.
What you are building
The core problem, expected build, and operating context for this challenge.
Develop an expert-level multi-agent research system using CrewAI, Gemini 2.5 Pro, and GPT-4o for complex problem-solving, supported by Pinecone and Ray Serve.
How work is evaluated
The evaluation will assess the generated research report for its conceptual depth, logical consistency, correctness, and the ability of the agent system to demonstrate 'expert-level' problem-solving capabilities akin to high ARC-AGI-2 scores. Emphasis on the complexity of the solution, not just factual recall.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
ReportStructureCompliance
Ensures the generated report contains all required sections.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
SolutionPlausibility
Checks if the core solution presented is logically sound given the problem constraints.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Conceptual_Depth_Score
A human-rated score (1-5) assessing the profundity and insightfulness of the analysis. • target: 4 • range: 1-5
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Logical_Consistency_Score
A human-rated score (1-5) evaluating the coherence and logical flow throughout the report. • target: 4 • range: 1-5
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Problem_Coverage_Ratio
Automated metric: Percentage of problem facets explicitly addressed in the report. • target: 0.85 • range: 0.6-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master CrewAI for defining roles, tasks, and hierarchical collaboration among specialized AI agents to solve complex, multi-faceted problems.
Utilize Gemini 2.5 Pro for advanced reasoning, problem decomposition, and knowledge synthesis tasks, leveraging its deep thinking and multi-modal capabilities if applicable.
Integrate GPT-4o as a specialized 'Critical Evaluator' agent within the CrewAI framework, focusing on its nuanced understanding for critique and refinement of research outputs.
Design and implement a long-term memory system for agents using Pinecone vector database, enabling persistent context, concept retrieval, and structured knowledge management.
Develop and deploy custom tool-using agents or specialized model endpoints using Ray Serve, allowing agents to access domain-specific functions or advanced analytical capabilities.
Orchestrate iterative research and refinement cycles within CrewAI, demonstrating how agents can collaborate to explore hypotheses, synthesize findings, and critically evaluate their own outputs to converge on high-quality solutions.
Reference links and supporting material
Sample data for 1 tasks
Challenge setup
The evaluation will assess the generated research report for its conceptual depth, logical consistency, correctness, and the ability of the agent system to demonstrate 'expert-level' problem-solving capabilities akin...
Challenge input
{"problem_statement": "string", "contextual_info": "string", "required_sections": ["Introduction", "Problem Decomposition", "Synthesized Findings",...
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{"report_title": "string", "report_content": {"Introduction": "string", "Problem Decomposition": "string", "Synthesized Findings": "string", "Criti...
- Ensures the generated report contains all required sections.
- Checks if the core solution presented is logically sound given the problem constraints.
- A human-rated score (1-5) assessing the profundity and insightfulness of the analysis. • target: 4 • range:...
- Conceptual Depth Score target: 4
- 1 public reference case
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- ReportStructureCompliance · Weight 1
- SolutionPlausibility · Weight 1
- Conceptual_Depth_Score · Weight 1
- Logical_Consistency_Score · Weight 1
- Problem_Coverage_Ratio · Weight 1
Metrics
- Conceptual_Depth_Score · Target 4
- Logical_Consistency_Score · Target 4
- Problem_Coverage_Ratio · Target 0.85
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: {"problem_statement": "string", "contextual_info": "string", "required_sections": ["Introduction", "Problem Decomposition", "Synthesized Findings", "Critical Analysis", "Conclusion"]}
Output: {"report_title": "string", "report_content": {"Introduction": "string", "Problem Decomposition": "string", "Synthesized Findings": "string", "Critical Analysis": "string", "Conclusion": "string"}}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.