Multi-Agent System for Commercial Real Estate Analysis
Develop a sophisticated multi-agent system using AutoGen to act as an expert commercial real estate (CRE) analyst for institutional investors. The system will comprise specialized agents (e.g., a 'Data Fetcher', an 'Economic Analyst', a 'Valuation Specialist', a 'Report Generator') that collaborate autonomously to research, analyze, and synthesize insights on CRE investment opportunities. This challenge emphasizes complex agent-to-agent communication, tool orchestration, and the generation of comprehensive, data-driven reports. The agents should be able to query external (simulated) CRE data APIs, perform financial modeling, and provide reasoned investment recommendations. Focus on designing robust communication protocols between agents to resolve conflicts and refine analyses. The solution must also incorporate an observability framework to trace the multi-agent deliberation process and provide a user-friendly interface for querying the system and visualizing its output.
What you are building
The core problem, expected build, and operating context for this challenge.
Implement an AutoGen multi-agent system with Claude Opus 4.1 for collaborative commercial real estate investment analysis and reporting.
How work is evaluated
The evaluation will assess the AutoGen system's ability to accurately analyze CRE opportunities, generate comprehensive reports, and demonstrate effective agent collaboration. Key metrics include report accuracy, reasoning depth, and the efficiency of agent communication.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
AutoGenAgentsInitialization
Verify all AutoGen agents (UserProxy, Assistant, specialized agents) can be initialized.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
CustomToolRegistration
Confirm custom tools for CRE data access are correctly registered and callable by agents.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
LangSmithIntegration
Ensure LangSmith can successfully trace an AutoGen conversation.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Report Accuracy Score
Accuracy of key financial figures and investment recommendations in the generated report (0-1). • target: 0.88 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Report Coherence and Completeness
Score reflecting the logical flow, clarity, and inclusion of all required report sections (0-1). • target: 0.9 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Agent Collaboration Efficiency (Messages)
Number of messages exchanged during a standard analysis task (lower is better). • target: 20 • range: 5-50
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Time to Generate Report (minutes)
End-to-end time from query submission to final report generation. • target: 3 • range: 0-10
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master AutoGen's multi-agent conversational patterns, including agent configuration, role assignment, and inter-agent communication protocols.
Leverage Claude Opus 4.1's advanced reasoning capabilities for complex financial analysis, valuation modeling, and nuanced investment recommendation generation.
Develop custom AutoGen tools for agents to query simulated commercial real estate databases, market data APIs, and economic indicators.
Implement collaborative problem-solving strategies within AutoGen, where agents refine tasks, provide feedback, and reach consensus on investment decisions.
Integrate LangSmith for end-to-end tracing of the multi-agent deliberation, visualizing message flows, tool calls, and LLM interactions for debugging and optimization.
Design data ingestion and preprocessing workflows using Prefect to prepare raw CRE data for agent consumption and analysis.
Build a simple Plotly Dash dashboard for institutional investors to submit queries, view real-time agent activity (via LangSmith), and review generated investment reports.
Formulate sophisticated prompt engineering for Claude Opus 4.1 to ensure structured, detailed, and accurate CRE reports, including executive summaries and risk assessments.
Reference links and supporting material
Sample data for 3 tasks
Challenge setup
The evaluation will assess the AutoGen system's ability to accurately analyze CRE opportunities, generate comprehensive reports, and demonstrate effective agent collaboration. Key metrics include report accuracy, reas...
Challenge input
{ "investment_query": "Analyze a potential retail property acquisition in downtown Austin, TX, with a budget of $10M, focusing on rental yield and...
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{ "recommendation": "Strong Buy", "key_findings": ["High rental yield", "Strong growth potential"], "reasoning_score": 0.9 }
- Verify all AutoGen agents (UserProxy, Assistant, specialized agents) can be initialized.
- Confirm custom tools for CRE data access are correctly registered and callable by agents.
- Ensure LangSmith can successfully trace an AutoGen conversation.
- Report Accuracy Score target: 0.88
- 3 public reference cases
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- AutoGenAgentsInitialization · Weight 1
- CustomToolRegistration · Weight 1
- LangSmithIntegration · Weight 1
- Report Accuracy Score · Weight 1
- Report Coherence and Completeness · Weight 1
- Agent Collaboration Efficiency (Messages) · Weight 1
- Time to Generate Report (minutes) · Weight 1
Metrics
- Report Accuracy Score · Target 0.88
- Report Coherence and Completeness · Target 0.9
- Agent Collaboration Efficiency (Messages) · Target 20
- Time to Generate Report (minutes) · Target 3
Reference cases
- 3 public reference cases
Workflow tasks
- Task 1
Input: { "investment_query": "Analyze a potential retail property acquisition in downtown Austin, TX, with a budget of $10M, focusing on rental yield and growth potential.", "simulated_market_data": { "austin_retail_yield": 0.05, "austin_retail_growth": 0.03 } }
Output: { "recommendation": "Strong Buy", "key_findings": ["High rental yield", "Strong growth potential"], "reasoning_score": 0.9 }
Execution harness: Python
- Task 2
Input: { "analysis_results": {"recommendation": "Buy", "summary": "...", "risks": "..."}, "required_sections": ["Executive Summary", "Market Analysis", "Financials", "Risks"] }
Output: { "report_sections_present": ["Executive Summary", "Market Analysis"], "coherence_score": 0.8 }
Execution harness: Python
- Task 3
Input: { "initial_task": "analyze CRE property", "max_messages": 50 }
Output: { "total_messages": 25, "task_resolved": true, "bottlenecks_identified": [] }
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.