Agent for Real Time Public Policy Analysis
This challenge focuses on building an intelligent agent that can perform real-time policy impact analysis. Participants will leverage Gemini 3 Pro for its advanced reasoning capabilities and DSPy for programmatic prompt optimization, ensuring robust and structured output. The agent will employ a hybrid reasoning approach to distinguish between instant interpretations and deep, contextualized analysis. A crucial component will be an MCP-enabled RAG system, integrating with external legal databases and real-time regulatory news feeds to provide the most accurate and up-to-date policy context, translating abstract rules into concrete business implications.
What you are building
The core problem, expected build, and operating context for this challenge.
Develop a DSPy agent using Gemini 3 Pro for real-time impact analysis of a new AI Executive Order, leveraging hybrid reasoning and MCP-enabled RAG for legal context.
How work is evaluated
The evaluation will assess the agent's accuracy, completeness, and structured output quality in interpreting a hypothetical AI Executive Order and generating its potential impact assessment. Emphasis will be placed on the agent's ability to leverage RAG and hybrid reasoning effectively.
Challenge datasets
Public datasets, plus the private datasets you have access to.
What you should walk away with
Build a robust MCP-enabled RAG pipeline for legal and policy document retrieval.
Generate structured, consistent output for policy impact assessments.
Master DSPy for programmatically structuring and optimizing LLM interactions.
Evaluate and refine agent performance using DSPy's optimization techniques.
Implement a hybrid reasoning system for varied analysis depth (instant vs. deep).
Reference links and supporting material
Sample data for 1 tasks
Challenge setup
The evaluation will assess the agent's accuracy, completeness, and structured output quality in interpreting a hypothetical 'ONE RULE' AI Executive Order and generating its potential impact assessment. Emphasis will b...
Challenge input
{'eo_text': 'string', 'prior_regulations': [{'title': 'string', 'content': 'string'}]}
Agent execution
No agent tool is listed for this challenge.
Evaluated output
A JSON object containing: {'policy_summary': 'string', 'potential_interpretations': ['string'], 'affected_industries': ['string'], 'compliance_chal...
- Ensure the output is a valid JSON adhering to the specified schema.
- Check if 'policy_summary', 'potential_interpretations', and 'affected_industries' are present and non-empty.
- Interpretative Accuracy Score target: 0.85
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Ensure the output is a valid JSON adhering to the specified schema.
- Check if 'policy_summary', 'potential_interpretations', and 'affected_industries' are present and non-empty.
Metrics
- Interpretative Accuracy Score · Target 0.85
- RAG Relevance Score · Target 4
- Structured Output Quality · Target 4
- Hybrid Reasoning Utilization · Target 0.9
Workflow tasks
- Task 1
Input: {'eo_text': 'string', 'prior_regulations': [{'title': 'string', 'content': 'string'}]}
Output: A JSON object containing: {'policy_summary': 'string', 'potential_interpretations': ['string'], 'affected_industries': ['string'], 'compliance_challenges': ['string'], 'estimated_business_impact': 'string', 'reasoning_path_log': [{'step': 'string', 'mode': 'instant|deep'}]}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.