Graph-Based Legal Aid
This challenge tasks you with building a robust, hallucination-resistant legal aid agent system. You will design a graph-based workflow using LangGraph, leveraging Claude Opus 4.5 for nuanced legal reasoning and LlamaIndex for advanced RAG over legal documents. The system must incorporate self-correction mechanisms via A2A protocol for agent verification, and adaptive thinking budgets to ensure accuracy and timely responses in complex probate cases. The core focus is on mitigating factual errors and improving response relevance in sensitive legal contexts. You will integrate MCP-enabled tools for secure access to enterprise legal data and orchestrate the multi-agent system using Letta AI for comprehensive performance monitoring and evaluation, pushing the boundaries of reliable generative AI in public services.
What you are building
The core problem, expected build, and operating context for this challenge.
Build a graph-based, hallucination-resistant multi-agent legal aid system using LangGraph, Claude Opus 4.1, and Letta AI, with advanced RAG and A2A protocol for self-correction.
How work is evaluated
The evaluation will focus on the agent's ability to provide accurate, relevant, and hallucination-free legal advice for probate-related queries, specifically assessing its self-correction mechanisms, data retrieval efficacy, and the integration of A2A verification.
Challenge datasets
Public datasets, plus the private datasets you have access to.
What you should walk away with
Implement robust RAG for hallucination mitigation in legal contexts using LlamaIndex.
Integrate MCP-enabled tools for secure court system data access.
Utilize A2A protocol for agent self-correction and verification.
Orchestrate and evaluate agent performance with Letta AI.
Design a graph-based multi-agent architecture for legal aid with self-correction.
Reference links and supporting material
Sample data for 1 tasks
Challenge setup
The evaluation will focus on the agent's ability to provide accurate, relevant, and hallucination-free legal advice for probate-related queries, specifically assessing its self-correction mechanisms, data retrieval ef...
Challenge input
{"query": "string", "context_docs": ["string", ...], "expected_facts": [{"fact": "string", "critical": "boolean"}, ...]}
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{"response": "string", "cited_sources": [{"source_name": "string", "confidence": "float"}], "self_correction_logs": "string", "a2a_verification_sta...
- The response must not contain factual information unsupported by the provided context or general legal prin...
- All critical expected facts must be present and correctly explained in the response.
- The 'a2a_verification_status' must indicate successful verification or a clear attempt at agent-to-agent co...
- Factual Accuracy Score target: 0.95
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- The response must not contain factual information unsupported by the provided context or general legal principles.
- All critical expected facts must be present and correctly explained in the response.
- The 'a2a_verification_status' must indicate successful verification or a clear attempt at agent-to-agent consultation.
Metrics
- Factual Accuracy Score · Target 0.95
- Response Completeness Score · Target 0.9
- Average Latency (seconds) · Target 20
Workflow tasks
- Task 1
Input: {"query": "string", "context_docs": ["string", ...], "expected_facts": [{"fact": "string", "critical": "boolean"}, ...]}
Output: {"response": "string", "cited_sources": [{"source_name": "string", "confidence": "float"}], "self_correction_logs": "string", "a2a_verification_status": "string"}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.