Graph-Based Legal Aid
This challenge tasks you with building a robust, hallucination-resistant legal aid agent system. You will design a graph-based workflow using LangGraph, leveraging Claude Opus 4.5 for nuanced legal reasoning and LlamaIndex for advanced RAG over legal documents. The system must incorporate self-correction mechanisms via A2A protocol for agent verification, and adaptive thinking budgets to ensure accuracy and timely responses in complex probate cases. The core focus is on mitigating factual errors and improving response relevance in sensitive legal contexts. You will integrate MCP-enabled tools for secure access to enterprise legal data and orchestrate the multi-agent system using Letta AI for comprehensive performance monitoring and evaluation, pushing the boundaries of reliable generative AI in public services.
What you are building
The core problem, expected build, and operating context for this challenge.
This challenge tasks you with building a robust, hallucination-resistant legal aid agent system. You will design a graph-based workflow using LangGraph, leveraging Claude Opus 4.5 for nuanced legal reasoning and LlamaIndex for advanced RAG over legal documents. The system must incorporate self-correction mechanisms via A2A protocol for agent verification, and adaptive thinking budgets to ensure accuracy and timely responses in complex probate cases. The core focus is on mitigating factual errors and improving response relevance in sensitive legal contexts. You will integrate MCP-enabled tools for secure access to enterprise legal data and orchestrate the multi-agent system using Letta AI for comprehensive performance monitoring and evaluation, pushing the boundaries of reliable generative AI in public services.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
What you should walk away with
Master LangGraph for building stateful, self-correcting agent workflows with dynamic graph modifications and adaptive reasoning paths.
Implement A2A protocol for secure, verifiable agent-to-agent communication to achieve consensus and fact-check legal interpretations.
Design MCP-enabled tool integration with Claude Opus 4.5 for secure, real-time access to enterprise legal databases, statutes, and case management systems.
Build advanced RAG pipelines using LlamaIndex and vector databases to ensure context-aware, hallucination-free legal responses from Claude Opus 4.5, focusing on legal citation accuracy.
Orchestrate multi-agent systems using Letta AI, focusing on performance monitoring, auditing, and iterative improvement cycles for legal accuracy.
Deploy adaptive thinking budgets within LangGraph agents to dynamically allocate computational resources based on query complexity and legal sensitivity.
Develop a domain-specific evaluation suite to measure accuracy, relevance, and hallucination rates in simulated legal scenarios using DSPy for prompt optimization.
How this agent runs
The evaluation will focus on the agent's ability to provide accurate, relevant, and hallucination-free legal advice for probate-related queries, specifically assessing its self-correction mechanisms, data retrieval ef...
Challenge input
{"query": "string", "context_docs": ["string", ...], "expected_facts": [{"fact": "string", "critical": "boolean"}, ...]}
Agent execution
The configured agent processes the input under the challenge policy.
Evaluated output
{"response": "string", "cited_sources": [{"source_name": "string", "confidence": "float"}], "self_correction_logs": "string", "a2a_verification_sta...
- The response must not contain factual information unsupported by the provided context or general legal prin...
- All critical expected facts must be present and correctly explained in the response.
- The 'a2a_verification_status' must indicate successful verification or a clear attempt at agent-to-agent co...
- Factual Accuracy Score target: 0.95
- Python execution harness
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- The evaluation module defines the checks.
Recipe state
This is a preview. The configuration can change before the evaluation recipe is locked.
Run this agent on your dataset and AI stack
Bring your dataset, model providers, and success criteria. We will scope the right managed run for your team.
Scope a managed run[ok] Wrote CHALLENGE.md
[ok] Wrote .versalist.json
[ok] Wrote eval/examples.json
Requires VERSALIST_API_KEY. Works with any MCP-aware editor.
DocsFind another challenge
Jump to a random challenge when you want a fresh benchmark or a different problem space.