Agent for Complex Policy & Contract Analysis
Develop a Claude Agent using the Claude Agents SDK capable of dissecting and analyzing complex legal or business policy documents, drawing inspiration from the startup Ivo's approach to breaking down legal reviews. The agent will focus on reducing 'hallucinations' by performing granular task decomposition and leveraging contextual retrieval. It will use Claude Opus 4.1 for sophisticated reasoning, Weaviate for efficient semantic search over a corpus of policy documents (RAG approach), and Prefect for orchestrating document ingestion workflows. A simple Gradio interface will allow users to submit documents for analysis.
What you are building
The core problem, expected build, and operating context for this challenge.
Develop a Claude Agent using the Claude Agents SDK capable of dissecting and analyzing complex legal or business policy documents, drawing inspiration from the startup Ivo's approach to breaking down legal reviews. The agent will focus on reducing 'hallucinations' by performing granular task decomposition and leveraging contextual retrieval. It will use Claude Opus 4.1 for sophisticated reasoning, Weaviate for efficient semantic search over a corpus of policy documents (RAG approach), and Prefect for orchestrating document ingestion workflows. A simple Gradio interface will allow users to submit documents for analysis.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
Correct Clause Identification
Checks if at least two relevant clauses are identified and summarized.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Hallucination Absence
Verifies that the agent self-reports no hallucinations, or an external check confirms veracity.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Semantic Accuracy of Summaries
Measures how accurately clause summaries reflect the original text (0-100). • target: 85 • range: 0-100
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Completeness of Implications
Evaluates how comprehensively the agent identifies implications for each clause (0-100). • target: 80 • range: 0-100
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master the **Claude Agents SDK** for defining tools, coordinating multi-step reasoning, and leveraging its 'computer use' capabilities for document interaction.
Utilize **Claude Opus 4.1** for its superior long-context understanding and complex reasoning in legal and policy interpretation.
Implement a **Weaviate** vector database for storing and semantically searching policy documents, forming the core of the RAG system.
Design and deploy data pipelines with **Prefect** for automated ingestion, chunking, embedding, and indexing of new policy documents into Weaviate.
Build custom tools for the Claude Agent to interact with Weaviate (e.g., `retrieve_relevant_clauses(query: str)`) and perform document summarization.
Develop a simple web interface using **Gradio** for users to upload policy documents and receive structured analysis results.
Implement strategies for granular task decomposition, breaking down complex analysis into smaller, verifiable steps to minimize hallucinations, inspired by the Ivo approach.
How this agent runs
The agent will be evaluated on its ability to accurately break down policy documents, retrieve relevant clauses, and provide insightful, hallucination-free summaries and risk assessments. RAG effectiveness and task de...
Challenge input
{ "document_text": "string", "analysis_goals": ["string"] }
Agent execution
The configured agent processes the input under the challenge policy.
Evaluated output
{ "key_clauses": [{ "clause_text": "string", "summary": "string", "implications": ["string"] }], "overall_risk_assessment": "string", "hallucinatio...
- Checks if at least two relevant clauses are identified and summarized.
- Verifies that the agent self-reports no hallucinations, or an external check confirms veracity.
- Measures how accurately clause summaries reflect the original text (0-100). • target: 85 • range: 0-100
- Semantic Accuracy Of Summaries target: 85
- 1 public reference case
- Python execution harness
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Correct Clause Identification · Weight 1
- Hallucination Absence · Weight 1
- Semantic Accuracy of Summaries · Weight 1
- Completeness of Implications · Weight 1
Recipe state
This is a preview. The configuration can change before the evaluation recipe is locked.
Run this agent on your dataset and AI stack
Bring your dataset, model providers, and success criteria. We will scope the right managed run for your team.
Scope a managed run[ok] Wrote CHALLENGE.md
[ok] Wrote .versalist.json
[ok] Wrote eval/examples.json
Requires VERSALIST_API_KEY. Works with any MCP-aware editor.
DocsFind another challenge
Jump to a random challenge when you want a fresh benchmark or a different problem space.