Agent for Complex Policy & Contract Analysis
Develop a Claude Agent using the Claude Agents SDK capable of dissecting and analyzing complex legal or business policy documents, drawing inspiration from the startup Ivo's approach to breaking down legal reviews. The agent will focus on reducing 'hallucinations' by performing granular task decomposition and leveraging contextual retrieval. It will use Claude Opus 4.1 for sophisticated reasoning, Weaviate for efficient semantic search over a corpus of policy documents (RAG approach), and Prefect for orchestrating document ingestion workflows. A simple Gradio interface will allow users to submit documents for analysis.
What you are building
The core problem, expected build, and operating context for this challenge.
Build a Claude agent for robust policy and contract analysis, utilizing RAG with Weaviate and Prefect for document processing.
How work is evaluated
The agent will be evaluated on its ability to accurately break down policy documents, retrieve relevant clauses, and provide insightful, hallucination-free summaries and risk assessments. RAG effectiveness and task decomposition will be key.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
Correct Clause Identification
Checks if at least two relevant clauses are identified and summarized.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Hallucination Absence
Verifies that the agent self-reports no hallucinations, or an external check confirms veracity.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Semantic Accuracy of Summaries
Measures how accurately clause summaries reflect the original text (0-100). • target: 85 • range: 0-100
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Completeness of Implications
Evaluates how comprehensively the agent identifies implications for each clause (0-100). • target: 80 • range: 0-100
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master the **Claude Agents SDK** for defining tools, coordinating multi-step reasoning, and leveraging its 'computer use' capabilities for document interaction.
Utilize **Claude Opus 4.1** for its superior long-context understanding and complex reasoning in legal and policy interpretation.
Implement a **Weaviate** vector database for storing and semantically searching policy documents, forming the core of the RAG system.
Design and deploy data pipelines with **Prefect** for automated ingestion, chunking, embedding, and indexing of new policy documents into Weaviate.
Build custom tools for the Claude Agent to interact with Weaviate (e.g., `retrieve_relevant_clauses(query: str)`) and perform document summarization.
Develop a simple web interface using **Gradio** for users to upload policy documents and receive structured analysis results.
Implement strategies for granular task decomposition, breaking down complex analysis into smaller, verifiable steps to minimize hallucinations, inspired by the Ivo approach.
Reference links and supporting material
Sample data for 1 tasks
Challenge setup
The agent will be evaluated on its ability to accurately break down policy documents, retrieve relevant clauses, and provide insightful, hallucination-free summaries and risk assessments. RAG effectiveness and task de...
Challenge input
{ "document_text": "string", "analysis_goals": ["string"] }
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{ "key_clauses": [{ "clause_text": "string", "summary": "string", "implications": ["string"] }], "overall_risk_assessment": "string", "hallucinatio...
- Checks if at least two relevant clauses are identified and summarized.
- Verifies that the agent self-reports no hallucinations, or an external check confirms veracity.
- Measures how accurately clause summaries reflect the original text (0-100). • target: 85 • range: 0-100
- Semantic Accuracy Of Summaries target: 85
- 1 public reference case
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Correct Clause Identification · Weight 1
- Hallucination Absence · Weight 1
- Semantic Accuracy of Summaries · Weight 1
- Completeness of Implications · Weight 1
Metrics
- Semantic Accuracy of Summaries · Target 85
- Completeness of Implications · Target 80
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: { "document_text": "string", "analysis_goals": ["string"] }
Output: { "key_clauses": [{ "clause_text": "string", "summary": "string", "implications": ["string"] }], "overall_risk_assessment": "string", "hallucination_check": "bool" }
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.