Autonomous Enterprise Security Compliance Agent with Claude Opus 4.6
Develop an advanced autonomous agent system using the Claude Agents SDK that leverages Claude Opus 4.6's 1M token context window and agentic capabilities to scrutinize large volumes of enterprise documents, regulatory filings, and internal policies. The agent team will identify potential security vulnerabilities, compliance gaps, and policy infringements without explicit prompting for specific flaws. This challenge focuses on building a robust, observable agent workflow that can process unstructured data, cross-reference information, and provide actionable compliance reports.
What you are building
The core problem, expected build, and operating context for this challenge.
Build a multi-agent system using Claude Agents SDK and Claude Opus 4.6 for autonomous enterprise security and compliance analysis.
How work is evaluated
The agent system will be evaluated based on its ability to accurately identify security vulnerabilities and compliance gaps in provided document sets, the clarity and completeness of its generated reports, and the efficiency of its multi-agent collaboration.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
JSON Format Adherence
Verify that the output is a valid JSON object matching the specified schema.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Risk Identification
Check if at least 3 relevant risks are identified from a benchmark document set.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Risk Precision
Percentage of identified risks that are truly relevant and accurate. • target: 0.85 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Risk Recall
Percentage of actual risks in the documents that the agent successfully identified. • target: 0.8 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Report Completeness
Score based on the presence of summary, identified risks, and recommendations. • target: 0.9 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master the Claude Agents SDK for defining agent roles, capabilities, and inter-agent communication protocols.
Implement advanced prompt engineering techniques for Claude Opus 4.6 to maximize large context window utilization for intricate document scrutiny.
Design and deploy a multi-agent architecture where specialized agents (e.g., Policy Analyst, Security Auditor, Report Generator) collaborate on a shared objective.
Integrate Braintrust for real-time monitoring, tracing, and evaluation of agent decision-making and performance metrics.
Build a Streamlit dashboard to serve as an intuitive interface for inputting compliance tasks and visualizing agent-generated reports and identified risks.
Orchestrate a data pipeline that uses OpenVINO for efficient local inference of specialized classification models to preprocess or categorize documents before LLM analysis.
Implement LangFuse for granular tracing and debugging of complex agentic workflows, understanding state transitions and tool invocations.
Reference links and supporting material
Sample data for 1 tasks
Challenge setup
The agent system will be evaluated based on its ability to accurately identify security vulnerabilities and compliance gaps in provided document sets, the clarity and completeness of its generated reports, and the eff...
Challenge input
{"company_policies": [{"id": "str", "content": "str"}], "regulatory_documents": [{"id": "str", "content": "str"}]}
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{"report_summary": "str", "identified_risks": [{"type": "str", "description": "str", "source_docs": ["str"], "severity": "high|medium|low"}], "reco...
- Verify that the output is a valid JSON object matching the specified schema.
- Check if at least 3 relevant risks are identified from a benchmark document set.
- Percentage of identified risks that are truly relevant and accurate. • target: 0.85 • range: 0-1
- Risk Precision target: 0.85
- 1 public reference case
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- JSON Format Adherence · Weight 1
- Risk Identification · Weight 1
- Risk Precision · Weight 1
- Risk Recall · Weight 1
- Report Completeness · Weight 1
Metrics
- Risk Precision · Target 0.85
- Risk Recall · Target 0.8
- Report Completeness · Target 0.9
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: {"company_policies": [{"id": "str", "content": "str"}], "regulatory_documents": [{"id": "str", "content": "str"}]}
Output: {"report_summary": "str", "identified_risks": [{"type": "str", "description": "str", "source_docs": ["str"], "severity": "high|medium|low"}], "recommendations": [{"action": "str", "justification": "str"}]}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.