Autonomous Enterprise Security Compliance Agent with Claude Opus 4.6
Develop an advanced autonomous agent system using the Claude Agents SDK that leverages Claude Opus 4.6's 1M token context window and agentic capabilities to scrutinize large volumes of enterprise documents, regulatory filings, and internal policies. The agent team will identify potential security vulnerabilities, compliance gaps, and policy infringements without explicit prompting for specific flaws. This challenge focuses on building a robust, observable agent workflow that can process unstructured data, cross-reference information, and provide actionable compliance reports.
What you are building
The core problem, expected build, and operating context for this challenge.
Develop an advanced autonomous agent system using the Claude Agents SDK that leverages Claude Opus 4.6's 1M token context window and agentic capabilities to scrutinize large volumes of enterprise documents, regulatory filings, and internal policies. The agent team will identify potential security vulnerabilities, compliance gaps, and policy infringements without explicit prompting for specific flaws. This challenge focuses on building a robust, observable agent workflow that can process unstructured data, cross-reference information, and provide actionable compliance reports.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
JSON Format Adherence
Verify that the output is a valid JSON object matching the specified schema.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Risk Identification
Check if at least 3 relevant risks are identified from a benchmark document set.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Risk Precision
Percentage of identified risks that are truly relevant and accurate. • target: 0.85 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Risk Recall
Percentage of actual risks in the documents that the agent successfully identified. • target: 0.8 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Report Completeness
Score based on the presence of summary, identified risks, and recommendations. • target: 0.9 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master the Claude Agents SDK for defining agent roles, capabilities, and inter-agent communication protocols.
Implement advanced prompt engineering techniques for Claude Opus 4.6 to maximize large context window utilization for intricate document scrutiny.
Design and deploy a multi-agent architecture where specialized agents (e.g., Policy Analyst, Security Auditor, Report Generator) collaborate on a shared objective.
Integrate Braintrust for real-time monitoring, tracing, and evaluation of agent decision-making and performance metrics.
Build a Streamlit dashboard to serve as an intuitive interface for inputting compliance tasks and visualizing agent-generated reports and identified risks.
Orchestrate a data pipeline that uses OpenVINO for efficient local inference of specialized classification models to preprocess or categorize documents before LLM analysis.
Implement LangFuse for granular tracing and debugging of complex agentic workflows, understanding state transitions and tool invocations.
How this agent runs
The agent system will be evaluated based on its ability to accurately identify security vulnerabilities and compliance gaps in provided document sets, the clarity and completeness of its generated reports, and the eff...
Challenge input
{"company_policies": [{"id": "str", "content": "str"}], "regulatory_documents": [{"id": "str", "content": "str"}]}
Agent execution
The configured agent processes the input under the challenge policy.
Evaluated output
{"report_summary": "str", "identified_risks": [{"type": "str", "description": "str", "source_docs": ["str"], "severity": "high|medium|low"}], "reco...
- Verify that the output is a valid JSON object matching the specified schema.
- Check if at least 3 relevant risks are identified from a benchmark document set.
- Percentage of identified risks that are truly relevant and accurate. • target: 0.85 • range: 0-1
- Risk Precision target: 0.85
- 1 public reference case
- Python execution harness
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- JSON Format Adherence · Weight 1
- Risk Identification · Weight 1
- Risk Precision · Weight 1
- Risk Recall · Weight 1
- Report Completeness · Weight 1
Recipe state
This is a preview. The configuration can change before the evaluation recipe is locked.
Run this agent on your dataset and AI stack
Bring your dataset, model providers, and success criteria. We will scope the right managed run for your team.
Scope a managed run[ok] Wrote CHALLENGE.md
[ok] Wrote .versalist.json
[ok] Wrote eval/examples.json
Requires VERSALIST_API_KEY. Works with any MCP-aware editor.
DocsFind another challenge
Jump to a random challenge when you want a fresh benchmark or a different problem space.