AI Underwriting Governance Auditor using AutoGen
Underwriters are increasingly required to serve as governance stewards for automated decision models. You will create a multi-agent debate and governance compliance verification network using AutoGen to inspect underwriting LLMs for bias and compliance violations. The multi-agent debate system must achieve 90% alignment with manual regulatory compliance audits.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an AI underwriting compliance audit system using AutoGen multi-agent debate conversations.
How work is evaluated
Evaluates compliance issue identification rate and accuracy of final audit consensus report.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
violation_detection_correctness
Ensures prohibited variables are flagged correctly.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
compliance_alignment_score
Alignment of agent consensus with ground-truth audit rules • target: 0.9 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Construct AutoGen ConversationalAgents configured for regulatory compliance auditing.
Implement automated debate protocols between AuditorAgent, UnderwriterAgent, and ComplianceOfficer.
Generate structured compliance audit reports highlighting potential regulatory flags.
Reference links and supporting material
Dataset of 100 sample underwriting prompts and model configurations tagged with regulatory compliance status across 5 US states.
Challenge setup
Evaluates compliance issue identification rate and accuracy of final audit consensus report.
Challenge input
JSON object with candidate_prompt, risk_variables (array), jurisdiction
AutoGen
Assigned framework for agent group conversation and debate.
Evaluated output
JSON object with compliance_passed (boolean), violations_detected (array), consensus_score
- Ensures prohibited variables are flagged correctly.
- Alignment of agent consensus with ground-truth audit rules • target: 0.9 • range: 0-1
- Benchmark: AI Safety & Governance Benchmark
- Compliance Alignment Score target: 0.9
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- AutoGen · Required
- Mistral Large 2 · Optional
- Fireworks AI · Optional
- Fireworks AI · Optional
Evaluation contract
- violation_detection_correctness · Weight 1
- compliance_alignment_score · Weight 1
Metrics
- compliance_alignment_score · Target 0.9
Benchmarks
- AI Safety & Governance Benchmark
- security
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON object with candidate_prompt, risk_variables (array), jurisdiction
Output: JSON object with compliance_passed (boolean), violations_detected (array), consensus_score
Execution harness: Python
Evaluation tools
- AutoGen · Multi-Agent Framework
Recipe freeze marker
No recipe freeze time is recorded.