AI Underwriting Governance Auditor using AutoGen
Underwriters are increasingly required to serve as governance stewards for automated decision models. You will create a multi-agent debate and governance compliance verification network using AutoGen to inspect underwriting LLMs for bias and compliance violations. The multi-agent debate system must achieve 90% alignment with manual regulatory compliance audits.
What you are building
Build an AI underwriting compliance audit system using AutoGen multi-agent debate conversations.
How work is evaluated
Evaluates compliance issue identification rate and accuracy of final audit consensus report.
Datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
violation_detection_correctness
Ensures prohibited variables are flagged correctly.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
compliance_alignment_score
Alignment of agent consensus with ground-truth audit rules • target: 0.9 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Learning goals
Construct AutoGen ConversationalAgents configured for regulatory compliance auditing.
Implement automated debate protocols between AuditorAgent, UnderwriterAgent, and ComplianceOfficer.
Generate structured compliance audit reports highlighting potential regulatory flags.
Resources
Dataset of 100 sample underwriting prompts and model configurations tagged with regulatory compliance status across 5 US states.
Challenge setup
Evaluates compliance issue identification rate and accuracy of final audit consensus report.
Challenge input
JSON object with candidate_prompt, risk_variables (array), jurisdiction
AutoGen
Assigned framework for agent group conversation and debate.
Evaluated output
JSON object with compliance_passed (boolean), violations_detected (array), consensus_score
- Ensures prohibited variables are flagged correctly.
- Alignment of agent consensus with ground-truth audit rules • target: 0.9 • range: 0-1
- Benchmark: AI Safety & Governance Benchmark
- Compliance Alignment Score target: 0.9
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Shows the first task, up to three tools, and up to three evidence items.
View technical recipe
Configured tools
- AutoGen · Required
- Mistral Large 2 · Optional
- Fireworks AI · Optional
- Fireworks AI · Optional
Evaluation contract
- violation_detection_correctness · Weight 1
- compliance_alignment_score · Weight 1
Metrics
- compliance_alignment_score · Target 0.9
Benchmarks
- AI Safety & Governance Benchmark
- security
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON object with candidate_prompt, risk_variables (array), jurisdiction
Output: JSON object with compliance_passed (boolean), violations_detected (array), consensus_score
Execution harness: Python
Evaluation tools
- AutoGen · Multi-Agent Framework
Recipe freeze marker
No recipe freeze time is recorded.