Skip to main content
Back to challenges
Challenge

AI Underwriting Governance Steward & Guardrail Validator

As underwriting transitions to automated AI decisions, establishing model governance and guardrails is paramount. Build a Pydantic AI governance monitor that audits underwriting model decisions in real-time, verifying compliance with anti-discrimination rules and institutional risk appetites.

Machine LearningHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Implement an automated AI underwriting governance steward using Pydantic AI to enforce compliance guardrails.

Delivery guide

How work is evaluated

Evaluation

Evaluates governance audit pass/fail precision and illegal proxy detection.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

violation_code_format

Verifies violation code schema format

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

governance_detection_f1

F1 score in identifying compliance violations • target: 0.95 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Design governance audit rules checking for prohibited underwriting proxies

  • Deploy Pydantic AI guardrails enforcing transparent decision reasoning

  • Monitor automated rate adjustments for fair lending / non-discrimination compliance

  • Generate real-time compliance audit pass/fail metrics

Resources and assets

Reference links and supporting material

Dataset notes

Synthetic automated insurance underwriting decision traces with annotated regulatory violations.

Challenge setup

Evaluates governance audit pass/fail precision and illegal proxy detection.

Challenge input

JSON containing underwriting AI decision inputs, factors, and output premium.

Pydantic AI

Structured output audit validation

Evaluated output

JSON containing audit_passed boolean, violation_codes list, and explanation.

Evaluation checks
  • Verifies violation code schema format
  • F1 score in identifying compliance violations • target: 0.95 • range: 0-1
Configured evaluation evidence
  • Benchmark: AIGovernanceBench
  • Governance Detection F1 target: 0.95
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • RAI · Required
  • Pydantic AI · Optional
  • llama.cpp · Optional
Orchestration
  • Pydantic AI · Optional

Evaluation contract

  • violation_code_format · Weight 1
  • governance_detection_f1 · Weight 1

Metrics

  • governance_detection_f1 · Target 0.95

Benchmarks

  • AIGovernanceBench
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON containing underwriting AI decision inputs, factors, and output premium.

    Output: JSON containing audit_passed boolean, violation_codes list, and explanation.

    Execution harness: Python

Evaluation tools

  • Pydantic AI · Type-Safe Agent Framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about AI Underwriting Governance Steward & Guardrail Validator