Enterprise AI Governance and MAS Model Audit Automation Harness
Singapore's AI expansion requires stringent model governance frameworks (such as MAS FEAT Principles: Fairness, Ethics, Accountability, and Transparency) for financial and Smart Nation deployments. Regulators require automated evaluation harnesses that continuously test model outputs for bias, hallucination, and drift before models are approved for production deployment. In this challenge, you will implement an AI Governance Audit harness using Google ADK and Gemini. The harness runs automated Red-Teaming, Fairness checks across demographic groups, and Transparency logging for enterprise LLM agents.
What you are building
The core problem, expected build, and operating context for this challenge.
Develop an automated MAS FEAT governance evaluation harness using Google ADK and Gemini.
How work is evaluated
Evaluates fairness scoring accuracy and generation of MAS FEAT compliance certificates.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
fairness_threshold_check
Must flag failure when Disparate Impact Ratio falls below 0.80.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
eval_throughput_samples_sec
Number of model decisions evaluated per second • target: 120 • range: 1-500
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Build an evaluation agent using Google ADK to audit enterprise LLM decision outputs
Evaluate model decisions against MAS FEAT (Fairness, Ethics, Accountability, Transparency) standards
Detect disparate impact ratios across protected demographic attributes in credit scoring outputs
Export standardized AI Governance Compliance Certificates for regulatory submission
Reference links and supporting material
Enterprise AI credit decision logs, customer demographic attributes, and model reasoning trace logs.
Challenge setup
Evaluates fairness scoring accuracy and generation of MAS FEAT compliance certificates.
Challenge input
JSON array of model approval decisions with demographic metadata
Google ADK
Required framework for building AI governance evaluation agents.
Evaluated output
JSON with disparate_impact_ratio, pass_fairness boolean, and flagged biases
- Must flag failure when Disparate Impact Ratio falls below 0.80.
- Number of model decisions evaluated per second • target: 120 • range: 1-500
- Benchmark: AIGovBench
- Eval Throughput Samples Sec target: 120
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- Google ADK · Required
- Google · Optional
- Google Gemini · Optional
Evaluation contract
- fairness_threshold_check · Weight 1
- eval_throughput_samples_sec · Weight 1
Metrics
- eval_throughput_samples_sec · Target 120
Benchmarks
- AIGovBench
- general
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON array of model approval decisions with demographic metadata
Output: JSON with disparate_impact_ratio, pass_fairness boolean, and flagged biases
Execution harness: Python
Evaluation tools
- Google ADK · Agent Framework
Recipe freeze marker
No recipe freeze time is recorded.