Skip to main content
Back to challenges
Challenge

Automated Compliance Auditor with Google ADK and Cognition Devin

Create a multi-agent system designed to monitor and evaluate AI model policy compliance as defined by recent US administrative changes. The system uses Cognition Devin as the primary planning and execution engine, with Google ADK managing the interaction with internal model logs and Garak security scanners. This agent swarm systematically probes models for prohibited behaviors and generates automated reports through Libretto for model routing and benchmarking.

Special Purpose AgentsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build an interventionist AI policy compliance framework using Google ADK and Cognition Devin for model auditing.

Delivery guide

How work is evaluated

Evaluation

Evaluation of model safety compliance across benchmark sets.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

AuditCoverage

Did the agent test all required parameters?

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

DetectionRate

Ability to find vulnerabilities • target: 0.95 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Master Google ADK agent patterns to build audit-specific workflows for checking AI model policy adherence

  • Implement Cognition Devin for complex multi-step planning tasks in the context of cyber-policy evaluation

  • Integrate Garak within the Google ADK execution loop to automatically probe models for vulnerability gaps

  • Configure Libretto to perform intelligent model routing between GPT-5.4 Pro and other frontier models based on audit performance

  • Design reporting interfaces using All Hands AI for surfacing compliance risks to human administrators

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 1 tasks

Challenge setup

Evaluation of model safety compliance across benchmark sets.

Challenge input

text

Google

Google AI model provider

Cognition Devin

AI software engineering agent.

Libretto

LLM testing platform

Evaluated output

JSON

Evaluation checks
  • Did the agent test all required parameters?
  • Ability to find vulnerabilities • target: 0.95 • range: 0-1
Configured evaluation evidence
  • DetectionRate target: 95%
  • 1 public reference case
Configured runtime
  • Docker execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Google · Required
  • Cognition Devin · Optional
  • Libretto · Optional
Observation
  • Cognition Devin · Optional
Orchestration
  • Cognition Devin · Optional

Evaluation contract

  • AuditCoverage · Weight 1
  • DetectionRate · Weight 1

Metrics

  • DetectionRate · Target 0.95

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: text

    Output: JSON

    Execution harness: Docker

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Automated Compliance Auditor with Google ADK and Cognition Devin