Skip to main content
Back to challenges
Challenge

Multi-Agent Ad Policy Auditor

Develop an advanced multi-agent system using AutoGen to autonomously audit proposed advertising strategies against simulated user conversation data. This system will focus on ensuring compliance with privacy policies and verifying ad claims before deployment. Agents will collaborate to extract topics, propose ad targeting, review privacy implications, and fact-check promotional content, providing explainable insights into their decisions. The challenge emphasizes building robust, privacy-aware AI systems that can operate with human oversight where needed.

Special Purpose AgentsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build a multi-agent system using AutoGen to audit ad targeting for privacy and compliance, leveraging Gemini 3 Flash for topic extraction and LIME for explainability.

Delivery guide

How work is evaluated

Evaluation

The evaluation will focus on the system's ability to accurately audit ad strategies, flag privacy violations, and provide clear, actionable explanations. Success will be measured by both automated tests and qualitative review of generated reports.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Learning goals

What you should walk away with

  • Master AutoGen for orchestrating complex, conversational agent workflows with custom roles, message passing, and human-in-the-loop capabilities.

  • Implement data simulation strategies to generate realistic user conversation topics and personalization data, serving as input for ad targeting analysis.

  • Integrate Gemini 2.5 Flash as a specialized agent within the AutoGen system for rapid and cost-effective topic extraction and initial ad strategy generation.

  • Utilize LIME (Local Interpretable Model-agnostic Explanations) to generate post-hoc explanations for agent decisions, particularly regarding which user data features influence ad targeting or privacy violation flags.

  • Design a dedicated 'Privacy Auditor' agent within AutoGen to apply simulated privacy policies, identify potential data misuse, and flag non-compliant ad strategies.

  • Orchestrate the AutoGen agent interactions using Capably to monitor workflow execution, manage task distribution, and trigger alerts or generate compliance summaries.

  • Develop robust evaluation metrics and a testing harness to assess the accuracy, safety, and explainability of generated ad strategies and audit reports.

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 1 tasks

Challenge setup

The evaluation will focus on the system's ability to accurately audit ad strategies, flag privacy violations, and provide clear, actionable explanations. Success will be measured by both automated tests and qualitativ...

Challenge input

{'user_topics': ['list of strings'], 'ad_proposal': {'title': 'string', 'target_audience': 'string', 'key_phrases': ['list of strings']}, 'privacy_...

Agent execution

No agent tool is listed for this challenge.

Evaluated output

{'audited_ad_strategy': {'title': 'string', 'target_audience': 'string', 'key_phrases': ['list of strings']}, 'privacy_violations': ['list of strin...

Evaluation checks
  • Verify if the 'privacy_violations' list correctly identifies policy breaches based on simulated data and pr...
  • Check if LIME explainability reports are present and contain meaningful text.
Configured evaluation evidence
  • Audit Accuracy Score target: 90
Configured runtime
  • Python execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

No tool records are attached.

Evaluation contract

  • Verify if the 'privacy_violations' list correctly identifies policy breaches based on simulated data and provided policies.
  • Check if LIME explainability reports are present and contain meaningful text.

Metrics

  • Audit Accuracy Score · Target 90
  • Explainability Clarity Score · Target 4

Workflow tasks

  1. Task 1

    Input: {'user_topics': ['list of strings'], 'ad_proposal': {'title': 'string', 'target_audience': 'string', 'key_phrases': ['list of strings']}, 'privacy_policy': 'string'}

    Output: {'audited_ad_strategy': {'title': 'string', 'target_audience': 'string', 'key_phrases': ['list of strings']}, 'privacy_violations': ['list of strings'], 'explainability_reports': {'LIME_ad_targeting': 'string', 'LIME_privacy_flag': 'string'}, 'compliance_status': 'PASS|FAIL'}

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Multi-Agent Ad Policy Auditor