Skip to main content
Back to challenges
Challenge

Agentic AI Chatbot Safety & Compliance with Gemini 2.5 Pro and CrewAI

Design and implement a cutting-edge agentic AI system inspired by California's new SB 243 law, which mandates safety protocols for AI chatbot companions. This challenge focuses on building a multi-agent system capable of proactively monitoring, identifying, and mitigating unsafe or non-compliant interactions in real-time. The system will leverage advanced LLMs for nuanced conversational understanding and ethical reasoning, orchestrating agents to ensure adherence to regulatory guidelines. Participants will construct a CrewAI-based team of specialized agents—a 'Conversation Monitor,' an 'Ethical Auditor,' and a 'Compliance Reporter'—that communicate via a robust A2A protocol. The system must integrate RAG capabilities for accessing regulatory documents and employ extended thinking with adaptive reasoning budgets to handle complex, ambiguous safety scenarios, generating detailed compliance logs and alerts for review.

Special Purpose AgentsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build an A2A protocol multi-agent system with CrewAI and Gemini 2.5 Pro for real-time AI chatbot safety and regulatory compliance.

Delivery guide

How work is evaluated

Evaluation

The evaluation will assess the system's ability to accurately identify safety violations, generate appropriate mitigation responses, and produce comprehensive compliance reports, with a focus on agent collaboration and efficient resource usage.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Learning goals

What you should walk away with

  • Design and implement a multi-agent system using CrewAI for AI chatbot safety monitoring.

  • Develop an A2A communication protocol for seamless agent collaboration.

  • Implement RAG for real-time access to regulatory documents and safety guidelines.

  • Build extended thinking pipelines with adaptive reasoning budgets to address complex safety scenarios.

  • Integrate Gemini 2.5 Pro Deep Think mode for advanced ethical and compliance reasoning.

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 2 tasks

Challenge setup

The evaluation will assess the system's ability to accurately identify safety violations, generate appropriate mitigation responses, and produce comprehensive compliance reports, with a focus on agent collaboration an...

Challenge input

JSON array of chat turn objects: [{'speaker': 'user'|'chatbot', 'text': '...'}]

Agent execution

No agent tool is listed for this challenge.

Evaluated output

JSON object: {'violations_detected': ['list of violation types'], 'flagged_segment': 'string', 'reasoning': 'string'}

Evaluation checks
  • Verify all CrewAI agents successfully communicate and complete their tasks within the workflow.
  • Check if relevant regulatory document excerpts are cited in compliance reports when applicable.
  • Confirm that the system intelligently applies extended thinking only for flagged complex scenarios.
Configured evaluation evidence
  • ViolationDetectionAccuracy target: 0.9
Configured runtime
  • Python execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

No tool records are attached.

Evaluation contract

  • Verify all CrewAI agents successfully communicate and complete their tasks within the workflow.
  • Check if relevant regulatory document excerpts are cited in compliance reports when applicable.
  • Confirm that the system intelligently applies extended thinking only for flagged complex scenarios.

Metrics

  • ViolationDetectionAccuracy · Target 0.9
  • FalsePositiveRate · Target 0.05
  • ReportCompletenessScore · Target 90

Workflow tasks

  1. Task 1

    Input: JSON array of chat turn objects: [{'speaker': 'user'|'chatbot', 'text': '...'}]

    Output: JSON object: {'violations_detected': ['list of violation types'], 'flagged_segment': 'string', 'reasoning': 'string'}

    Execution harness: Python

  2. Task 2

    Input: JSON array of detected violation objects from SafetyViolationDetection task.

    Output: Markdown string of a compliance report, including summary, incident details, and proposed mitigation.

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Agentic AI Chatbot Safety & Compliance with Gemini 2.5 Pro and CrewAI