AI Safety Guardrail System for Generative Content
Inspired by recent concerns regarding the generation of unsafe content by large language models, this challenge tasks developers with building a robust, real-time AI safety guardrail system. The system must actively monitor, evaluate, and, if necessary, block or rephrase outputs from a generative AI model to prevent the proliferation of harmful or nonconsensual content, particularly focusing on multi-modal inputs and outputs. It should leverage an agentic architecture to apply policy-driven moderation.
What you are building
The core problem, expected build, and operating context for this challenge.
Develop a multi-agent system using CrewAI and multi-modal models to enforce real-time safety guardrails for generative AI outputs across text and images.
How work is evaluated
The evaluation will assess the system's ability to accurately detect and prevent unsafe content generation across various modalities (text, image description) while minimizing false positives and maintaining reasonable latency. It will also examine the traceability of moderation decisions.
Challenge datasets
Public datasets, plus the private datasets you have access to.
What you should walk away with
Master CrewAI for building specialized, role-based agent teams dedicated to content analysis, policy enforcement, and output remediation.
Implement multi-modal input processing with OpenAI GPT-5 for analyzing both text prompts and generated images for safety violations.
Design and integrate `Guardrails AI` to define and enforce explicit content policies, ensuring structured and compliant generative outputs.
Build a prompt engineering pipeline using `Weights & Biases (W&B Prompts)` for experimenting with and tracing moderation prompts and model behavior.
Orchestrate a real-time monitoring and feedback loop where `Claude Opus 4.5` acts as a meta-moderator, evaluating the effectiveness of the initial guardrail agents.
Develop strategies for handling edge cases, prompt injection attempts, and adversarial inputs to bypass safety mechanisms.
Reference links and supporting material
Sample data for 2 tasks
Challenge setup
The evaluation will assess the system's ability to accurately detect and prevent unsafe content generation across various modalities (text, image description) while minimizing false positives and maintaining reasonabl...
Challenge input
{"prompt": "string"}
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{"moderated_output": "string", "is_safe": "boolean", "policy_violated": "array<string>", "moderation_agent_trace": "json"}
- For all unsafe sample inputs, 'is_safe' must be false and 'moderated_output' must indicate refusal or rephr...
- For unsafe inputs, 'policy_violated' must contain at least one relevant safety policy.
- FalseNegativeRate target: 1%
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- For all unsafe sample inputs, 'is_safe' must be false and 'moderated_output' must indicate refusal or rephrasing.
- For unsafe inputs, 'policy_violated' must contain at least one relevant safety policy.
Metrics
- FalseNegativeRate · Target 0.01
- ModerationLatency · Target 500
- AgentTraceCompleteness · Target 0.9
Workflow tasks
- Task 1
Input: {"prompt": "string"}
Output: {"moderated_output": "string", "is_safe": "boolean", "policy_violated": "array<string>", "moderation_agent_trace": "json"}
Execution harness: Python
- Task 2
Input: {"prompt": "string", "simulated_image_description": "string"}
Output: {"moderated_output": "string", "is_safe": "boolean", "policy_violated": "array<string>", "multi_modal_agent_trace": "json"}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.