Skip to main content
Back to challenges
Challenge

Creative Integrity Orchestrator

In response to the WGA and AMPTP agreement regarding AI training protection and creative workflows, you will build an autonomous agent team that automates script review and IP protection. Using the OpenAI Agents SDK, you will orchestrate a multi-agent system where one agent specializes in deep reasoning about copyright law and union agreements, while another manages creative content generation. The system must verify that any generated content adheres to the 2026 labor agreements and does not inadvertently ingest restricted training data. You will implement a governance layer using Fairlearn to ensure the creative output is free from specific demographic biases that often plague large models. The workflow will be optimized by Yupp AI, which routes reasoning tasks to DeepSeek R1 for logic-heavy analysis and creative tasks to GPT-5.4-mini for rapid iteration. Finally, you will integrate Cognition (Devin) as an autonomous engineering agent to perform code reviews on the system's own tool-calling logic to prevent data leaks.

Special Purpose AgentsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build a multi-agent compliance system using OpenAI Agents SDK to orchestrate script development while enforcing WGA-inspired AI training protections.

Delivery guide

How work is evaluated

Evaluation

The system will be evaluated on its ability to detect a copyright-violating prompt and generate a creative script that passes a Fairlearn fairness audit.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

Fairness Threshold

Does the Fairlearn demographic parity difference stay below 0.1?

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

Reasoning Accuracy

DeepSeek R1 explanation quality score • target: 90 • range: 0-100

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Master the OpenAI Agents SDK for building role-based autonomous agents with shared memory and tool access

  • Orchestrate complex logic chains by delegating legal reasoning tasks to DeepSeek R1 via the Yupp AI routing layer

  • Implement Fairlearn assessment pipelines to evaluate and mitigate biases in AI-generated narrative content

  • Design autonomous code-correction loops with Cognition (Devin) to audit agentic tool usage and API calls

  • Build a multi-turn conversation manager that maintains script context across different agent specializations

  • Integrate Coplay AI as the primary interface for human-in-the-loop creative approval workflows

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 1 tasks

Challenge setup

The system will be evaluated on its ability to detect a copyright-violating prompt and generate a creative script that passes a Fairlearn fairness audit.

Challenge input

JSON containing user prompt and contract constraints

OpenAI

OpenAI AI model provider

RAI

Agentic framework for robotics using ROS 2

DeepSeek R1

Policy Serving in the agent workflow.

Evaluated output

Boolean compliance flag and reasoning string

Evaluation checks
  • Does the Fairlearn demographic parity difference stay below 0.1?
  • DeepSeek R1 explanation quality score • target: 90 • range: 0-100
Configured evaluation evidence
  • Reasoning Accuracy target: 90
  • 1 public reference case
Configured runtime
  • Python execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • OpenAI · Required
  • RAI · Optional
Policy Serving
  • DeepSeek R1 · Optional

Evaluation contract

  • Fairness Threshold · Weight 1
  • Reasoning Accuracy · Weight 1

Metrics

  • Reasoning Accuracy · Target 90

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON containing user prompt and contract constraints

    Output: Boolean compliance flag and reasoning string

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Creative Integrity Orchestrator