AI Policy Audit Agent with OpenAI Agents
Develop an autonomous AI agent leveraging the OpenAI Agents SDK to assist in auditing frontier AI models for policy compliance and ethical guidelines. This agent will ingest large volumes of policy documents, ethical frameworks, and internal model documentation, performing sophisticated RAG to identify potential risks, non-compliance, or areas requiring further human review. Persistent memory via Mem0 will allow the agent to maintain context across multiple audit sessions and learn from prior findings, enhancing its capabilities over time. The system will integrate with Supabase for vector storage of documents and OpenRouter for resilient model access and cost monitoring.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an auditable RAG-centric AI policy agent using OpenAI Agents SDK with persistent memory for compliance checks of frontier AI models.
How work is evaluated
The agent will be evaluated on its ability to accurately identify policy violations and provide actionable recommendations based on a given set of simulated audit documents. Evaluation includes precision, recall, and the quality of generated summaries and recommendations.
Challenge datasets
Public datasets, plus the private datasets you have access to.
What you should walk away with
Master the OpenAI Agents SDK for building sophisticated, multi-turn conversational agents with tool use and state management.
Implement persistent, long-term memory for your agent using Mem0, understanding its API for session recall and knowledge accretion.
Design and populate a vector database in Supabase (Vector) for efficient RAG, optimizing embedding strategies for policy documents.
Integrate `GPT-5-2` for advanced reasoning, natural language understanding, and complex policy analysis within your agent's workflow.
Utilize OpenRouter for routing AI model requests, enabling capabilities like fallback models, cost optimization, and unified API access for OpenAI models.
Develop custom tools for the OpenAI Agents SDK to interact with external systems, such as document parsers or internal policy databases.
Build an evaluation harness to assess the agent's accuracy in identifying policy non-compliance and ethical risks.
Reference links and supporting material
Sample data for 2 tasks
Challenge setup
The agent will be evaluated on its ability to accurately identify policy violations and provide actionable recommendations based on a given set of simulated audit documents. Evaluation includes precision, recall, and...
Challenge input
JSON: {'document_id': 'string', 'text': 'string', 'policy_area': 'string'}
Agent execution
No agent tool is listed for this challenge.
Evaluated output
JSON: {'document_id': 'string', 'violations': [{'violation_type': 'string', 'description': 'string', 'severity': 'low|medium|high', 'policy_referen...
- Checks if the agent's output for PolicyViolationIdentification matches the expected JSON format.
- Checks if the agent's output for RecommendationGeneration matches the expected JSON format.
- Violation F1 Score target: 0.85
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Checks if the agent's output for PolicyViolationIdentification matches the expected JSON format.
- Checks if the agent's output for RecommendationGeneration matches the expected JSON format.
Metrics
- Violation_F1_Score · Target 0.85
- Recommendation_Quality_Score · Target 0.9
Workflow tasks
- Task 1
Input: JSON: {'document_id': 'string', 'text': 'string', 'policy_area': 'string'}
Output: JSON: {'document_id': 'string', 'violations': [{'violation_type': 'string', 'description': 'string', 'severity': 'low|medium|high', 'policy_reference': 'string'}], 'summary': 'string'}
Execution harness: Python
- Task 2
Input: JSON: {'violation': 'string', 'policy_reference': 'string', 'context_summary': 'string'}
Output: JSON: {'recommendations': [{'action': 'string', 'priority': 'low|medium|high', 'justification': 'string'}]}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.