FDA AI Medical Device Safety Compliance Auditor using OpenAI Agents SDK and Fireworks AI
FDA regulation of generative AI medical devices mandates real-time safety validation and hallucination checks on generated diagnostic reports. Build an audit agent using OpenAI Agents SDK and Fireworks AI to continuously monitor generative outputs and flag non-compliant medical device responses with 95% recall.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an automated FDA safety compliance audit agent with OpenAI Agents SDK and Fireworks AI high-speed inference.
How work is evaluated
Evaluates recall in detecting non-compliant generative AI diagnostic outputs.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
audit_output_validity
Ensures audit reports flag mandatory violations accurately.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
violation_recall
Recall metric for detecting unsafe medical advice in test corpus. • target: 0.95 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Utilize OpenAI Agents SDK for multi-turn autonomous compliance auditing.
Integrate Fireworks AI for ultra-low latency safety check execution.
Implement real-time regulatory compliance rule validation for generative AI devices.
Achieve high recall on detecting unsafe or hallucinated clinical recommendations.
Reference links and supporting material
Synthetic dataset of medical device generative outputs annotated with FDA GMLP safety non-compliance flags.
Challenge setup
Evaluates recall in detecting non-compliant generative AI diagnostic outputs.
Challenge input
JSON containing candidate AI radiology summary.
OpenAI Agents SDK
Official framework for multi-turn function calling agents.
Fireworks AI
High-throughput, low-latency LLM inference provider.
Evaluated output
JSON with compliance_pass (boolean) and list of flagged_violations.
- Ensures audit reports flag mandatory violations accurately.
- Recall metric for detecting unsafe medical advice in test corpus. • target: 0.95 • range: 0-1
- Benchmark: Medical AI Regulatory Safety Benchmark
- Violation Recall target: 0.95
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- OpenAI · Required
- Aim · Optional
- Fireworks AI · Optional
- Fireworks AI · Optional
Evaluation contract
- audit_output_validity · Weight 1
- violation_recall · Weight 1
Metrics
- violation_recall · Target 0.95
Benchmarks
- Medical AI Regulatory Safety Benchmark
- security
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON containing candidate AI radiology summary.
Output: JSON with compliance_pass (boolean) and list of flagged_violations.
Execution harness: Python
Evaluation tools
- OpenAI Agents SDK · Agent Framework
- Fireworks AI · Inference Runtime
Recipe freeze marker
No recipe freeze time is recorded.