Automated Security Red-Teaming with OpenAI Agents SDK and Langfuse
Inspired by the emergence of automated vulnerability detection systems in enterprise environments, this challenge tasks developers with creating an agentic red-teaming pipeline. You will use the OpenAI Agents SDK to build specialized agents capable of probing software endpoints for prompt injection and security regressions. By integrating Langfuse for observability, you will track agent decision-making paths and failure points in real-time. This system will mimic the internal workflows now being adopted by major tech companies to secure LLM deployments before public release. You will utilize Claude Opus 4.6.6 as a reasoning engine within the agentic structure to evaluate vulnerabilities flagged by the primary agents. Sourcegraph Amp will be used to generate robust testing harnesses that bridge the gap between agent findings and actionable code patches.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an automated security red-teaming agentic pipeline using OpenAI Agents SDK, Claude Opus 4.6.6, and Langfuse for observability.
How work is evaluated
Evaluate the agent's effectiveness in identifying injection vectors and providing valid security patches.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
Detection Accuracy
Verify successful identification of known vulnerabilities
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Trace Coverage
Percentage of agent steps captured by Langfuse • target: 95 • range: 0-100
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Orchestrate OpenAI Agents SDK agents to perform multi-stage security audits
Implement Langfuse tracking to visualize agent thought sequences and tool calls
Design Steamship-based workflows for persistent agent storage and management
Master integration of Deepgram for voice-based audit logging and reporting
Build automated code correction loops using Sourcegraph Amp for rapid remediation
Reference links and supporting material
Sample data for 1 tasks
Challenge setup
Evaluate the agent's effectiveness in identifying injection vectors and providing valid security patches.
Challenge input
Target endpoint schema
OpenAI
OpenAI AI model provider
Sourcegraph Amp
Agentic coding tool from Sourcegraph.
Langfuse
Open-source LLM observability and evals.
Evaluated output
JSON report of identified vulnerability
- Verify successful identification of known vulnerabilities
- Percentage of agent steps captured by Langfuse • target: 95 • range: 0-100
- Trace Coverage target: 95
- 1 public reference case
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- OpenAI · Required
- Sourcegraph Amp · Optional
- Langfuse · Optional
- Sourcegraph Amp · Optional
- Langfuse · Optional
Evaluation contract
- Detection Accuracy · Weight 1
- Trace Coverage · Weight 1
Metrics
- Trace Coverage · Target 95
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: Target endpoint schema
Output: JSON report of identified vulnerability
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.