Automated Security Red-Teaming with OpenAI Agents SDK and Langfuse
Inspired by the emergence of automated vulnerability detection systems in enterprise environments, this challenge tasks developers with creating an agentic red-teaming pipeline. You will use the OpenAI Agents SDK to build specialized agents capable of probing software endpoints for prompt injection and security regressions. By integrating Langfuse for observability, you will track agent decision-making paths and failure points in real-time. This system will mimic the internal workflows now being adopted by major tech companies to secure LLM deployments before public release. You will utilize Claude Opus 4.6.6 as a reasoning engine within the agentic structure to evaluate vulnerabilities flagged by the primary agents. Sourcegraph Amp will be used to generate robust testing harnesses that bridge the gap between agent findings and actionable code patches.
What you are building
The core problem, expected build, and operating context for this challenge.
Inspired by the emergence of automated vulnerability detection systems in enterprise environments, this challenge tasks developers with creating an agentic red-teaming pipeline. You will use the OpenAI Agents SDK to build specialized agents capable of probing software endpoints for prompt injection and security regressions. By integrating Langfuse for observability, you will track agent decision-making paths and failure points in real-time. This system will mimic the internal workflows now being adopted by major tech companies to secure LLM deployments before public release. You will utilize Claude Opus 4.6.6 as a reasoning engine within the agentic structure to evaluate vulnerabilities flagged by the primary agents. Sourcegraph Amp will be used to generate robust testing harnesses that bridge the gap between agent findings and actionable code patches.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
Detection Accuracy
Verify successful identification of known vulnerabilities
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Trace Coverage
Percentage of agent steps captured by Langfuse • target: 95 • range: 0-100
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Orchestrate OpenAI Agents SDK agents to perform multi-stage security audits
Implement Langfuse tracking to visualize agent thought sequences and tool calls
Design Steamship-based workflows for persistent agent storage and management
Master integration of Deepgram for voice-based audit logging and reporting
Build automated code correction loops using Sourcegraph Amp for rapid remediation
How this agent runs
Evaluate the agent's effectiveness in identifying injection vectors and providing valid security patches.
Challenge input
Target endpoint schema
OpenAI
OpenAI AI model provider
Sourcegraph Amp
Agentic coding tool from Sourcegraph.
Langfuse
Open-source LLM observability and evals.
Evaluated output
JSON report of identified vulnerability
- Verify successful identification of known vulnerabilities
- Percentage of agent steps captured by Langfuse • target: 95 • range: 0-100
- Trace Coverage target: 95
- 1 public reference case
- Python execution harness
View technical recipe
Configured tools
- OpenAI · Required
- Sourcegraph Amp · Optional
- Langfuse · Optional
- Sourcegraph Amp · Optional
- Langfuse · Optional
Evaluation contract
- Detection Accuracy · Weight 1
- Trace Coverage · Weight 1
Recipe state
This is a preview. The configuration can change before the evaluation recipe is locked.
Run this agent on your dataset and AI stack
Bring your dataset, model providers, and success criteria. We will scope the right managed run for your team.
Scope a managed run[ok] Wrote CHALLENGE.md
[ok] Wrote .versalist.json
[ok] Wrote eval/examples.json
Requires VERSALIST_API_KEY. Works with any MCP-aware editor.
DocsFind another challenge
Jump to a random challenge when you want a fresh benchmark or a different problem space.