Multi-Model Safety Evaluator with Claude Agents SDK and Triton
Addressing the recent report on robotaxi safety backsliding, this challenge tasks you with building a safety evaluation framework for autonomous systems. You will utilize the Claude Agents SDK and Claude Sonnet 4.6.6 to build a supervisor agent that audits the visual perception of other models. The system will deploy GPT-5.4 Pro and specialized vision models using Triton Inference Server and TorchServe for high-performance model serving. Your agents will use Claude's extended thinking capabilities to reason through complex traffic violation scenarios and use Speakeasy to generate integrations for simulation platform APIs. This project focuses on high-concurrency model deployment and cross-model reasoning to identify traffic safety risks in real-time video metadata.
What you are building
The core problem, expected build, and operating context for this challenge.
Addressing the recent report on robotaxi safety backsliding, this challenge tasks you with building a safety evaluation framework for autonomous systems. You will utilize the Claude Agents SDK and Claude Sonnet 4.6.6 to build a supervisor agent that audits the visual perception of other models. The system will deploy GPT-5.4 Pro and specialized vision models using Triton Inference Server and TorchServe for high-performance model serving. Your agents will use Claude's extended thinking capabilities to reason through complex traffic violation scenarios and use Speakeasy to generate integrations for simulation platform APIs. This project focuses on high-concurrency model deployment and cross-model reasoning to identify traffic safety risks in real-time video metadata.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
Model Synchronization
Triton and TorchServe must respond within 100ms of each other
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Safety Recall
Percentage of simulated violations correctly identified • target: 0.99 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Orchestrate Claude Sonnet 4.6.6 using the Claude Agents SDK for complex reasoning about traffic laws and safety violations
Configure Triton Inference Server to manage multiple model versions and optimize GPU utilization for real-time inference
Deploy specialized PyTorch safety models on TorchServe for granular object detection auditing
Implement Claude's extended thinking blocks to perform multi-step chain-of-thought analysis on road incident data
Master the use of Speakeasy to automate the generation of SDKs for disparate simulation and telemetry data sources
Build a consensus mechanism where Claude Opus 4.6.6 validates the outputs of lower-latency models before generating a safety report
How this agent runs
The evaluation measures the system's ability to detect traffic violations in simulation logs and the latency of the multi-model inference pipeline.
Challenge input
JSON of vehicle position and signal state
Triton Inference Server
NVIDIA multi-framework inference serving solution
TorchServe
Inference & Model Runtime · Model Serving
Evaluated output
Boolean violation flag and reasoning trace
- Triton and TorchServe must respond within 100ms of each other
- Percentage of simulated violations correctly identified • target: 0.99 • range: 0-1
- Safety Recall target: 0.99
- 1 public reference case
- Docker execution harness
View technical recipe
Configured tools
- Triton Inference Server · Required
- TorchServe · Optional
- Triton Inference Server · Optional
Evaluation contract
- Model Synchronization · Weight 1
- Safety Recall · Weight 1
Recipe state
This is a preview. The configuration can change before the evaluation recipe is locked.
Run this agent on your dataset and AI stack
Bring your dataset, model providers, and success criteria. We will scope the right managed run for your team.
Scope a managed run[ok] Wrote CHALLENGE.md
[ok] Wrote .versalist.json
[ok] Wrote eval/examples.json
Requires VERSALIST_API_KEY. Works with any MCP-aware editor.
DocsFind another challenge
Jump to a random challenge when you want a fresh benchmark or a different problem space.