Multi-Model Safety Evaluator with Claude Agents SDK and Triton
Addressing the recent report on robotaxi safety backsliding, this challenge tasks you with building a safety evaluation framework for autonomous systems. You will utilize the Claude Agents SDK and Claude Sonnet 4.6.6 to build a supervisor agent that audits the visual perception of other models. The system will deploy GPT-5.4 Pro and specialized vision models using Triton Inference Server and TorchServe for high-performance model serving. Your agents will use Claude's extended thinking capabilities to reason through complex traffic violation scenarios and use Speakeasy to generate integrations for simulation platform APIs. This project focuses on high-concurrency model deployment and cross-model reasoning to identify traffic safety risks in real-time video metadata.
What you are building
Build a high-performance safety auditing system using Claude Agents SDK and Triton Inference Server for multi-model verification.
How work is evaluated
The evaluation measures the system's ability to detect traffic violations in simulation logs and the latency of the multi-model inference pipeline.
Datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
Model Synchronization
Triton and TorchServe must respond within 100ms of each other
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Safety Recall
Percentage of simulated violations correctly identified • target: 0.99 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Learning goals
Orchestrate Claude Sonnet 4.6.6 using the Claude Agents SDK for complex reasoning about traffic laws and safety violations
Configure Triton Inference Server to manage multiple model versions and optimize GPU utilization for real-time inference
Deploy specialized PyTorch safety models on TorchServe for granular object detection auditing
Implement Claude's extended thinking blocks to perform multi-step chain-of-thought analysis on road incident data
Master the use of Speakeasy to automate the generation of SDKs for disparate simulation and telemetry data sources
Build a consensus mechanism where Claude Opus 4.6.6 validates the outputs of lower-latency models before generating a safety report
Resources
Sample data for 1 tasks
Challenge setup
The evaluation measures the system's ability to detect traffic violations in simulation logs and the latency of the multi-model inference pipeline.
Challenge input
JSON of vehicle position and signal state
Triton Inference Server
NVIDIA multi-framework inference serving solution
TorchServe
Inference & Model Runtime · Model Serving
Evaluated output
Boolean violation flag and reasoning trace
- Triton and TorchServe must respond within 100ms of each other
- Percentage of simulated violations correctly identified • target: 0.99 • range: 0-1
- Safety Recall target: 0.99
- 1 public reference case
- Docker execution harness
Shows the first task, up to three tools, and up to three evidence items.
View technical recipe
Configured tools
- Triton Inference Server · Required
- TorchServe · Optional
- Triton Inference Server · Optional
Evaluation contract
- Model Synchronization · Weight 1
- Safety Recall · Weight 1
Metrics
- Safety Recall · Target 0.99
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON of vehicle position and signal state
Output: Boolean violation flag and reasoning trace
Execution harness: Docker
Recipe freeze marker
No recipe freeze time is recorded.