Multi-Model Safety Evaluator with Claude Agents SDK and Triton
Addressing the recent report on robotaxi safety backsliding, this challenge tasks you with building a safety evaluation framework for autonomous systems. You will utilize the Claude Agents SDK and Claude Sonnet 4.6.6 to build a supervisor agent that audits the visual perception of other models. The system will deploy GPT-5.4 Pro and specialized vision models using Triton Inference Server and TorchServe for high-performance model serving. Your agents will use Claude's extended thinking capabilities to reason through complex traffic violation scenarios and use Speakeasy to generate integrations for simulation platform APIs. This project focuses on high-concurrency model deployment and cross-model reasoning to identify traffic safety risks in real-time video metadata.
What you are building
The core problem, expected build, and operating context for this challenge.
Addressing the recent report on robotaxi safety backsliding, this challenge tasks you with building a safety evaluation framework for autonomous systems. You will utilize the Claude Agents SDK and Claude Sonnet 4.6.6 to build a supervisor agent that audits the visual perception of other models. The system will deploy GPT-5.4 Pro and specialized vision models using Triton Inference Server and TorchServe for high-performance model serving. Your agents will use Claude's extended thinking capabilities to reason through complex traffic violation scenarios and use Speakeasy to generate integrations for simulation platform APIs. This project focuses on high-concurrency model deployment and cross-model reasoning to identify traffic safety risks in real-time video metadata.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
How submissions are scored
These dimensions define what the evaluator checks, how much each dimension matters, and which criteria separate a passable run from a strong one.
Model Synchronization
Triton and TorchServe must respond within 100ms of each other
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Safety Recall
Percentage of simulated violations correctly identified • target: 0.99 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Orchestrate Claude Sonnet 4.6.6 using the Claude Agents SDK for complex reasoning about traffic laws and safety violations
Configure Triton Inference Server to manage multiple model versions and optimize GPU utilization for real-time inference
Deploy specialized PyTorch safety models on TorchServe for granular object detection auditing
Implement Claude's extended thinking blocks to perform multi-step chain-of-thought analysis on road incident data
Master the use of Speakeasy to automate the generation of SDKs for disparate simulation and telemetry data sources
Build a consensus mechanism where Claude Opus 4.6.6 validates the outputs of lower-latency models before generating a safety report
[ok] Wrote CHALLENGE.md
[ok] Wrote .versalist.json
[ok] Wrote eval/examples.json
Requires VERSALIST_API_KEY. Works with any MCP-aware editor.
DocsAI Research & Mentorship
Participation status
You haven't started this challenge yet
Operating window
Key dates and the organization behind this challenge.
Find another challenge
Jump to a random challenge when you want a fresh benchmark or a different problem space.