Public evaluation

Violation Detection

The evaluation measures the system's ability to detect traffic violations in simulation logs and the latency of the multi-model inference pipeline.

Evaluation type

task based

Challenge

Multi-Model Safety Evaluator with Claude Agents SDK and Triton

Difficulty

Advanced

Rigor

Unspecified

Open linked challenge

Evaluation overview

How the linked challenge is judged: tasks, benchmarks, and criteria count.

Tasks

Benchmarks

Criteria

Task templates

Inputs and expected outputs.

Task 1

Violation Detection

Identify a failure to stop at a red light in a telemetry stream

Input format

JSON of vehicle position and signal state

Output format

Boolean violation flag and reasoning trace

Linked challenge

Multi-Model Safety Evaluator with Claude Agents SDK and Triton

Addressing the recent report on robotaxi safety backsliding, this challenge tasks you with building a safety evaluation framework for autonomous systems. You will utilize the Claude Agents SDK and Claude Sonnet 4.6.6 to build a supervisor agent that audits the visual perception of other models. The system will deploy GPT-5.4 Pro and specialized vision models using Triton Inference Server and TorchServe for high-performance model serving. Your agents will use Claude's extended thinking capabilities to reason through complex traffic violation scenarios and use Speakeasy to generate integrations for simulation platform APIs. This project focuses on high-concurrency model deployment and cross-model reasoning to identify traffic safety risks in real-time video metadata.

Open challenge