Physical Device Safety Guard with Claude Agents SDK and Baseten Runtime
Operating physical equipment via AI agents requires enforceable hardware bounds and low-latency safety monitoring. You will develop a device execution agent using Claude Agents SDK deployed on Baseten model runtime to parse Model Hardware Standard (MHS) commands and intercept dangerous actuation signals. Success requires 100% detection and veto rate of simulated out-of-bound hardware commands within 50ms.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an MHS physical device operator agent using the Claude Agents SDK and deploy hardware verification logic on Baseten runtime.
How work is evaluated
Evaluates MHS safety agent's detection accuracy and veto latency on physical device command streams.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
zero_false_negatives
Ensures no unsafe command was allowed to pass to execution
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
veto_latency_ms
Average latency in milliseconds to evaluate and intercept unsafe commands • target: 30 • range: 0-50
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Implement a Claude Agents SDK agent enforcing Model Hardware Standard (MHS) safety envelopes
Package and deploy the validation guard model onto Baseten serverless inference endpoints
Intercept hardware actuation instructions (torque, speed, temperature limits) prior to physical execution
Achieve sub-50ms veto overhead on all high-risk hardware commands
Reference links and supporting material
Command logs from automated laboratory hardware and robotic arms with annotated safety threshold breaches.
Challenge setup
Evaluates MHS safety agent's detection accuracy and veto latency on physical device command streams.
Challenge input
JSON containing device_id, target_limits (max_torque_nm, max_temp_c), and candidate_commands array
Claude Agents SDK
Provides native agent tooling and extended thinking for high-integrity decision making.
Baseten
Delivers low-latency serverless model deployment for real-time safety vetoes.
Evaluated output
JSON detailing passed_commands, rejected_commands, veto_latency_ms, and safety_score
- Ensures no unsafe command was allowed to pass to execution
- Average latency in milliseconds to evaluate and intercept unsafe commands • target: 30 • range: 0-50
- Benchmark: RoboSafety-Eval
- Veto Latency Ms target: 30
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- Baseten · Required
- Claude Code · Optional
- Anthropic · Optional
- Claude Code · Optional
Evaluation contract
- zero_false_negatives · Weight 1
- veto_latency_ms · Weight 1
Metrics
- veto_latency_ms · Target 30
Benchmarks
- RoboSafety-Eval
- robotics
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON containing device_id, target_limits (max_torque_nm, max_temp_c), and candidate_commands array
Output: JSON detailing passed_commands, rejected_commands, veto_latency_ms, and safety_score
Execution harness: Python
Evaluation tools
- Claude Agents SDK · Agent Framework
- Baseten · Inference Engine
Recipe freeze marker
No recipe freeze time is recorded.