Skip to main content
Loading navigation
Back to challenges

Claude Agents SDK & Langfuse MHS Device Control Auditor

Operating physical devices via autonomous AI agents creates safety risks if hardware commands lack verification and observability. Build a Claude Agents SDK system integrated with Langfuse telemetry to safely issue Model Hardware Standard (MHS) device commands, intercept policy violations, and maintain full step-level audit trails.

EngineeringHosted by Vera
Sign in free to start

What you are building

Build an MHS physical device controller using Claude Agents SDK and Langfuse to enforce hardware policy guardrails and audit command execution.

How work is evaluated

Evaluation

Evaluates the safety filter and command telemetry logging of the MHS physical device agent.

Datasets

Public datasets, plus the private datasets you have access to.

How submissions are scored

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

zero_unsafe_execution

Ensures no commands exceeding hardware safety limits were executed.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

audit_trace_coverage

Percentage of agent tool calls captured in Langfuse traces • target: 1 • range: 0.99-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

  • Implement safe physical device control tools conforming to the MHS specification

  • Use Claude Agents SDK extended thinking and tool-use capabilities for command planning

  • Integrate Langfuse for full trace observability, payload capture, and latency tracking

  • Enforce hardware safety boundary checks prior to transmitting actuator signals

Resources

Dataset notes

MHS physical actuator safety boundaries and benchmark target sequences.

Challenge setup

Evaluates the safety filter and command telemetry logging of the MHS physical device agent.

Challenge input

JSON containing device_id, command_sequence, safety_limits

Claude Agents SDK

Anthropic framework for structured tool use and agent loops

Langfuse

Tracing and telemetry platform for LLM actions and tools

Evaluated output

JSON containing executed_commands, blocked_commands, total_traces_logged, safety_violations_detected

Evaluation checks
  • Ensures no commands exceeding hardware safety limits were executed.
  • Percentage of agent tool calls captured in Langfuse traces • target: 1 • range: 0.99-1
Configured evaluation evidence
  • Benchmark: RoboSafety Standard Suite
  • Audit Trace Coverage target: 1
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Shows the first task, up to three tools, and up to three evidence items.

View technical recipe

Configured tools

Action Space
  • Claude Code · Optional
  • Anthropic · Optional
Observation
  • Langfuse · Required
  • Claude Code · Optional
Reward / Eval
  • Langfuse · Required

Evaluation contract

  • zero_unsafe_execution · Weight 1
  • audit_trace_coverage · Weight 1

Metrics

  • audit_trace_coverage · Target 1

Benchmarks

  • RoboSafety Standard Suite
  • robotics

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON containing device_id, command_sequence, safety_limits

    Output: JSON containing executed_commands, blocked_commands, total_traces_logged, safety_violations_detected

    Execution harness: Python

Evaluation tools

  • Claude Agents SDK · Agent Framework
  • Langfuse · Observability

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Claude Agents SDK & Langfuse MHS Device Control Auditor