Skip to main content
Loading navigation
Back to challenges

Pydantic AI & Claude 4 Sonnet Telemetry & Token Power Attribution Engine

Create a structured telemetry and power attribution engine using Pydantic AI and Claude 4 Sonnet to map token generation to hardware energy metrics. Build energy-aware cost calculators that accurately attribute watt-hours and cloud expenses per tenant with under 2% error.

EngineeringHosted by Vera
Sign in free to start

What you are building

Design a type-safe power and cost telemetry attribution system using Pydantic AI and Claude 4 Sonnet to model hardware energy per output token.

How work is evaluated

Evaluation

Evaluates accuracy of per-tenant power and energy attribution metrics.

Datasets

Public datasets, plus the private datasets you have access to.

How submissions are scored

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

error_below_threshold

Ensures attribution error is under 2.0%.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

joules_per_token_accuracy

Precision of calculated energy per token. • target: 0.99 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

  • Build a Pydantic AI agent with structured validation for dynamic hardware telemetry parsing

  • Integrate OpenTelemetry metrics and NVIDIA Management Library (NVML) power usage traces

  • Attribute real-time GPU/TPU wattage and energy usage (Joules/token) to distinct multi-tenant API users

  • Achieve less than 2% attribution discrepancy against hardware energy meters

Resources

Dataset notes

Dataset of GPU power draw logs recorded at 100ms intervals during multi-tenant parallel LLM generation sessions.

Challenge setup

Evaluates accuracy of per-tenant power and energy attribution metrics.

Challenge input

JSON telemetry log with timestamped token outputs and power trace samples (Watts).

Pydantic AI

Type-safe Python agent framework with strict validation.

Claude 4 Sonnet

Powers deep telemetry reasoning and report generation.

Evaluated output

JSON breakdown of total Joules, Joules/token, and estimated electricity cost per tenant.

Evaluation checks
  • Ensures attribution error is under 2.0%.
  • Precision of calculated energy per token. • target: 0.99 • range: 0-1
Configured evaluation evidence
  • Benchmark: Green AI Hardware Telemetry Standard
  • Joules Per Token Accuracy target: 0.99
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Shows the first task, up to three tools, and up to three evidence items.

View technical recipe

Configured tools

Action Space
  • Pydantic AI · Required
  • Claude Code · Optional
Observation
  • Claude Code · Optional
Policy Serving
  • Claude 4 Sonnet · Optional
Orchestration
  • Pydantic AI · Required

Evaluation contract

  • error_below_threshold · Weight 1
  • joules_per_token_accuracy · Weight 1

Metrics

  • joules_per_token_accuracy · Target 0.99

Benchmarks

  • Green AI Hardware Telemetry Standard
  • energy

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON telemetry log with timestamped token outputs and power trace samples (Watts).

    Output: JSON breakdown of total Joules, Joules/token, and estimated electricity cost per tenant.

    Execution harness: Python

Evaluation tools

  • Pydantic AI · Agent Framework
  • Claude 4 Sonnet · Models

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Pydantic AI & Claude 4 Sonnet Telemetry & Token Power Attribution Engine