Skip to main content
Back to challenges
Challenge

Type-Safe Token and Power Attribution Engine with Pydantic AI

Attribute hardware power consumption and financial cost directly to individual tenant API requests. Build a type-safe token telemetry and energy accounting service using Pydantic AI that parses Prometheus hardware metrics and maps Watts and execution time back to tenant request contexts.

EngineeringHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build a type-safe energy attribution engine with Pydantic AI to validate, parse, and assign server Watt consumption to specific tenant token generation requests.

Delivery guide

How work is evaluated

Evaluation

Evaluates Pydantic schema validation accuracy and energy attribution calculations.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

Pydantic Schema Validation

Ensure output strictly conforms to Pydantic model structure.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

Attribution Accuracy

Precision of calculated energy consumption vs energy formula. • target: 1 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Define strict Pydantic schemas for multi-tenant power and token usage payloads.

  • Build a Pydantic AI agent agentic pipeline to extract power draw from Prometheus time-series metrics.

  • Accurately calculate micro-dollar and Joules-per-token metrics per query.

Resources and assets

Reference links and supporting material

Dataset notes

Prometheus energy metrics dataset containing GPU Watt measurements and tenant request timestamps.

Challenge setup

Evaluates Pydantic schema validation accuracy and energy attribution calculations.

Challenge input

JSON tenant_id, execution_time_s, avg_power_draw_watts, energy_cost_per_kwh

Pydantic AI

Type-safe Python agent framework built on Pydantic.

Evaluated output

JSON valid Pydantic model containing energy_consumed_joules, financial_cost_usd

Evaluation checks
  • Ensure output strictly conforms to Pydantic model structure.
  • Precision of calculated energy consumption vs energy formula. • target: 1 • range: 0-1
Configured evaluation evidence
  • Benchmark: Green AI Hardware Telemetry Standard
  • Attribution Accuracy target: 1
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Pydantic AI · Required
  • Anthropic · Optional
  • Anthropic Claude · Optional
Orchestration
  • Pydantic AI · Required

Evaluation contract

  • Pydantic Schema Validation · Weight 1
  • Attribution Accuracy · Weight 1

Metrics

  • Attribution Accuracy · Target 1

Benchmarks

  • Green AI Hardware Telemetry Standard
  • energy

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON tenant_id, execution_time_s, avg_power_draw_watts, energy_cost_per_kwh

    Output: JSON valid Pydantic model containing energy_consumed_joules, financial_cost_usd

    Execution harness: Python

Evaluation tools

  • Pydantic AI · Agent Framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Type-Safe Token and Power Attribution Engine with Pydantic AI