Skip to main content
Back to challenges
Challenge

OpenAI Agents SDK Hardware Telemetry & Energy Attribution System

Multi-tenant cloud infrastructure often fails to accurately attribute energy consumption and accelerator costs to individual model workloads. Build an OpenAI Agents SDK system that parses Prometheus and OpenTelemetry metrics to allocate energy (Joules), power (Watts), and cost ($) down to specific tenant requests.

EngineeringHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Construct an OpenAI Agents SDK telemetry pipeline that correlates token generation with GPU energy usage and tenant cost attribution.

Delivery guide

How work is evaluated

Evaluation

Tests tenant energy attribution accuracy given overlapping multi-tenant inference workloads.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

energy_balance_check

Checks that total attributed energy equals total energy input.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

attribution_precision

Fidelity of tenant cost and power attribution • target: 0.99 • range: 0.95-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Use OpenAI Agents SDK function calling to ingest OpenTelemetry hardware metrics

  • Attribute GPU power draw (via NVML/Prometheus) to specific tenant request sessions

  • Calculate energy usage per 1k prompt and generation tokens

  • Generate tenant billback reports detailing compute time, energy, and financial cost

Resources and assets

Reference links and supporting material

Dataset notes

Prometheus metric snapshots containing NVML energy counter streams and API gateway request logs.

Challenge setup

Tests tenant energy attribution accuracy given overlapping multi-tenant inference workloads.

Challenge input

JSON containing total_energy_joules, active_tenants, tenant_token_logs

OpenAI Agents SDK

OpenAI function calling and agent orchestration SDK

Evaluated output

JSON containing tenant_attribution_joules, tenant_cost_usd, unassigned_overhead_pct

Evaluation checks
  • Checks that total attributed energy equals total energy input.
  • Fidelity of tenant cost and power attribution • target: 0.99 • range: 0.95-1
Configured evaluation evidence
  • Benchmark: Cloud FinOps Cost Attribution Standard
  • Attribution Precision target: 0.99
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • OpenAI · Required
  • Anthropic · Optional
  • Azure OpenAI Service · Optional

Evaluation contract

  • energy_balance_check · Weight 1
  • attribution_precision · Weight 1

Metrics

  • attribution_precision · Target 0.99

Benchmarks

  • Cloud FinOps Cost Attribution Standard
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON containing total_energy_joules, active_tenants, tenant_token_logs

    Output: JSON containing tenant_attribution_joules, tenant_cost_usd, unassigned_overhead_pct

    Execution harness: Python

Evaluation tools

  • OpenAI Agents SDK · Agent Framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about OpenAI Agents SDK Hardware Telemetry & Energy Attribution System