Skip to main content
Loading navigation
Back to challenges

OpenAI Agents SDK Hardware Telemetry & Energy Attribution System

Multi-tenant cloud infrastructure often fails to accurately attribute energy consumption and accelerator costs to individual model workloads. Build an OpenAI Agents SDK system that parses Prometheus and OpenTelemetry metrics to allocate energy (Joules), power (Watts), and cost ($) down to specific tenant requests.

EngineeringHosted by Vera
Sign in free to start

What you are building

Construct an OpenAI Agents SDK telemetry pipeline that correlates token generation with GPU energy usage and tenant cost attribution.

How work is evaluated

Evaluation

Tests tenant energy attribution accuracy given overlapping multi-tenant inference workloads.

Datasets

Public datasets, plus the private datasets you have access to.

How submissions are scored

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

energy_balance_check

Checks that total attributed energy equals total energy input.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

attribution_precision

Fidelity of tenant cost and power attribution • target: 0.99 • range: 0.95-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

  • Use OpenAI Agents SDK function calling to ingest OpenTelemetry hardware metrics

  • Attribute GPU power draw (via NVML/Prometheus) to specific tenant request sessions

  • Calculate energy usage per 1k prompt and generation tokens

  • Generate tenant billback reports detailing compute time, energy, and financial cost

Resources

Dataset notes

Prometheus metric snapshots containing NVML energy counter streams and API gateway request logs.

Challenge setup

Tests tenant energy attribution accuracy given overlapping multi-tenant inference workloads.

Challenge input

JSON containing total_energy_joules, active_tenants, tenant_token_logs

OpenAI Agents SDK

OpenAI function calling and agent orchestration SDK

Evaluated output

JSON containing tenant_attribution_joules, tenant_cost_usd, unassigned_overhead_pct

Evaluation checks
  • Checks that total attributed energy equals total energy input.
  • Fidelity of tenant cost and power attribution • target: 0.99 • range: 0.95-1
Configured evaluation evidence
  • Benchmark: Cloud FinOps Cost Attribution Standard
  • Attribution Precision target: 0.99
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Shows the first task, up to three tools, and up to three evidence items.

View technical recipe

Configured tools

Action Space
  • OpenAI · Required
  • Anthropic · Optional
  • Azure OpenAI Service · Optional

Evaluation contract

  • energy_balance_check · Weight 1
  • attribution_precision · Weight 1

Metrics

  • attribution_precision · Target 0.99

Benchmarks

  • Cloud FinOps Cost Attribution Standard
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON containing total_energy_joules, active_tenants, tenant_token_logs

    Output: JSON containing tenant_attribution_joules, tenant_cost_usd, unassigned_overhead_pct

    Execution harness: Python

Evaluation tools

  • OpenAI Agents SDK · Agent Framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about OpenAI Agents SDK Hardware Telemetry & Energy Attribution System