Skip to main content
Back to challenges
Challenge

Hardware Observability Pipeline for Token, Watt, and Cost Attribution with LangChain

Design a hardware observability pipeline using LangChain and OpenTelemetry to attribute energy usage, GPU queue time, and cost to specific tenant agent sessions. The pipeline must reconcile billing metrics within 1% accuracy across 10,000 concurrent streaming requests.

EngineeringHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build a stateful LangChain/LangGraph workflow integrated with OpenTelemetry and Prometheus to log fine-grained hardware metrics.

Delivery guide

How work is evaluated

Evaluation

Evaluates cost and energy attribution accuracy across multi-tenant trace runs.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

Reconciliation Accuracy Test

Cost attribution error margin must be under 1.0%.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

Attribution Precision

Precision of chargeback mapping to active session spans. • target: 0.998 • range: 0.99-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Build a telemetry aggregation pipeline using LangChain and LangGraph.

  • Correlate OpenTelemetry trace spans with GPU watt consumption and token counts.

  • Implement multi-tenant chargeback calculation algorithms.

  • Export structured metric telemetry to Prometheus dashboards.

Resources and assets

Reference links and supporting material

Dataset notes

Prometheus metric snapshots and OpenTelemetry spans for 10,000 synthetic agent requests.

Challenge setup

Evaluates cost and energy attribution accuracy across multi-tenant trace runs.

Challenge input

JSON array of trace events with NVML power samples

LangChain

Standard framework for building composable chain and graph workflows

OpenTelemetry

Provides distributed tracing and span context instrumentation

Evaluated output

JSON report containing cost_attribution_by_tenant and error_margin_percent

Evaluation checks
  • Cost attribution error margin must be under 1.0%.
  • Precision of chargeback mapping to active session spans. • target: 0.998 • range: 0.99-1
Configured evaluation evidence
  • Benchmark: LLM Infrastructure Observability Suite
  • Attribution Precision target: 0.998
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Langchain · Required
  • LangChain · Optional
  • LangGraph · Optional
Orchestration
  • Langchain · Required
  • LangChain · Optional
  • LangGraph · Optional

Evaluation contract

  • Reconciliation Accuracy Test · Weight 1
  • Attribution Precision · Weight 1

Metrics

  • Attribution Precision · Target 0.998

Benchmarks

  • LLM Infrastructure Observability Suite
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON array of trace events with NVML power samples

    Output: JSON report containing cost_attribution_by_tenant and error_margin_percent

    Execution harness: Python

Evaluation tools

  • LangChain · Agent Framework
  • OpenTelemetry · Observability Protocol

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Hardware Observability Pipeline for Token, Watt, and Cost Attribution with LangChain