Hardware Observability Pipeline for Token, Watt, and Cost Attribution with LangChain
Design a hardware observability pipeline using LangChain and OpenTelemetry to attribute energy usage, GPU queue time, and cost to specific tenant agent sessions. The pipeline must reconcile billing metrics within 1% accuracy across 10,000 concurrent streaming requests.
What you are building
The core problem, expected build, and operating context for this challenge.
Build a stateful LangChain/LangGraph workflow integrated with OpenTelemetry and Prometheus to log fine-grained hardware metrics.
How work is evaluated
Evaluates cost and energy attribution accuracy across multi-tenant trace runs.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
Reconciliation Accuracy Test
Cost attribution error margin must be under 1.0%.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Attribution Precision
Precision of chargeback mapping to active session spans. • target: 0.998 • range: 0.99-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Build a telemetry aggregation pipeline using LangChain and LangGraph.
Correlate OpenTelemetry trace spans with GPU watt consumption and token counts.
Implement multi-tenant chargeback calculation algorithms.
Export structured metric telemetry to Prometheus dashboards.
Reference links and supporting material
Prometheus metric snapshots and OpenTelemetry spans for 10,000 synthetic agent requests.
How this agent runs
Evaluates cost and energy attribution accuracy across multi-tenant trace runs.
Challenge input
JSON array of trace events with NVML power samples
LangChain
Standard framework for building composable chain and graph workflows
OpenTelemetry
Provides distributed tracing and span context instrumentation
Evaluated output
JSON report containing cost_attribution_by_tenant and error_margin_percent
- Cost attribution error margin must be under 1.0%.
- Precision of chargeback mapping to active session spans. • target: 0.998 • range: 0.99-1
- Benchmark: LLM Infrastructure Observability Suite
- Attribution Precision target: 0.998
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
View technical recipe
Configured tools
- Langchain · Required
- LangChain · Optional
- LangGraph · Optional
- Langchain · Required
- LangChain · Optional
- LangGraph · Optional
Evaluation contract
- Reconciliation Accuracy Test · Weight 1
- Attribution Precision · Weight 1
Recipe state
This is a preview. The configuration can change before the evaluation recipe is locked.