Hardware Observability Pipeline for Token, Watt, and Cost Attribution with LangChain
Design a hardware observability pipeline using LangChain and OpenTelemetry to attribute energy usage, GPU queue time, and cost to specific tenant agent sessions. The pipeline must reconcile billing metrics within 1% accuracy across 10,000 concurrent streaming requests.
What you are building
Build a stateful LangChain/LangGraph workflow integrated with OpenTelemetry and Prometheus to log fine-grained hardware metrics.
How work is evaluated
Evaluates cost and energy attribution accuracy across multi-tenant trace runs.
Datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
Reconciliation Accuracy Test
Cost attribution error margin must be under 1.0%.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Attribution Precision
Precision of chargeback mapping to active session spans. • target: 0.998 • range: 0.99-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Learning goals
Build a telemetry aggregation pipeline using LangChain and LangGraph.
Correlate OpenTelemetry trace spans with GPU watt consumption and token counts.
Implement multi-tenant chargeback calculation algorithms.
Export structured metric telemetry to Prometheus dashboards.
Resources
Prometheus metric snapshots and OpenTelemetry spans for 10,000 synthetic agent requests.
Challenge setup
Evaluates cost and energy attribution accuracy across multi-tenant trace runs.
Challenge input
JSON array of trace events with NVML power samples
LangChain
Standard framework for building composable chain and graph workflows
OpenTelemetry
Provides distributed tracing and span context instrumentation
Evaluated output
JSON report containing cost_attribution_by_tenant and error_margin_percent
- Cost attribution error margin must be under 1.0%.
- Precision of chargeback mapping to active session spans. • target: 0.998 • range: 0.99-1
- Benchmark: LLM Infrastructure Observability Suite
- Attribution Precision target: 0.998
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Shows the first task, up to three tools, and up to three evidence items.
View technical recipe
Configured tools
- Langchain · Required
- LangChain · Optional
- LangGraph · Optional
- Langchain · Required
- LangChain · Optional
- LangGraph · Optional
Evaluation contract
- Reconciliation Accuracy Test · Weight 1
- Attribution Precision · Weight 1
Metrics
- Attribution Precision · Target 0.998
Benchmarks
- LLM Infrastructure Observability Suite
- general
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON array of trace events with NVML power samples
Output: JSON report containing cost_attribution_by_tenant and error_margin_percent
Execution harness: Python
Evaluation tools
- LangChain · Agent Framework
- OpenTelemetry · Observability Protocol
Recipe freeze marker
No recipe freeze time is recorded.