Automated Benchmark Noise Filter and GPU/TPU Profiler with LlamaIndex
Infrastructure noise like thermal throttling, cloud host co-tenancy, and background process spikes distorts agentic benchmark evaluations. You will build an automated noise-filtering agent using LlamaIndex index structures to ingest raw execution logs, isolate infrastructure anomalies, and compute clean true-latency baselines. Success is achieved by identifying 100% of injected latency outliers while maintaining baseline accuracy.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an infrastructure noise filter and benchmark analyzer using LlamaIndex document stores and structured extraction query engines.
How work is evaluated
Evaluates LlamaIndex noise filter accuracy when processing noisy benchmark telemetry.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
outlier_precision_recall
Ensures all throttled runs were identified and no normal runs were misclassified
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
variance_reduction_pct
Percentage reduction in latency variance after filtering infrastructure noise • target: 95 • range: 0-100
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Build a LlamaIndex Document Store and VectorIndex holding system metrics and benchmark event logs
Design custom LlamaIndex Query Engines that identify hardware-induced variance vs model capability drops
Filter out CPU thermal throttling spikes and network jitter from benchmark scores
Calculate normalized execution variance metrics across repetitive eval runs
Reference links and supporting material
Telemetry logs from evaluation harness execution runs containing artificially injected background CPU noise and throttling.
How this agent runs
Evaluates LlamaIndex noise filter accuracy when processing noisy benchmark telemetry.
Challenge input
JSON containing raw_runs array with run_id, latency_ms, and host_cpu_throttled_pct
LlamaIndex
Provides data indexing, vector search, and query engine capabilities for telemetry log analysis.
Evaluated output
JSON returning clean_mean_latency_ms, detected_outlier_run_ids, and noise_variance_reduced_pct
- Ensures all throttled runs were identified and no normal runs were misclassified
- Percentage reduction in latency variance after filtering infrastructure noise • target: 95 • range: 0-100
- Benchmark: Eval-Noise-Reduction-Bench
- Variance Reduction Pct target: 95
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
View technical recipe
Configured tools
- Llama Index · Required
- LlamaIndex · Optional
- Llama 3.2 · Optional
Evaluation contract
- outlier_precision_recall · Weight 1
- variance_reduction_pct · Weight 1
Recipe state
This is a preview. The configuration can change before the evaluation recipe is locked.