Automated Benchmark Noise Filter and GPU/TPU Profiler with LlamaIndex
Infrastructure noise like thermal throttling, cloud host co-tenancy, and background process spikes distorts agentic benchmark evaluations. You will build an automated noise-filtering agent using LlamaIndex index structures to ingest raw execution logs, isolate infrastructure anomalies, and compute clean true-latency baselines. Success is achieved by identifying 100% of injected latency outliers while maintaining baseline accuracy.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an infrastructure noise filter and benchmark analyzer using LlamaIndex document stores and structured extraction query engines.
How work is evaluated
Evaluates LlamaIndex noise filter accuracy when processing noisy benchmark telemetry.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
outlier_precision_recall
Ensures all throttled runs were identified and no normal runs were misclassified
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
variance_reduction_pct
Percentage reduction in latency variance after filtering infrastructure noise • target: 95 • range: 0-100
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Build a LlamaIndex Document Store and VectorIndex holding system metrics and benchmark event logs
Design custom LlamaIndex Query Engines that identify hardware-induced variance vs model capability drops
Filter out CPU thermal throttling spikes and network jitter from benchmark scores
Calculate normalized execution variance metrics across repetitive eval runs
Reference links and supporting material
Telemetry logs from evaluation harness execution runs containing artificially injected background CPU noise and throttling.
Challenge setup
Evaluates LlamaIndex noise filter accuracy when processing noisy benchmark telemetry.
Challenge input
JSON containing raw_runs array with run_id, latency_ms, and host_cpu_throttled_pct
LlamaIndex
Provides data indexing, vector search, and query engine capabilities for telemetry log analysis.
Evaluated output
JSON returning clean_mean_latency_ms, detected_outlier_run_ids, and noise_variance_reduced_pct
- Ensures all throttled runs were identified and no normal runs were misclassified
- Percentage reduction in latency variance after filtering infrastructure noise • target: 95 • range: 0-100
- Benchmark: Eval-Noise-Reduction-Bench
- Variance Reduction Pct target: 95
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- Llama Index · Required
- LlamaIndex · Optional
- Llama 3.2 · Optional
Evaluation contract
- outlier_precision_recall · Weight 1
- variance_reduction_pct · Weight 1
Metrics
- variance_reduction_pct · Target 95
Benchmarks
- Eval-Noise-Reduction-Bench
- general
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON containing raw_runs array with run_id, latency_ms, and host_cpu_throttled_pct
Output: JSON returning clean_mean_latency_ms, detected_outlier_run_ids, and noise_variance_reduced_pct
Execution harness: Python
Evaluation tools
- LlamaIndex · Agent Framework
Recipe freeze marker
No recipe freeze time is recorded.