LlamaIndex Infrastructure Noise & Benchmark Variance Analyzer
Infrastructure noise such as noisy neighbors and network jitter obscures true performance improvements in agentic benchmarks. Build a LlamaIndex RAG and workflow framework that indexes benchmark execution logs, computes statistical variance confidence intervals, and isolates true algorithmic progress from hardware noise.
What you are building
Construct a LlamaIndex RAG and statistical workflow engine to quantify hardware infrastructure noise and filter benchmark variance.
How work is evaluated
Evaluates noise reduction and statistical significance detection across benchmark test runs.
Datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
significance_test_accuracy
Ensures correct determination of statistical significance.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
noise_identification_f1
F1-score for identifying specific root causes of infrastructure noise • target: 0.94 • range: 0.85-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Learning goals
Index benchmark execution logs using LlamaIndex vector indices and document parsers
Build a statistical analysis workflow to separate variance from model score improvements
Calculate 95% confidence intervals and p-values for agentic evaluation runs
Identify hardware noise factors such as CPU throttling and background OS tasks
Resources
Execution benchmark run logs containing pass rates, host CPU steal times, and disk IOPS counters.
Challenge setup
Evaluates noise reduction and statistical significance detection across benchmark test runs.
Challenge input
JSON containing baseline_scores, treatment_scores, host_metrics_logs
LlamaIndex
Framework for indexing document logs and statistical analysis tools
Evaluated output
JSON containing is_statistically_significant, p_value, noise_variance_ratio, identified_noise_source
- Ensures correct determination of statistical significance.
- F1-score for identifying specific root causes of infrastructure noise • target: 0.94 • range: 0.85-1
- Benchmark: SWE-bench Infrastructure Variance Suite
- Noise Identification F1 target: 0.94
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Shows the first task, up to three tools, and up to three evidence items.
View technical recipe
Configured tools
- Llama Index · Required
- LlamaIndex · Optional
- Llama 3.2 · Optional
Evaluation contract
- significance_test_accuracy · Weight 1
- noise_identification_f1 · Weight 1
Metrics
- noise_identification_f1 · Target 0.94
Benchmarks
- SWE-bench Infrastructure Variance Suite
- general
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON containing baseline_scores, treatment_scores, host_metrics_logs
Output: JSON containing is_statistically_significant, p_value, noise_variance_ratio, identified_noise_source
Execution harness: Python
Evaluation tools
- LlamaIndex · RAG Framework
Recipe freeze marker
No recipe freeze time is recorded.