AWS Neuron vs GPU Accelerator Profiler with LlamaIndex and Llama 3.3 70B
Evaluate accelerator performance for large-scale RAG indexing using LlamaIndex and Llama 3.3 70B across AWS Neuron and NVIDIA TensorRT-LLM specs. Build an automated profiler that dynamically selects optimal accelerator topologies to minimize latency and unit cost under strict SLA bounds.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an accelerator benchmarking pipeline in LlamaIndex using Llama 3.3 70B specs to compute real-time throughput, latency, and cost per million tokens across hardware architectures.
How work is evaluated
Evaluates profiler accelerator recommendation accuracy and cost-latency tradeoff calculations.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
SLA Compliance Test
Check if chosen target meets the target SLA requirement.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Cost Optimization Efficiency
Measures ratio of optimal cost vs chosen hardware cost. • target: 0.95 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Benchmark Llama 3.3 70B token generation throughput across simulated AWS Trainium/Inferentia and NVIDIA H100 specs.
Integrate LlamaIndex custom vector store routing based on dynamic accelerator profiling metrics.
Compute auditable cost and latency tradeoffs per query workload.
Reference links and supporting material
Hardware specification metrics dataset containing memory bandwidth, compute TFLOPS, and pricing for AWS Neuron and NVIDIA GPUs.
Challenge setup
Evaluates profiler accelerator recommendation accuracy and cost-latency tradeoff calculations.
Challenge input
JSON containing batch size, prompt length, target SLA in ms
LlamaIndex
Required data framework for LLM workflows.
Llama 3.3 70B
Target frontier open model spec.
Evaluated output
JSON containing chosen accelerator, expected latency, cost per 1M tokens
- Check if chosen target meets the target SLA requirement.
- Measures ratio of optimal cost vs chosen hardware cost. • target: 0.95 • range: 0-1
- Benchmark: Roofline Model Performance Benchmark
- Cost Optimization Efficiency target: 0.95
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- Llama Index · Required
- LlamaIndex · Optional
- Llama 3.3 70B · Optional
Evaluation contract
- SLA Compliance Test · Weight 1
- Cost Optimization Efficiency · Weight 1
Metrics
- Cost Optimization Efficiency · Target 0.95
Benchmarks
- Roofline Model Performance Benchmark
- general
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON containing batch size, prompt length, target SLA in ms
Output: JSON containing chosen accelerator, expected latency, cost per 1M tokens
Execution harness: Python
Evaluation tools
- LlamaIndex · Agent Framework
- Llama 3.3 70B · Model
Recipe freeze marker
No recipe freeze time is recorded.