Back to challenges
Challenge

AWS Neuron vs GPU Accelerator Profiler with LlamaIndex and Llama 3.3 70B

Evaluate accelerator performance for large-scale RAG indexing using LlamaIndex and Llama 3.3 70B across AWS Neuron and NVIDIA TensorRT-LLM specs. Build an automated profiler that dynamically selects optimal accelerator topologies to minimize latency and unit cost under strict SLA bounds.

EngineeringHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build an accelerator benchmarking pipeline in LlamaIndex using Llama 3.3 70B specs to compute real-time throughput, latency, and cost per million tokens across hardware architectures.

Delivery guide

How work is evaluated

Evaluation

Evaluates profiler accelerator recommendation accuracy and cost-latency tradeoff calculations.

Datasets

Shared data for this challenge

Review public datasets and any private uploads tied to your build.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

SLA Compliance Test

Check if chosen target meets the target SLA requirement.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

Cost Optimization Efficiency

Measures ratio of optimal cost vs chosen hardware cost. • target: 0.95 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Benchmark Llama 3.3 70B token generation throughput across simulated AWS Trainium/Inferentia and NVIDIA H100 specs.

  • Integrate LlamaIndex custom vector store routing based on dynamic accelerator profiling metrics.

  • Compute auditable cost and latency tradeoffs per query workload.

Resources and assets

Reference links and supporting material

Dataset notes

Hardware specification metrics dataset containing memory bandwidth, compute TFLOPS, and pricing for AWS Neuron and NVIDIA GPUs.

How this agent runs

Evaluates profiler accelerator recommendation accuracy and cost-latency tradeoff calculations.

Challenge input

JSON containing batch size, prompt length, target SLA in ms

LlamaIndex

Required data framework for LLM workflows.

Llama 3.3 70B

Target frontier open model spec.

Evaluated output

JSON containing chosen accelerator, expected latency, cost per 1M tokens

Checks for
  • Check if chosen target meets the target SLA requirement.
  • Measures ratio of optimal cost vs chosen hardware cost. • target: 0.95 • range: 0-1
Proof of success
  • Benchmark: Roofline Model Performance Benchmark
  • Cost Optimization Efficiency target: 0.95
  • 1 public reference case
Runtime evidence
  • Python execution harness
  • Python sandbox (unavailable on Versalist)
View technical recipe

Configured tools

Action Space
  • Llama Index · Required
  • LlamaIndex · Optional
  • Llama 3.3 70B · Optional

Evaluation contract

  • SLA Compliance Test · Weight 1
  • Cost Optimization Efficiency · Weight 1

Recipe state

This is a preview. The configuration can change before the evaluation recipe is locked.

Frequently Asked Questions about AWS Neuron vs GPU Accelerator Profiler with LlamaIndex and Llama 3.3 70B