Back to challenges
Challenge

AWS Neuron vs vLLM Serving Benchmark Harness with OpenAI Agents SDK

Deploying models safely across heterogeneous cloud environments requires benchmarking AWS Trainium/Inferentia against vLLM GPU targets. You will build an automated benchmarking agent using the OpenAI Agents SDK to execute parallel workload tests, measure throughput (tok/s), and calculate cost per million tokens. The agent must generate an auditable trade-off matrix with less than 3% execution variance.

EngineeringHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build an automated cross-hardware profiling agent using the OpenAI Agents SDK to evaluate AWS Neuron and vLLM target workloads.

Delivery guide

How work is evaluated

Evaluation

Evaluates benchmarking agent's capability to compare AWS Neuron and CUDA endpoints accurately.

Datasets

Shared data for this challenge

Review public datasets and any private uploads tied to your build.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

math_correctness

Verifies calculated cost savings percentage strictly matches ground truth ratio

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

analysis_accuracy

Accuracy of cost and throughput comparison calculations • target: 0.99 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Construct OpenAI Agents SDK function calls to manage AWS Neuron and vLLM benchmarking pipelines

  • Execute reproducible synthetic token generation benchmarks across target instance types

  • Collect and normalize latency metrics (TTFT and inter-token latency ITL)

  • Synthesize an auditable deployment recommendation report

Resources and assets

Reference links and supporting material

Dataset notes

Latency and throughput measurements recorded across AWS Inf2 and NVIDIA GPU instances.

How this agent runs

Evaluates benchmarking agent's capability to compare AWS Neuron and CUDA endpoints accurately.

Challenge input

JSON containing neuron_bench_data and vllm_bench_data with throughput, TTFT, and hourly_cost

OpenAI Agents SDK

Provides direct tool calling and execution loop orchestration.

Evaluated output

JSON returning optimal_target, cost_savings_pct, ttft_diff_ms, and trade_off_summary

Checks for
  • Verifies calculated cost savings percentage strictly matches ground truth ratio
  • Accuracy of cost and throughput comparison calculations • target: 0.99 • range: 0-1
Proof of success
  • Benchmark: Cloud-Inference-Cost-Bench
  • Analysis Accuracy target: 0.99
  • 1 public reference case
Runtime evidence
  • Python execution harness
  • Python sandbox (unavailable on Versalist)
View technical recipe

Configured tools

Action Space
  • OpenAI · Required
  • Anthropic · Optional
  • Azure OpenAI Service · Optional

Evaluation contract

  • math_correctness · Weight 1
  • analysis_accuracy · Weight 1

Recipe state

This is a preview. The configuration can change before the evaluation recipe is locked.

Frequently Asked Questions about AWS Neuron vs vLLM Serving Benchmark Harness with OpenAI Agents SDK