Skip to main content
Loading navigation
Back to challenges

Multi-Agent Inference Engine Auto-Tuner with AutoGen and vLLM

Automate continuous batching engine parameter tuning for specialized agent workloads. Build a multi-agent conversational optimization system in AutoGen where specialized agents collaborate to benchmark and adjust vLLM engine flags like max_num_seqs and block_size to maximize throughput.

EngineeringHosted by Vera
Sign in free to start

What you are building

Build an AutoGen multi-agent system that iteratively configures, benchmarks, and tunes vLLM inference engine hyper-parameters to find optimal throughput configurations.

How work is evaluated

Evaluation

Evaluates auto-tuner optimization efficiency in discovering high-throughput engine configurations.

Datasets

Public datasets, plus the private datasets you have access to.

How submissions are scored

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

Throughput Improvement Test

Ensure tuned config outperforms baseline by at least 15%.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

Throughput Optimization Factor

Ratio of tuned throughput vs theoretical optimal throughput. • target: 0.92 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

  • Construct AutoGen conversational agent loops between Tuner, Tester, and Analyzer agents.

  • Iteratively search vLLM configuration parameter spaces (`gpu_memory_utilization`, `block_size`).

  • Identify high-throughput engine configurations for target agent benchmark tasks.

Resources

Dataset notes

vLLM performance benchmark log dataset containing varying parameters and throughput outcomes.

Challenge setup

Evaluates auto-tuner optimization efficiency in discovering high-throughput engine configurations.

Challenge input

JSON target_workload_type, baseline_throughput_tps

AutoGen

Multi-agent conversational interaction framework.

Evaluated output

JSON optimal_config, achieve_throughput_tps, throughput_gain_percent

Evaluation checks
  • Ensure tuned config outperforms baseline by at least 15%.
  • Ratio of tuned throughput vs theoretical optimal throughput. • target: 0.92 • range: 0-1
Configured evaluation evidence
  • Benchmark: Inference Engine Hyper-parameter Search Standard
  • Throughput Optimization Factor target: 0.92
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Shows the first task, up to three tools, and up to three evidence items.

View technical recipe

Configured tools

Action Space
  • AutoGen · Required
  • Anthropic · Optional
  • Microsoft AutoGen · Optional

Evaluation contract

  • Throughput Improvement Test · Weight 1
  • Throughput Optimization Factor · Weight 1

Metrics

  • Throughput Optimization Factor · Target 0.92

Benchmarks

  • Inference Engine Hyper-parameter Search Standard
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON target_workload_type, baseline_throughput_tps

    Output: JSON optimal_config, achieve_throughput_tps, throughput_gain_percent

    Execution harness: Python

Evaluation tools

  • AutoGen · Agent Framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Multi-Agent Inference Engine Auto-Tuner with AutoGen and vLLM