Back to challenges
Challenge

Continuous Batching Memory Planner with Google ADK and LlamaIndex

High-throughput model serving requires dynamic memory allocation and continuous batching across TPU and GPU clusters. You will implement an agent using Google ADK integrated with LlamaIndex indexing to monitor page table fragmentation in KV caches and rebalance execution queues. Your deployment must achieve a 15% reduction in request eviction under burst traffic.

EngineeringHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Develop an automated memory planner for continuous batching LLM inference using Google ADK agents and LlamaIndex knowledge retrieval.

Delivery guide

How work is evaluated

Evaluation

Simulates continuous request traffic to evaluate queue admission decisions and KV cache memory utilization.

Datasets

Shared data for this challenge

Review public datasets and any private uploads tied to your build.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

no_memory_overflow

Ensures total allocated KV memory does not exceed total available VRAM capacity

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

eviction_rate

Proportion of requests dropped due to dynamic memory exhaustion • target: 0.02 • range: 0-0.15

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Build a Google ADK agent capable of dynamically controlling vLLM dynamic batch sizes and dynamic KV page allocation

  • Construct a LlamaIndex vector index of hardware-specific TPU and GPU memory performance profiles

  • Implement dynamic queue rebalancing rules for streaming agent requests

  • Measure and optimize p99 latency under continuous request influx

Resources and assets

Reference links and supporting material

Dataset notes

Traces of multi-tenant inference workloads with varying prompt lengths and context concurrency.

How this agent runs

Simulates continuous request traffic to evaluate queue admission decisions and KV cache memory utilization.

Challenge input

JSON containing hardware_spec (total_vram_gb, page_size_mb) and concurrent_requests array

Google ADK

Provides native integration with Vertex AI tools and agent workflows.

LlamaIndex

Allows quick retrieval of hardware memory footprints and capacity limits.

Evaluated output

JSON detailing scheduled_batch_size, admitted_request_ids, rejected_request_ids, and estimated_p99_ms

Checks for
  • Ensures total allocated KV memory does not exceed total available VRAM capacity
  • Proportion of requests dropped due to dynamic memory exhaustion • target: 0.02 • range: 0-0.15
Proof of success
  • Benchmark: PagedAttention-Stress-Bench
  • Eviction Rate target: 2%
  • 1 public reference case
Runtime evidence
  • Python execution harness
  • Python sandbox (unavailable on Versalist)
View technical recipe

Configured tools

Action Space
  • Llama Index · Required
  • LlamaIndex · Optional
  • Google · Optional

Evaluation contract

  • no_memory_overflow · Weight 1
  • eviction_rate · Weight 1

Recipe state

This is a preview. The configuration can change before the evaluation recipe is locked.

Frequently Asked Questions about Continuous Batching Memory Planner with Google ADK and LlamaIndex