Skip to main content
Back to challenges
Challenge

Continuous Batching Memory Planner with Google ADK and LlamaIndex

High-throughput model serving requires dynamic memory allocation and continuous batching across TPU and GPU clusters. You will implement an agent using Google ADK integrated with LlamaIndex indexing to monitor page table fragmentation in KV caches and rebalance execution queues. Your deployment must achieve a 15% reduction in request eviction under burst traffic.

EngineeringHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Develop an automated memory planner for continuous batching LLM inference using Google ADK agents and LlamaIndex knowledge retrieval.

Delivery guide

How work is evaluated

Evaluation

Simulates continuous request traffic to evaluate queue admission decisions and KV cache memory utilization.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

no_memory_overflow

Ensures total allocated KV memory does not exceed total available VRAM capacity

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

eviction_rate

Proportion of requests dropped due to dynamic memory exhaustion • target: 0.02 • range: 0-0.15

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Build a Google ADK agent capable of dynamically controlling vLLM dynamic batch sizes and dynamic KV page allocation

  • Construct a LlamaIndex vector index of hardware-specific TPU and GPU memory performance profiles

  • Implement dynamic queue rebalancing rules for streaming agent requests

  • Measure and optimize p99 latency under continuous request influx

Resources and assets

Reference links and supporting material

Dataset notes

Traces of multi-tenant inference workloads with varying prompt lengths and context concurrency.

Challenge setup

Simulates continuous request traffic to evaluate queue admission decisions and KV cache memory utilization.

Challenge input

JSON containing hardware_spec (total_vram_gb, page_size_mb) and concurrent_requests array

Google ADK

Provides native integration with Vertex AI tools and agent workflows.

LlamaIndex

Allows quick retrieval of hardware memory footprints and capacity limits.

Evaluated output

JSON detailing scheduled_batch_size, admitted_request_ids, rejected_request_ids, and estimated_p99_ms

Evaluation checks
  • Ensures total allocated KV memory does not exceed total available VRAM capacity
  • Proportion of requests dropped due to dynamic memory exhaustion • target: 0.02 • range: 0-0.15
Configured evaluation evidence
  • Benchmark: PagedAttention-Stress-Bench
  • Eviction Rate target: 2%
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Llama Index · Required
  • LlamaIndex · Optional
  • Google · Optional

Evaluation contract

  • no_memory_overflow · Weight 1
  • eviction_rate · Weight 1

Metrics

  • eviction_rate · Target 0.02

Benchmarks

  • PagedAttention-Stress-Bench
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON containing hardware_spec (total_vram_gb, page_size_mb) and concurrent_requests array

    Output: JSON detailing scheduled_batch_size, admitted_request_ids, rejected_request_ids, and estimated_p99_ms

    Execution harness: Python

Evaluation tools

  • Google ADK · Agent Orchestrator
  • LlamaIndex · Knowledge Index

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Continuous Batching Memory Planner with Google ADK and LlamaIndex