Build a Continuous Batching KV-Cache Memory Headroom Planner

Simulate memory fragmentation and KV-cache exhaustion in open-weight continuous batching systems. Formulate safe admission limits to prevent out-of-memory worker crashes.

LLM Inference Systems Architect

Ingest prompt length distributions and max generation token requirements.

Model PagedAttention block allocation, prefix sharing, and fragmentation levels.

Call hosted Claude to evaluate headroom safety factors and propose admission caps.

Export an optimized vLLM serving configuration file with safe memory limits.

Memory allocation simulation console showing dynamic KV-cache block usage and queue rejection triggers.

Construct synthetic traces of token sequence lengths and prompt sharing ratios modeled after typical chat and code generation workloads. Starter examples below are synthetic. They are not downloaded records.

vLLM open-model serving Directly implements vLLM PagedAttention block size and memory allocation formulas to simulate KV-cache utilization.

pydantic-ai, anthropic-claude, ray-serve. Validates boundary conditions on memory allocation parameters, block counts, and admission control thresholds. Hosted Claude API for risk assessment and configuration file generation. Local execution applies only to open models.

Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

Accurately models memory block exhaustion across synthetic prompt bursts.

Identifies threshold where prefix caching saves >20% KV-cache allocations.

Outputs validated command-line arguments for vLLM deployment.

```json [ { "input": { "gpu_vram_gb": 24, "model_weights_gb": 14.5, "block_size_tokens": 16, "workload": { "concurrency": 32, "avg_prompt_tokens": 800, "avg_gen_tokens": 200, "prefix_share_pct": 40 } }, "expected_output": { "kv_cache_usable_gb": 7.1, "max_supported_concurrent_seqs": 28, "block_fragmentation_risk": "low", "recommended_args": "--gpu-memory-utilization 0.90 --block-size 16 --max-num-seqs 28 --enable-prefix-caching", "safety_headroom_pct": 12.5 } }, { "input": { "gpu_vram_gb": 16, "model_weights_gb": 15, "block_size_tokens": 16, "workload": { "concurrency": 4, "avg_prompt_tokens": 1000, "avg_gen_tokens": 500, "prefix_share_pct": 0 } }, "expected_output": { "kv_cache_usable_gb": 0.2, "max_supported_concurrent_seqs": 0, "block_fragmentation_risk": "critical_oom", "recommended_args": "Deployment rejected: insufficient VRAM for KV-cache after model loading", "safety_headroom_pct": 0 } } ] ```

Simulation workbench only; does not adjust memory locks or kernel pointers on operational inference nodes. Use mock external writes. Require human review before any real deployment.