Build a Continuous Batching KV-Cache Memory Headroom Planner — implementation

## Build
Simulate memory fragmentation and KV-cache exhaustion in open-weight continuous batching systems. Formulate safe admission limits to prevent out-of-memory worker crashes.

## User
LLM Inference Systems Architect

## Workflow
1. Ingest prompt length distributions and max generation token requirements.
2. Model PagedAttention block allocation, prefix sharing, and fragmentation levels.
3. Call hosted Claude to evaluate headroom safety factors and propose admission caps.
4. Export an optimized vLLM serving configuration file with safe memory limits.

## Interface
Memory allocation simulation console showing dynamic KV-cache block usage and queue rejection triggers.

## Data
Construct synthetic traces of token sequence lengths and prompt sharing ratios modeled after typical chat and code generation workloads.
Starter examples below are synthetic. They are not downloaded records.

## Source
[vLLM open-model serving](https://raw.githubusercontent.com/vllm-project/vllm/main/README.md)
Directly implements vLLM PagedAttention block size and memory allocation formulas to simulate KV-cache utilization.

## Tools
pydantic-ai, anthropic-claude, ray-serve.
Validates boundary conditions on memory allocation parameters, block counts, and admission control thresholds.
Hosted Claude API for risk assessment and configuration file generation. Local execution applies only to open models.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Accurately models memory block exhaustion across synthetic prompt bursts.
- Identifies threshold where prefix caching saves >20% KV-cache allocations.
- Outputs validated command-line arguments for vLLM deployment.

## Starter Examples
```json
[
  {
    "input": {
      "gpu_vram_gb": 24,
      "model_weights_gb": 14.5,
      "block_size_tokens": 16,
      "workload": {
        "concurrency": 32,
        "avg_prompt_tokens": 800,
        "avg_gen_tokens": 200,
        "prefix_share_pct": 40
      }
    },
    "expected_output": {
      "kv_cache_usable_gb": 7.1,
      "max_supported_concurrent_seqs": 28,
      "block_fragmentation_risk": "low",
      "recommended_args": "--gpu-memory-utilization 0.90 --block-size 16 --max-num-seqs 28 --enable-prefix-caching",
      "safety_headroom_pct": 12.5
    }
  },
  {
    "input": {
      "gpu_vram_gb": 16,
      "model_weights_gb": 15,
      "block_size_tokens": 16,
      "workload": {
        "concurrency": 4,
        "avg_prompt_tokens": 1000,
        "avg_gen_tokens": 500,
        "prefix_share_pct": 0
      }
    },
    "expected_output": {
      "kv_cache_usable_gb": 0.2,
      "max_supported_concurrent_seqs": 0,
      "block_fragmentation_risk": "critical_oom",
      "recommended_args": "Deployment rejected: insufficient VRAM for KV-cache after model loading",
      "safety_headroom_pct": 0
    }
  }
]
```

## Safety
Simulation workbench only; does not adjust memory locks or kernel pointers on operational inference nodes.
Use mock external writes. Require human review before any real deployment.