Build the application

## Build
Model PagedAttention KV-cache memory pressure across open-weight model instances under dynamic prompt loads. Formulate safe continuous batching limits and preemption mitigation strategies using hosted Claude.

## User
Local Inference Infrastructure Specialist

## Workflow
1. Extract active block allocations, KV-cache utilization, and preemption counts from open-weight vLLM serving engine logs.
2. Compute available KV memory headroom, fragmentation factors, and maximum concurrent token generation limits.
3. Query hosted Claude via Anthropic API to evaluate memory starvation risks and generate parameter optimization runbooks.
4. Output structured continuous batching configuration manifests with verified safe concurrency boundaries for cluster operators.

## Interface
Terminal dashboard and web control console showing block allocator fragmentation, KV-cache fill rates, and configuration diffs.

## Data
Simulate 35 sequential vLLM engine log snapshots showing KV block allocation tables, sequence lengths, and GPU memory states.
Starter examples below are synthetic. They are not downloaded records.

## Source
[vLLM open-model serving](https://raw.githubusercontent.com/vllm-project/vllm/main/README.md)
Extracts PagedAttention KV-cache memory concepts and continuous batching parameters directly from vLLM architecture documentation.

## Tools
ollama, pydantic.
Ollama provides local execution of open-weight benchmark models to generate baseline memory allocation traces without cloud dependencies.
Hosted Claude API for expert memory optimization advice; local open models run strictly on test rig.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Calculates KV-cache block utilization and memory fragmentation percentiles with mathematical precision.
- Identifies imminent sequence preemption events when KV headroom drops below safety margin.
- Emits validated YAML continuous batching parameters compatible with open-source serving runtimes.

## Starter Examples
```json
[
  {
    "input": {
      "open_model_name": "llama-3-8b-instruct",
      "gpu_vram_gb": 24,
      "weights_vram_gb": 16.2,
      "total_kv_blocks": 500,
      "allocated_kv_blocks": 475,
      "incoming_requests_per_sec": 30
    },
    "expected_output": {
      "kv_cache_utilization_pct": 95,
      "remaining_headroom_blocks": 25,
      "risk_level": "CRITICAL_STARVATION",
      "recommended_max_num_seqs": 16,
      "recommended_gpu_memory_utilization": 0.95
    }
  },
  {
    "input": {
      "open_model_name": "qwen-2.5-7b",
      "gpu_vram_gb": 40,
      "weights_vram_gb": 14,
      "total_kv_blocks": 1600,
      "allocated_kv_blocks": 400,
      "incoming_requests_per_sec": 12
    },
    "expected_output": {
      "kv_cache_utilization_pct": 25,
      "remaining_headroom_blocks": 1200,
      "risk_level": "NOMINAL",
      "recommended_max_num_seqs": 64,
      "recommended_gpu_memory_utilization": 0.9
    }
  }
]
```

## Safety
Applies strictly to locally managed open-weight models. Never attempts to infer memory structure of hosted Claude endpoints.
Use mock external writes. Require human review before any real deployment.