Build a Speculative Decoding Draft Model Memory Balancer — implementation

## Build
Calculate VRAM footprint tradeoffs between target and draft open-weight models for speculative decoding deployments. Use hosted Claude to recommend draft model sizes that prevent KV-cache eviction.

## User
Inference Infrastructure Engineer

## Workflow
1. Extract parameter sizes and context limits for proposed target and draft open-weight model pairings.
2. Calculate combined VRAM allocation required for dual-model weights and shared KV-cache spaces.
3. Invoke hosted Claude API to evaluate the theoretical speedup versus memory pressure tradeoffs.
4. Compile an infrastructure sizing brief recommending draft model acceptance thresholds and batch limits.

## Interface
Web dashboard with interactive concurrency sliders and markdown capacity brief exporter.

## Data
Normalized architectural specs for open models (e.g. Llama-3-70B, Qwen-2.5) loaded from local JSON fixtures.
Starter examples below are synthetic. They are not downloaded records.

## Source
[vLLM open-model serving](https://raw.githubusercontent.com/vllm-project/vllm/main/README.md)
Uses vLLM speculative decoding memory mechanics and dual-model loading constraints.

## Tools
anthropic, pydantic-ai, pandas, fastapi, pydantic.
Provides nuanced technical explanation of memory fragmentation and token latency trade-offs.
Hosted Claude API for capacity analysis; profiles open-weight models only.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Calculates combined static weights footprint for target and draft pairs.
- Identifies when draft model size exceeds safe KV-cache margins for a given target batch size.
- Highlights latency tradeoffs when context window scales.
- Exports structured JSON configuration suitable for vLLM speculative decoding launchers.

## Starter Examples
```json
[
  {
    "input": {
      "target_model": "llama-70b",
      "draft_model": "llama-8b",
      "target_vram_gb": 140,
      "draft_vram_gb": 16,
      "total_gpu_vram_gb": 160,
      "target_concurrency": 8
    },
    "expected_output": {
      "vram_available_for_kv_gb": 4,
      "status": "INSUFFICIENT_MEMORY",
      "recommendation": "REDUCE_CONCURRENCY",
      "claude_explanation": "Combined weights use 156GB, leaving only 4GB for KV-cache."
    }
  },
  {
    "input": {
      "target_model": "llama-70b",
      "draft_model": "llama-1b",
      "target_vram_gb": 140,
      "draft_vram_gb": 2,
      "total_gpu_vram_gb": 160,
      "target_concurrency": 8
    },
    "expected_output": {
      "vram_available_for_kv_gb": 18,
      "status": "SAFE",
      "recommendation": "PROCEED",
      "claude_explanation": "Combined weights use 142GB, leaving 18GB for KV-cache."
    }
  }
]
```

## Safety
Applies strictly to open-weight models. Does not infer or simulate proprietary Claude infrastructure.
Use mock external writes. Require human review before any real deployment.