Build a Speculative Decoding Draft Model Memory Balancer

Calculate VRAM footprint tradeoffs between target and draft open-weight models for speculative decoding deployments. Use hosted Claude to recommend draft model sizes that prevent KV-cache eviction.

Inference Infrastructure Engineer

Extract parameter sizes and context limits for proposed target and draft open-weight model pairings.

Calculate combined VRAM allocation required for dual-model weights and shared KV-cache spaces.

Invoke hosted Claude API to evaluate the theoretical speedup versus memory pressure tradeoffs.

Compile an infrastructure sizing brief recommending draft model acceptance thresholds and batch limits.

Web dashboard with interactive concurrency sliders and markdown capacity brief exporter.

Normalized architectural specs for open models (e.g. Llama-3-70B, Qwen-2.5) loaded from local JSON fixtures. Starter examples below are synthetic. They are not downloaded records.

vLLM open-model serving Uses vLLM speculative decoding memory mechanics and dual-model loading constraints.

anthropic, pydantic-ai, pandas, fastapi, pydantic. Provides nuanced technical explanation of memory fragmentation and token latency trade-offs. Hosted Claude API for capacity analysis; profiles open-weight models only.

Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

Calculates combined static weights footprint for target and draft pairs.

Identifies when draft model size exceeds safe KV-cache margins for a given target batch size.

Highlights latency tradeoffs when context window scales.

Exports structured JSON configuration suitable for vLLM speculative decoding launchers.

```json [ { "input": { "target_model": "llama-70b", "draft_model": "llama-8b", "target_vram_gb": 140, "draft_vram_gb": 16, "total_gpu_vram_gb": 160, "target_concurrency": 8 }, "expected_output": { "vram_available_for_kv_gb": 4, "status": "INSUFFICIENT_MEMORY", "recommendation": "REDUCE_CONCURRENCY", "claude_explanation": "Combined weights use 156GB, leaving only 4GB for KV-cache." } }, { "input": { "target_model": "llama-70b", "draft_model": "llama-1b", "target_vram_gb": 140, "draft_vram_gb": 2, "total_gpu_vram_gb": 160, "target_concurrency": 8 }, "expected_output": { "vram_available_for_kv_gb": 18, "status": "SAFE", "recommendation": "PROCEED", "claude_explanation": "Combined weights use 142GB, leaving 18GB for KV-cache." } } ] ```

Applies strictly to open-weight models. Does not infer or simulate proprietary Claude infrastructure. Use mock external writes. Require human review before any real deployment.