Build a Local Open-Weight Continuous Batching Sizing Calculator

Evaluate memory headroom and PagedAttention KV-cache capacity across open-weight models on local servers. Synthesize safe concurrency limits and deployment guidance using hosted Claude.

Inference Operations Engineer

Ingest open-weight model parameter counts, context lengths, and physical GPU memory specifications.

Calculate baseline model weight footprints and PagedAttention KV-cache block allocations using standard formulas.

Query hosted Claude via the Anthropic Python SDK to explain memory exhaustion risks and suggest concurrency thresholds.

Generate a structured capacity sizing report detailing maximum concurrent requests before out-of-memory preemption.

CLI tool that accepts model memory parameters and emits an interactive console sizing report with simulated allocations.

Local synthetic JSON configurations defining open-weight model architectures (e.g., 7B and 70B parameters) and mock GPU hardware memory limits. Starter examples below are synthetic. They are not downloaded records.

vLLM open-model serving Applies vLLM PagedAttention KV-cache memory management concepts to compute per-token memory allocation limits for open weights.

dspy, anthropic. DSPy structures prompt signatures to ensure hosted Claude returns strictly typed capacity sizing rationales without schema drift. Hosted a currently available hosted Claude model accessed via API key; local execution handles open-weight parameter calculations only.

Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

Computes exact static parameter footprint and remaining memory for KV-cache blocks.

Invokes hosted Claude using Anthropic SDK to explain continuous batching trade-offs.

Emits structured capacity recommendations without simulating execution of proprietary weights.

```json [ { "input": { "model_name": "OpenLLaMA-7B", "parameter_count_billions": 7, "precision_bytes": 2, "gpu_vram_gb": 24, "context_len": 4096 }, "expected_output": { "model_weights_gb": 14, "available_kv_cache_gb": 10, "max_concurrent_sequences": 8, "concurrency_guidance": "Recommended maximum batch size is 8 to prevent KV-cache paging evictions under peak context lengths." } }, { "input": { "model_name": "OpenLLaMA-70B", "parameter_count_billions": 70, "precision_bytes": 2, "gpu_vram_gb": 24, "context_len": 2048 }, "expected_output": { "model_weights_gb": 140, "available_kv_cache_gb": 0, "max_concurrent_sequences": 0, "concurrency_guidance": "Model weights exceed single-device capacity. Tensor parallelism or weight quantization is strictly required before sizing." } } ] ```

Ensures purely simulated memory sizing without issuing live OS or cluster process controls. Use mock external writes. Require human review before any real deployment.