Build the application

## Build
Evaluate memory headroom and PagedAttention KV-cache capacity across open-weight models on local servers. Synthesize safe concurrency limits and deployment guidance using hosted Claude.

## User
Inference Operations Engineer

## Workflow
1. Ingest open-weight model parameter counts, context lengths, and physical GPU memory specifications.
2. Calculate baseline model weight footprints and PagedAttention KV-cache block allocations using standard formulas.
3. Query hosted Claude via the Anthropic Python SDK to explain memory exhaustion risks and suggest concurrency thresholds.
4. Generate a structured capacity sizing report detailing maximum concurrent requests before out-of-memory preemption.

## Interface
CLI tool that accepts model memory parameters and emits an interactive console sizing report with simulated allocations.

## Data
Local synthetic JSON configurations defining open-weight model architectures (e.g., 7B and 70B parameters) and mock GPU hardware memory limits.
Starter examples below are synthetic. They are not downloaded records.

## Source
[vLLM open-model serving](https://raw.githubusercontent.com/vllm-project/vllm/main/README.md)
Applies vLLM PagedAttention KV-cache memory management concepts to compute per-token memory allocation limits for open weights.

## Tools
dspy, anthropic.
DSPy structures prompt signatures to ensure hosted Claude returns strictly typed capacity sizing rationales without schema drift.
Hosted a currently available hosted Claude model accessed via API key; local execution handles open-weight parameter calculations only.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Computes exact static parameter footprint and remaining memory for KV-cache blocks.
- Invokes hosted Claude using Anthropic SDK to explain continuous batching trade-offs.
- Emits structured capacity recommendations without simulating execution of proprietary weights.

## Starter Examples
```json
[
  {
    "input": {
      "model_name": "OpenLLaMA-7B",
      "parameter_count_billions": 7,
      "precision_bytes": 2,
      "gpu_vram_gb": 24,
      "context_len": 4096
    },
    "expected_output": {
      "model_weights_gb": 14,
      "available_kv_cache_gb": 10,
      "max_concurrent_sequences": 8,
      "concurrency_guidance": "Recommended maximum batch size is 8 to prevent KV-cache paging evictions under peak context lengths."
    }
  },
  {
    "input": {
      "model_name": "OpenLLaMA-70B",
      "parameter_count_billions": 70,
      "precision_bytes": 2,
      "gpu_vram_gb": 24,
      "context_len": 2048
    },
    "expected_output": {
      "model_weights_gb": 140,
      "available_kv_cache_gb": 0,
      "max_concurrent_sequences": 0,
      "concurrency_guidance": "Model weights exceed single-device capacity. Tensor parallelism or weight quantization is strictly required before sizing."
    }
  }
]
```

## Safety
Ensures purely simulated memory sizing without issuing live OS or cluster process controls.
Use mock external writes. Require human review before any real deployment.