Build the application

## Build
Simulate bursty prompt arrival distributions across open-weight continuous batching runtimes like vLLM. Formulate request admission boundaries and quantify queue wait times before deploying models.

## User
LLM Serving Performance Architect

## Workflow
1. Ingest synthetic trace records containing prompt token lengths and arrival timestamps.
2. Simulate vLLM PagedAttention continuous batching execution loops across fixed KV-cache blocks.
3. Quantify time-to-first-token spikes and queue wait times under simulated concurrency bursts.
4. Invoke hosted Claude API to synthesize serving parameter guidance including max batch size.

## Interface
Serving configuration simulator web UI with interactive latency distribution histograms.

## Data
Generate 1,000 synthetic request arrival timestamps with log-normal input and output token distributions.
Starter examples below are synthetic. They are not downloaded records.

## Source
[vLLM open-model serving](https://raw.githubusercontent.com/vllm-project/vllm/main/README.md)
Utilize vLLM principles of continuous batching, PagedAttention block allocation, and chunked prefill.

## Tools
optuna, anthropic, pydantic.
optuna evaluates parameter spaces for continuous batch size and KV-cache blocks to minimize latency.
Hosted Claude API for explaining batching trade-offs; simulation operates entirely on local open-weight mechanics.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Simulates token processing step-by-step using PagedAttention memory allocation logic.
- Identifies arrival rate thresholds that cause queue saturation and latency degradation.
- Outputs recommended max_num_seqs and max_num_batched_tokens serving parameters.

## Starter Examples
```json
[
  {
    "input": {
      "simulated_qps": 25,
      "mean_input_tokens": 1200,
      "mean_output_tokens": 150,
      "kv_cache_blocks_available": 1024,
      "block_size": 16
    },
    "expected_output": {
      "simulated_p99_ttft_ms": 1840,
      "cache_exhaustion_events": 12,
      "status": "QUEUE_SATURATED",
      "recommended_max_num_seqs": 32,
      "recommended_chunked_prefill": true
    }
  },
  {
    "input": {
      "simulated_qps": 5,
      "mean_input_tokens": 512,
      "mean_output_tokens": 100,
      "kv_cache_blocks_available": 2048,
      "block_size": 16
    },
    "expected_output": {
      "simulated_p99_ttft_ms": 210,
      "cache_exhaustion_events": 0,
      "status": "STABLE",
      "recommended_max_num_seqs": 64,
      "recommended_chunked_prefill": false
    }
  }
]
```

## Safety
Executes purely numerical simulations of batching queues without altering live inference production servers.
Use mock external writes. Require human review before any real deployment.