Build a Continuous Batching Workload Trace Replay and Sizing Tool

Simulate bursty prompt arrival distributions across open-weight continuous batching runtimes like vLLM. Formulate request admission boundaries and quantify queue wait times before deploying models.

LLM Serving Performance Architect

Ingest synthetic trace records containing prompt token lengths and arrival timestamps.

Simulate vLLM PagedAttention continuous batching execution loops across fixed KV-cache blocks.

Quantify time-to-first-token spikes and queue wait times under simulated concurrency bursts.

Invoke hosted Claude API to synthesize serving parameter guidance including max batch size.

Serving configuration simulator web UI with interactive latency distribution histograms.

Generate 1,000 synthetic request arrival timestamps with log-normal input and output token distributions. Starter examples below are synthetic. They are not downloaded records.

vLLM open-model serving Utilize vLLM principles of continuous batching, PagedAttention block allocation, and chunked prefill.

optuna, anthropic, pydantic. optuna evaluates parameter spaces for continuous batch size and KV-cache blocks to minimize latency. Hosted Claude API for explaining batching trade-offs; simulation operates entirely on local open-weight mechanics.

Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

Simulates token processing step-by-step using PagedAttention memory allocation logic.

Identifies arrival rate thresholds that cause queue saturation and latency degradation.

Outputs recommended max_num_seqs and max_num_batched_tokens serving parameters.

```json [ { "input": { "simulated_qps": 25, "mean_input_tokens": 1200, "mean_output_tokens": 150, "kv_cache_blocks_available": 1024, "block_size": 16 }, "expected_output": { "simulated_p99_ttft_ms": 1840, "cache_exhaustion_events": 12, "status": "QUEUE_SATURATED", "recommended_max_num_seqs": 32, "recommended_chunked_prefill": true } }, { "input": { "simulated_qps": 5, "mean_input_tokens": 512, "mean_output_tokens": 100, "kv_cache_blocks_available": 2048, "block_size": 16 }, "expected_output": { "simulated_p99_ttft_ms": 210, "cache_exhaustion_events": 0, "status": "STABLE", "recommended_max_num_seqs": 64, "recommended_chunked_prefill": false } } ] ```

Executes purely numerical simulations of batching queues without altering live inference production servers. Use mock external writes. Require human review before any real deployment.