Build a Local Open-Weight Prefix Caching Efficiency Analyzer

Benchmark open-weight models locally under conversational workloads to measure vLLM prefix caching hit rates. Synthesize memory trade-offs and time-to-first-token reductions to optimize serving configurations using hosted Claude.

Serving Optimization Engineer

Load synthetic trace logs containing multi-turn prompt lengths and vLLM prefix cache hit statistics.

Calculate effective time-to-first-token (TTFT) reductions and KV-cache block utilization memory overhead.

Invoke hosted Claude API to synthesize an architectural justification for the selected cache block size.

Output an optimal serving configuration YAML with verified prefix caching parameters.

Streamlit benchmarking dashboard with TTFT reduction charts and config exporter.

Simplified synthetic JSON logs of multi-turn conversational evaluation runs comparing vLLM TTFT with and without prefix caching. Starter examples below are synthetic. They are not downloaded records.

vLLM open-model serving Implements vLLM prefix caching parameters and metrics.

anthropic-claude, pydantic-ai, pandas. Analyzes prefix caching memory trade-offs and TTFT improvements. Hosted Claude API for analysis; local vLLM profiles for open-source models.

Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

Accurately computes effective TTFT using weighted averages of cached and uncached latencies.

Flags configurations where cache memory overhead is unjustified by the latency improvements.

Outputs executable vLLM prefix caching argument configurations.

Maintains clear separation between Claude evaluation synthesis and local vLLM parameters.

```json [ { "input": { "model_name": "open-llama-3-8b", "workload_type": "multi-turn-chat", "cache_hit_rate": 0.8, "baseline_ttft_ms": 400, "cached_ttft_ms": 100, "vram_overhead_gb": 12 }, "expected_output": { "effective_ttft_ms": 160, "net_speedup_ratio": 2.5, "viable_configuration": true, "vllm_config": { "enable_prefix_caching": true, "gpu_memory_utilization": 0.9 }, "claude_summary": "High 80% cache hit rate delivers a 2.5x TTFT speedup, justifying the 12GB VRAM allocation for conversational history." } }, { "input": { "model_name": "open-llama-3-8b", "workload_type": "zero-shot-summarization", "cache_hit_rate": 0.1, "baseline_ttft_ms": 300, "cached_ttft_ms": 280, "vram_overhead_gb": 8 }, "expected_output": { "effective_ttft_ms": 298, "net_speedup_ratio": 1.01, "viable_configuration": false, "vllm_config": null, "claude_summary": "Low 10% hit rate yields a negligible 1.01x speedup; disable caching to reclaim 8GB VRAM." } } ] ```

Simulates vLLM prefix caching memory trade-offs using synthetic trace data without exposing live production multi-turn chat records. Use mock external writes. Require human review before any real deployment.