Build the application

## Build
Benchmark open-weight models locally under conversational workloads to measure vLLM prefix caching hit rates. Synthesize memory trade-offs and time-to-first-token reductions to optimize serving configurations using hosted Claude.

## User
Serving Optimization Engineer

## Workflow
1. Load synthetic trace logs containing multi-turn prompt lengths and vLLM prefix cache hit statistics.
2. Calculate effective time-to-first-token (TTFT) reductions and KV-cache block utilization memory overhead.
3. Invoke hosted Claude API to synthesize an architectural justification for the selected cache block size.
4. Output an optimal serving configuration YAML with verified prefix caching parameters.

## Interface
Streamlit benchmarking dashboard with TTFT reduction charts and config exporter.

## Data
Simplified synthetic JSON logs of multi-turn conversational evaluation runs comparing vLLM TTFT with and without prefix caching.
Starter examples below are synthetic. They are not downloaded records.

## Source
[vLLM open-model serving](https://raw.githubusercontent.com/vllm-project/vllm/main/README.md)
Implements vLLM prefix caching parameters and metrics.

## Tools
anthropic-claude, pydantic-ai, pandas.
Analyzes prefix caching memory trade-offs and TTFT improvements.
Hosted Claude API for analysis; local vLLM profiles for open-source models.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Accurately computes effective TTFT using weighted averages of cached and uncached latencies.
- Flags configurations where cache memory overhead is unjustified by the latency improvements.
- Outputs executable vLLM prefix caching argument configurations.
- Maintains clear separation between Claude evaluation synthesis and local vLLM parameters.

## Starter Examples
```json
[
  {
    "input": {
      "model_name": "open-llama-3-8b",
      "workload_type": "multi-turn-chat",
      "cache_hit_rate": 0.8,
      "baseline_ttft_ms": 400,
      "cached_ttft_ms": 100,
      "vram_overhead_gb": 12
    },
    "expected_output": {
      "effective_ttft_ms": 160,
      "net_speedup_ratio": 2.5,
      "viable_configuration": true,
      "vllm_config": {
        "enable_prefix_caching": true,
        "gpu_memory_utilization": 0.9
      },
      "claude_summary": "High 80% cache hit rate delivers a 2.5x TTFT speedup, justifying the 12GB VRAM allocation for conversational history."
    }
  },
  {
    "input": {
      "model_name": "open-llama-3-8b",
      "workload_type": "zero-shot-summarization",
      "cache_hit_rate": 0.1,
      "baseline_ttft_ms": 300,
      "cached_ttft_ms": 280,
      "vram_overhead_gb": 8
    },
    "expected_output": {
      "effective_ttft_ms": 298,
      "net_speedup_ratio": 1.01,
      "viable_configuration": false,
      "vllm_config": null,
      "claude_summary": "Low 10% hit rate yields a negligible 1.01x speedup; disable caching to reclaim 8GB VRAM."
    }
  }
]
```

## Safety
Simulates vLLM prefix caching memory trade-offs using synthetic trace data without exposing live production multi-turn chat records.
Use mock external writes. Require human review before any real deployment.