Build the application

## Build
Simulate dynamic prompt arrival distributions across local open-weight model serving instances. Formulate continuous batching thresholds and quantify queue wait times under bursty traffic.

## User
Machine Learning Platform Engineer

## Workflow
1. Ingest synthetic request trace logs with arrival timestamps and token sequence lengths.
2. Simulate vLLM chunked prefill and continuous batching mechanics across worker nodes.
3. Call hosted Claude to analyze batch efficiency tradeoffs and suggest queue timeout settings.
4. Generate a schedule recommendation report detailing latency percentiles and token throughput.

## Interface
Interactive simulation workbench with trace file upload, throughput curve graphs, and scheduler tuning parameters.

## Data
Assemble synthetic CSV traces containing prompt token counts, requested output tokens, and timestamp arrivals based on open LLM trace patterns.
Starter examples below are synthetic. They are not downloaded records.

## Source
[vLLM open-model serving](https://raw.githubusercontent.com/vllm-project/vllm/main/README.md)
Uses vLLM continuous batching and chunked prefill specifications to model request scheduling behavior and token execution cycles.

## Tools
ray-serve, anthropic-claude, pydantic-ai.
Provides the distributed execution runtime for mocking inference queue workers and measuring synthetic throughput.
Hosted Claude API for analyzing latency distributions and tuning scheduling hyper-parameters. No local Claude weights.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Simulates arrival of at least 1,000 synthetic requests with continuous batching logic.
- Calculates simulated P50, P90, and P99 latency within verified bounds.
- Produces structured scheduler configuration recommendations validated by Pydantic.

## Starter Examples
```json
[
  {
    "input": {
      "arrival_distribution": "poisson_100_rps",
      "batch_config": {
        "max_num_seqs": 64,
        "max_num_batched_tokens": 2048
      },
      "requests": [
        {
          "req_id": "r-1",
          "prompt_len": 512,
          "expected_out": 128
        },
        {
          "req_id": "r-2",
          "prompt_len": 1024,
          "expected_out": 256
        }
      ]
    },
    "expected_output": {
      "simulated_batch_count": 1,
      "estimated_p90_ttft_ms": 45.2,
      "estimated_tpot_ms": 14.1,
      "scheduler_recommendation": "Maintain chunked prefill at 2048 to prevent decode phase head-of-line blocking"
    }
  },
  {
    "input": {
      "arrival_distribution": "burst_empty",
      "batch_config": {
        "max_num_seqs": 64,
        "max_num_batched_tokens": 2048
      },
      "requests": []
    },
    "expected_output": {
      "simulated_batch_count": 0,
      "estimated_p90_ttft_ms": 0,
      "estimated_tpot_ms": 0,
      "scheduler_recommendation": "No traffic detected; queue parameters unaffected"
    }
  }
]
```

## Safety
Strictly a simulation benchmark; does not deploy live models or bind network ingress ports.
Use mock external writes. Require human review before any real deployment.