Build a Workload Trace Simulator and Inference Batching Scheduler

Simulate dynamic prompt arrival distributions across local open-weight model serving instances. Formulate continuous batching thresholds and quantify queue wait times under bursty traffic.

Machine Learning Platform Engineer

Ingest synthetic request trace logs with arrival timestamps and token sequence lengths.

Simulate vLLM chunked prefill and continuous batching mechanics across worker nodes.

Call hosted Claude to analyze batch efficiency tradeoffs and suggest queue timeout settings.

Generate a schedule recommendation report detailing latency percentiles and token throughput.

Interactive simulation workbench with trace file upload, throughput curve graphs, and scheduler tuning parameters.

Assemble synthetic CSV traces containing prompt token counts, requested output tokens, and timestamp arrivals based on open LLM trace patterns. Starter examples below are synthetic. They are not downloaded records.

vLLM open-model serving Uses vLLM continuous batching and chunked prefill specifications to model request scheduling behavior and token execution cycles.

ray-serve, anthropic-claude, pydantic-ai. Provides the distributed execution runtime for mocking inference queue workers and measuring synthetic throughput. Hosted Claude API for analyzing latency distributions and tuning scheduling hyper-parameters. No local Claude weights.

Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

Simulates arrival of at least 1,000 synthetic requests with continuous batching logic.

Calculates simulated P50, P90, and P99 latency within verified bounds.

Produces structured scheduler configuration recommendations validated by Pydantic.

```json [ { "input": { "arrival_distribution": "poisson_100_rps", "batch_config": { "max_num_seqs": 64, "max_num_batched_tokens": 2048 }, "requests": [ { "req_id": "r-1", "prompt_len": 512, "expected_out": 128 }, { "req_id": "r-2", "prompt_len": 1024, "expected_out": 256 } ] }, "expected_output": { "simulated_batch_count": 1, "estimated_p90_ttft_ms": 45.2, "estimated_tpot_ms": 14.1, "scheduler_recommendation": "Maintain chunked prefill at 2048 to prevent decode phase head-of-line blocking" } }, { "input": { "arrival_distribution": "burst_empty", "batch_config": { "max_num_seqs": 64, "max_num_batched_tokens": 2048 }, "requests": [] }, "expected_output": { "simulated_batch_count": 0, "estimated_p90_ttft_ms": 0, "estimated_tpot_ms": 0, "scheduler_recommendation": "No traffic detected; queue parameters unaffected" } } ] ```

Strictly a simulation benchmark; does not deploy live models or bind network ingress ports. Use mock external writes. Require human review before any real deployment.