Build a Continuous Batching Prompt Interruption and Rescheduling Replayer

Simulate client request preemption in open-weight continuous batching runtimes like vLLM. Hosted Claude analyzes prompt arrival bursts and calculates optimal rescheduling queues.

AI Serving Systems Engineer

Simulate bursty incoming prompt arrival streams and sudden inference worker preemption signals.

Evaluate interrupted continuous batching states to measure evicted prompt prefixes and chunked prefills.

Send normalized batch execution metrics to hosted Claude to optimize queue re-insertion priorities.

Render a simulated scheduling queue highlighting predicted completion latency and service-level violations.

Terminal dashboard showing active continuous batch slots, queue re-prioritizations, and latency gauges.

Generate synthetic JSON request logs containing arrival timestamps, prompt token counts, and simulated preemption events. Starter examples below are synthetic. They are not downloaded records.

vLLM open-model serving Adopts continuous batching, chunked prefill, and prefix caching principles from vLLM runtime documentation.

dspy, anthropic, pydantic, pandas. DSPy compiles robust prompt optimization pipelines for structured continuous batch queue scheduling decisions. Hosted Claude API for prompt scheduling optimization and reasoning over request priorities.

Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

Computes prefix caching token reuse for re-queued prompts without loss.

Generates rescheduling order that prioritizes near-completion batches within simulated SLA boundaries.

Detects request starvation for low-priority long-context jobs after three consecutive preemption cycles.

```json [ { "input": { "batch_capacity": 4, "interrupted_requests": [ { "req_id": "req-101", "tokens_processed": 512, "tokens_remaining": 16, "sla_budget_ms": 150 }, { "req_id": "req-102", "tokens_processed": 64, "tokens_remaining": 400, "sla_budget_ms": 1200 } ] }, "expected_output": { "rescheduled_queue": [ "req-101", "req-102" ], "prefix_cache_savings_tokens": 576, "sla_violation_projected": false } }, { "input": { "batch_capacity": 1, "interrupted_requests": [ { "req_id": "req-201", "tokens_processed": 4096, "tokens_remaining": 2048, "sla_budget_ms": 100 } ] }, "expected_output": { "rescheduled_queue": [ "req-201" ], "prefix_cache_savings_tokens": 4096, "sla_violation_projected": true } } ] ```

Simulation harness only. Operates purely on synthetic workload records without modifying production serving instances. Use mock external writes. Require human review before any real deployment.