Build the application

## Build
Simulate client request preemption in open-weight continuous batching runtimes like vLLM. Hosted Claude analyzes prompt arrival bursts and calculates optimal rescheduling queues.

## User
AI Serving Systems Engineer

## Workflow
1. Simulate bursty incoming prompt arrival streams and sudden inference worker preemption signals.
2. Evaluate interrupted continuous batching states to measure evicted prompt prefixes and chunked prefills.
3. Send normalized batch execution metrics to hosted Claude to optimize queue re-insertion priorities.
4. Render a simulated scheduling queue highlighting predicted completion latency and service-level violations.

## Interface
Terminal dashboard showing active continuous batch slots, queue re-prioritizations, and latency gauges.

## Data
Generate synthetic JSON request logs containing arrival timestamps, prompt token counts, and simulated preemption events.
Starter examples below are synthetic. They are not downloaded records.

## Source
[vLLM open-model serving](https://raw.githubusercontent.com/vllm-project/vllm/main/README.md)
Adopts continuous batching, chunked prefill, and prefix caching principles from vLLM runtime documentation.

## Tools
dspy, anthropic, pydantic, pandas.
DSPy compiles robust prompt optimization pipelines for structured continuous batch queue scheduling decisions.
Hosted Claude API for prompt scheduling optimization and reasoning over request priorities.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Computes prefix caching token reuse for re-queued prompts without loss.
- Generates rescheduling order that prioritizes near-completion batches within simulated SLA boundaries.
- Detects request starvation for low-priority long-context jobs after three consecutive preemption cycles.

## Starter Examples
```json
[
  {
    "input": {
      "batch_capacity": 4,
      "interrupted_requests": [
        {
          "req_id": "req-101",
          "tokens_processed": 512,
          "tokens_remaining": 16,
          "sla_budget_ms": 150
        },
        {
          "req_id": "req-102",
          "tokens_processed": 64,
          "tokens_remaining": 400,
          "sla_budget_ms": 1200
        }
      ]
    },
    "expected_output": {
      "rescheduled_queue": [
        "req-101",
        "req-102"
      ],
      "prefix_cache_savings_tokens": 576,
      "sla_violation_projected": false
    }
  },
  {
    "input": {
      "batch_capacity": 1,
      "interrupted_requests": [
        {
          "req_id": "req-201",
          "tokens_processed": 4096,
          "tokens_remaining": 2048,
          "sla_budget_ms": 100
        }
      ]
    },
    "expected_output": {
      "rescheduled_queue": [
        "req-201"
      ],
      "prefix_cache_savings_tokens": 4096,
      "sla_violation_projected": true
    }
  }
]
```

## Safety
Simulation harness only. Operates purely on synthetic workload records without modifying production serving instances.
Use mock external writes. Require human review before any real deployment.