Build a Cloud Accelerator Preemption and Job Rescheduling Replayer — implementation

## Build
Simulate spot preemption events and reconstruct KV-cache checkpointing costs for open-weight serving nodes. Formulate dynamic job migration and re-computation schedules to preserve inference SLAs.

## User
Distributed Systems Reliability Engineer

## Workflow
1. Ingest simulated spot preemption notification timestamps across heterogeneous cloud worker instances.
2. Calculate KV-cache serialization state and transfer overhead to designated backup workers.
3. Replay request backlog to estimate latency penalties under varying re-computation strategies.
4. Invoke hosted Claude via Anthropic API to draft a preemption resiliency runbook.

## Interface
Interactive preemption chaos simulator UI with worker state diagrams and SLA recovery timelines.

## Data
Construct synthetic cluster state records modeling 10 spot GPU worker preemption events over a 24-hour cycle.
Starter examples below are synthetic. They are not downloaded records.

## Source
[vLLM open-model serving](https://raw.githubusercontent.com/vllm-project/vllm/main/README.md)
Reference vLLM distributed inference parallelism models and KV-cache state preservation concepts.

## Tools
ray-serve, anthropic, pydantic.
ray-serve models multi-worker replica failover topologies and request re-routing state.
Hosted Claude API for runbook generation and risk synthesis; simulation targets open-model serving logic.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Models the exact latency tradeoff between cold-restart re-prefill versus snapshot KV-cache loading.
- Identifies cluster failover capacity deficits that risk dropping client queries.
- Exports structured migration schedules that maintain service availability above 99 percent.

## Starter Examples
```json
[
  {
    "input": {
      "preempted_node": "worker-spot-08",
      "active_sessions": 24,
      "kv_cache_size_mb": 4200,
      "network_transfer_mbps": 1000,
      "target_node": "worker-spot-02",
      "target_node_available_ram_mb": 8000
    },
    "expected_output": {
      "migration_strategy": "transfer_cache_snapshot",
      "transfer_time_sec": 4.2,
      "sla_breach": false,
      "target_admissibility": "APPROVED"
    }
  },
  {
    "input": {
      "preempted_node": "worker-spot-03",
      "active_sessions": 48,
      "kv_cache_size_mb": 12000,
      "network_transfer_mbps": 500,
      "target_node": "worker-spot-04",
      "target_node_available_ram_mb": 4000
    },
    "expected_output": {
      "migration_strategy": "evict_and_cold_recompute",
      "transfer_time_sec": null,
      "sla_breach": true,
      "target_admissibility": "REJECTED_INSUFFICIENT_MEMORY"
    }
  }
]
```

## Safety
Simulates cluster fault tolerance and failover policies purely in software without terminating active cloud nodes.
Use mock external writes. Require human review before any real deployment.