Build a Spot Accelerator Preemption and Checkpoint Recovery Simulator — implementation

## Build
Simulate spot preemption events and KV-cache recovery costs across heterogeneous open-model workers. Plan migration schedules and evaluate resumption overhead using hosted Claude.

## User
Distributed Training and Serving Architect

## Workflow
1. Model synthetic cloud accelerator spot termination notices with variable countdown windows.
2. Calculate uncommitted KV-cache flush times and checkpoint state transfer latencies across nodes.
3. Use hosted Claude API to synthesize a failover sequence that minimizes request dropped-token counts.
4. Produce an incident replay report detailing service degradation and cost recovery.

## Interface
Chaos-engineering console with preemption trigger buttons and timeline reconstruction.

## Data
Simulated cloud termination event streams and distributed state snapshot metrics in test JSON.
Starter examples below are synthetic. They are not downloaded records.

## Source
[AWS Neuron documentation](https://raw.githubusercontent.com/aws-neuron/aws-neuron-sdk/master/README.md)
Models AWS EC2 spot lifecycle behavior and accelerated instance preemption intervals.

## Tools
ray-serve, anthropic-claude, pydantic-ai.
Provides deployment abstractions for distributed inference actors and replica state migration.
Hosted Claude API for post-preemption recovery planning; strictly remote API.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Accurately simulates 120-second cloud spot preemption notice constraint.
- Quantifies time required to serialize and migrate open-model serving context.
- Formulates worker draining sequence that avoids hard connection drops for active users.
- Outputs complete JSON recovery audit trail without invoking live cloud APIs.

## Starter Examples
```json
[
  {
    "input": {
      "preempted_node": "spot-worker-inf2-09",
      "notice_window_sec": 120,
      "active_requests": 8,
      "kv_cache_state_mb": 420,
      "interconnect_egress_mbps": 100,
      "backup_target_node": "on-demand-worker-inf2-01"
    },
    "expected_output": {
      "transfer_duration_sec": 4.2,
      "migration_feasible": true,
      "action_sequence": [
        "Mark spot-worker-inf2-09 as unschedulable in router.",
        "Flush 420MB active KV context to on-demand-worker-inf2-01 (4.2s estimated).",
        "Drain in-flight generation requests with remaining 115.8s buffer.",
        "Deregister spot worker instance prior to hardware termination."
      ],
      "dropped_request_count": 0
    }
  },
  {
    "input": {
      "preempted_node": "spot-worker-g5-02",
      "notice_window_sec": 15,
      "active_requests": 40,
      "kv_cache_state_mb": 2400,
      "interconnect_egress_mbps": 50,
      "backup_target_node": "on-demand-worker-g5-01"
    },
    "expected_output": {
      "transfer_duration_sec": 48,
      "migration_feasible": false,
      "action_sequence": [
        "Insufficient time for full context migration (48s required vs 15s window).",
        "Abort KV-cache serialization to preserve network egress.",
        "Reroute 40 active requests to fallback queue with prompt re-prefill.",
        "Issue emergency worker disconnect signal."
      ],
      "dropped_request_count": 40
    }
  }
]
```

## Safety
Simulates preemption logic without executing cloud VM termination commands.
Use mock external writes. Require human review before any real deployment.