Build the application

## Build
Simulate compute preemption across inference clusters using vLLM PagedAttention state. Compare token recomputation overhead against KV-cache network migration costs using pandas, and use Anthropic Claude to optimize fallback strategies.

## User
Inference Infrastructure Reliability Engineer

## Workflow
1. Ingest synthetic JSON traces of inference node interruptions with PagedAttention prefix cache offsets and remaining token counts.
2. Calculate the token recomputation overhead versus the theoretical network transfer cost of migrating the KV-cache snapshot using pandas.
3. Send comparative migration vs. recomputation profiles to hosted Claude to generate an optimized fallback strategy.
4. Render a decision matrix routing each interrupted job to either cold-start recomputation or stateful migration.

## Interface
CLI simulation harness displaying job recovery progress and tabular JSON job handoff manifests.

## Data
Generate synthetic JSON event logs containing batch identifiers, token lengths, checkpoint timestamps, and simulated network bandwidth capabilities.
Starter examples below are synthetic. They are not downloaded records.

## Source
[vLLM open-model serving](https://raw.githubusercontent.com/vllm-project/vllm/main/README.md)
Models PagedAttention key-value memory management and continuous batching recovery concepts from vLLM documentation.

## Tools
ray-serve, anthropic, pydantic, pandas.
Ray Serve provides flexible actor abstractions for simulating distributed inference replica preemption and recovery routing.
Hosted Claude API for evaluating token recomputation penalty tradeoffs versus network migration costs; no local model weights.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Calculates total recomputation tokens across all interrupted jobs with zero negative values.
- Claude generates a fallback strategy assigning all valid pending jobs to either MIGRATE_STATE or COLD_RESTART.
- Correctly identifies scenarios where network bandwidth makes cold restarts more efficient than KV-cache transfer.

## Starter Examples
```json
[
  {
    "input": {
      "record_type": "simplified synthetic app record",
      "interrupted_node": "worker-04",
      "preemption_timestamp": "2025-05-10T14:22:00Z",
      "active_jobs": [
        {
          "job_id": "job-801",
          "prompt_tokens": 1024,
          "generated_tokens": 128,
          "kv_cache_checkpointed": true
        },
        {
          "job_id": "job-802",
          "prompt_tokens": 2048,
          "generated_tokens": 64,
          "kv_cache_checkpointed": false
        }
      ],
      "network_bandwidth_gbps": 10
    },
    "expected_output": {
      "total_recompute_tokens": 2112,
      "scheduled_migrations": [
        {
          "job_id": "job-801",
          "strategy": "MIGRATE_STATE",
          "recompute_required": false
        },
        {
          "job_id": "job-802",
          "strategy": "COLD_RESTART",
          "recompute_required": true
        }
      ],
      "capacity_status": "SUFFICIENT"
    }
  },
  {
    "input": {
      "record_type": "simplified synthetic app record",
      "interrupted_node": "worker-08",
      "preemption_timestamp": "2025-05-10T15:00:00Z",
      "active_jobs": [
        {
          "job_id": "job-990",
          "prompt_tokens": 8192,
          "generated_tokens": 512,
          "kv_cache_checkpointed": true
        }
      ],
      "network_bandwidth_gbps": 1
    },
    "expected_output": {
      "total_recompute_tokens": 8704,
      "scheduled_migrations": [
        {
          "job_id": "job-990",
          "strategy": "COLD_RESTART",
          "recompute_required": true
        }
      ],
      "capacity_status": "SUFFICIENT"
    }
  }
]
```

## Safety
Simulation harness only. No live infrastructure commands are executed and no internal Claude weights or infra are accessed.
Use mock external writes. Require human review before any real deployment.