Build an Inference Token Recomputation and Migration Cost Estimator

Simulate compute preemption across inference clusters using vLLM PagedAttention state. Compare token recomputation overhead against KV-cache network migration costs using pandas, and use Anthropic Claude to optimize fallback strategies.

Inference Infrastructure Reliability Engineer

Ingest synthetic JSON traces of inference node interruptions with PagedAttention prefix cache offsets and remaining token counts.

Calculate the token recomputation overhead versus the theoretical network transfer cost of migrating the KV-cache snapshot using pandas.

Send comparative migration vs. recomputation profiles to hosted Claude to generate an optimized fallback strategy.

Render a decision matrix routing each interrupted job to either cold-start recomputation or stateful migration.

CLI simulation harness displaying job recovery progress and tabular JSON job handoff manifests.

Generate synthetic JSON event logs containing batch identifiers, token lengths, checkpoint timestamps, and simulated network bandwidth capabilities. Starter examples below are synthetic. They are not downloaded records.

vLLM open-model serving Models PagedAttention key-value memory management and continuous batching recovery concepts from vLLM documentation.

ray-serve, anthropic, pydantic, pandas. Ray Serve provides flexible actor abstractions for simulating distributed inference replica preemption and recovery routing. Hosted Claude API for evaluating token recomputation penalty tradeoffs versus network migration costs; no local model weights.

Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

Calculates total recomputation tokens across all interrupted jobs with zero negative values.

Claude generates a fallback strategy assigning all valid pending jobs to either MIGRATE_STATE or COLD_RESTART.

Correctly identifies scenarios where network bandwidth makes cold restarts more efficient than KV-cache transfer.

```json [ { "input": { "record_type": "simplified synthetic app record", "interrupted_node": "worker-04", "preemption_timestamp": "2025-05-10T14:22:00Z", "active_jobs": [ { "job_id": "job-801", "prompt_tokens": 1024, "generated_tokens": 128, "kv_cache_checkpointed": true }, { "job_id": "job-802", "prompt_tokens": 2048, "generated_tokens": 64, "kv_cache_checkpointed": false } ], "network_bandwidth_gbps": 10 }, "expected_output": { "total_recompute_tokens": 2112, "scheduled_migrations": [ { "job_id": "job-801", "strategy": "MIGRATE_STATE", "recompute_required": false }, { "job_id": "job-802", "strategy": "COLD_RESTART", "recompute_required": true } ], "capacity_status": "SUFFICIENT" } }, { "input": { "record_type": "simplified synthetic app record", "interrupted_node": "worker-08", "preemption_timestamp": "2025-05-10T15:00:00Z", "active_jobs": [ { "job_id": "job-990", "prompt_tokens": 8192, "generated_tokens": 512, "kv_cache_checkpointed": true } ], "network_bandwidth_gbps": 1 }, "expected_output": { "total_recompute_tokens": 8704, "scheduled_migrations": [ { "job_id": "job-990", "strategy": "COLD_RESTART", "recompute_required": true } ], "capacity_status": "SUFFICIENT" } } ] ```

Simulation harness only. No live infrastructure commands are executed and no internal Claude weights or infra are accessed. Use mock external writes. Require human review before any real deployment.