Build a Thermal Throttling Incident Playbook and Workload Evacuation Planner — implementation

## Build
Monitor continuous temperature telemetry from DCGM metrics to detect localized hot spots. Plan progressive workload evacuation from overheating chassis to prevent ungraceful job failures.

## User
HPC Systems Administrator

## Workflow
1. Process simulated DCGM thermal and power telemetry streams across rack enclosures.
2. Detect persistent thermal spikes exceeding safe operating margins.
3. Prompt hosted Claude to synthesize evacuation playbooks and rank recipient target nodes.
4. Construct an executable migration plan with container checkpointing checkpoints.

## Interface
Terminal dashboard showing rack-level thermal contours and interactive workload evacuation action cards.

## Data
Generate synthetic multi-node DCGM metrics including GPU temperature, memory temperature, and power draw under load.
Starter examples below are synthetic. They are not downloaded records.

## Source
[NVIDIA DCGM exporter](https://raw.githubusercontent.com/NVIDIA/dcgm-exporter/main/README.md)
Extracts DCGM thermal metric structures and device status flags to identify hardware cooling failure modes.

## Tools
anthropic-claude, pydantic-ai, mlflow.
Reasons over complex thermal incident progressions and formulates prioritized workload draining instructions.
Hosted Claude API for synthesizing evacuation playbooks and assessing cooling risk. Local models restricted to open weights.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Detects thermal anomaly patterns 5 minutes prior to simulated critical thermal shutdown.
- Generates evacuation plans ordering jobs by SLA criticality and checkpoint overhead.
- Validates target node thermal headroom before assigning migrated workloads.

## Starter Examples
```json
[
  {
    "input": {
      "chassis_id": "chassis-08",
      "sensors": {
        "intake_temp_c": 34.5,
        "gpu0_temp_c": 88,
        "gpu1_temp_c": 89.5
      },
      "active_tasks": [
        {
          "task_id": "train-job-4",
          "gpu_count": 2,
          "priority": "high",
          "checkpoint_ready": true
        }
      ]
    },
    "expected_output": {
      "evacuation_required": true,
      "urgency": "immediate",
      "target_chassis": "chassis-02",
      "evacuation_sequence": [
        "Pause task train-job-4",
        "Commit state checkpoint",
        "Drain chassis-08",
        "Resume on chassis-02"
      ],
      "operator_alert": "Suspected chassis intake airflow restriction"
    }
  },
  {
    "input": {
      "chassis_id": "chassis-01",
      "sensors": {
        "intake_temp_c": 22,
        "gpu0_temp_c": 62,
        "gpu1_temp_c": 64
      },
      "active_tasks": [
        {
          "task_id": "inference-prod",
          "gpu_count": 1,
          "priority": "critical",
          "checkpoint_ready": false
        }
      ]
    },
    "expected_output": {
      "evacuation_required": false,
      "urgency": "nominal",
      "target_chassis": null,
      "evacuation_sequence": [],
      "operator_alert": "Normal thermal envelope"
    }
  }
]
```

## Safety
Simulation and guidance only; does not send shutdown signals or adjust physical chiller controls.
Use mock external writes. Require human review before any real deployment.