Build a Thermal Throttling Incident Playbook and Workload Evacuation Planner
Monitor continuous temperature telemetry from DCGM metrics to detect localized hot spots. Plan progressive workload evacuation from overheating chassis to prevent ungraceful job failures.
What you are building
The core problem, expected build, and operating context for this challenge.
Monitor continuous temperature telemetry from DCGM metrics to detect localized hot spots. Plan progressive workload evacuation from overheating chassis to prevent ungraceful job failures.
HPC Systems Administrator
Process simulated DCGM thermal and power telemetry streams across rack enclosures.
Detect persistent thermal spikes exceeding safe operating margins.
Prompt hosted Claude to synthesize evacuation playbooks and rank recipient target nodes.
Construct an executable migration plan with container checkpointing checkpoints.
Terminal dashboard showing rack-level thermal contours and interactive workload evacuation action cards.
Generate synthetic multi-node DCGM metrics including GPU temperature, memory temperature, and power draw under load. Starter examples below are synthetic. They are not downloaded records.
NVIDIA DCGM exporter Extracts DCGM thermal metric structures and device status flags to identify hardware cooling failure modes.
anthropic-claude, pydantic-ai, mlflow. Reasons over complex thermal incident progressions and formulates prioritized workload draining instructions. Hosted Claude API for synthesizing evacuation playbooks and assessing cooling risk. Local models restricted to open weights.
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.
Detects thermal anomaly patterns 5 minutes prior to simulated critical thermal shutdown.
Generates evacuation plans ordering jobs by SLA criticality and checkpoint overhead.
Validates target node thermal headroom before assigning migrated workloads.
```json [ { "input": { "chassis_id": "chassis-08", "sensors": { "intake_temp_c": 34.5, "gpu0_temp_c": 88, "gpu1_temp_c": 89.5 }, "active_tasks": [ { "task_id": "train-job-4", "gpu_count": 2, "priority": "high", "checkpoint_ready": true } ] }, "expected_output": { "evacuation_required": true, "urgency": "immediate", "target_chassis": "chassis-02", "evacuation_sequence": [ "Pause task train-job-4", "Commit state checkpoint", "Drain chassis-08", "Resume on chassis-02" ], "operator_alert": "Suspected chassis intake airflow restriction" } }, { "input": { "chassis_id": "chassis-01", "sensors": { "intake_temp_c": 22, "gpu0_temp_c": 62, "gpu1_temp_c": 64 }, "active_tasks": [ { "task_id": "inference-prod", "gpu_count": 1, "priority": "critical", "checkpoint_ready": false } ] }, "expected_output": { "evacuation_required": false, "urgency": "nominal", "target_chassis": null, "evacuation_sequence": [], "operator_alert": "Normal thermal envelope" } } ] ```
Simulation and guidance only; does not send shutdown signals or adjust physical chiller controls. Use mock external writes. Require human review before any real deployment.
Participation guide
Simulation and guidance only; does not send shutdown signals or adjust physical chiller controls. Use mock external writes and human review.
How work is evaluated
Detects thermal anomaly patterns 5 minutes prior to simulated critical thermal shutdown. Generates evacuation plans ordering jobs by SLA criticality and checkpoint overhead. Validates target node thermal headroom before assigning migrated workloads.
Runnable source, pinned dependencies, setup commands, working interface, example outputs, recorded demo, tests, and action logs.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
Complete workflow
Process simulated DCGM thermal and power telemetry streams across rack enclosures. Detect persistent thermal spikes exceeding safe operating margins. Prompt hosted Claude to synthesize evacuation playbooks and rank recipient target nodes. Construct an executable migration plan with container checkpointing checkpoints.
Missing or incorrect.
Partly complete, with documented gaps.
Complete, with reproducible evidence.
Correctness and evidence
Detects thermal anomaly patterns 5 minutes prior to simulated critical thermal shutdown. Generates evacuation plans ordering jobs by SLA criticality and checkpoint overhead. Validates target node thermal headroom before assigning migrated workloads.
Missing or incorrect.
Partly complete, with documented gaps.
Complete, with reproducible evidence.
Review interface
Terminal dashboard showing rack-level thermal contours and interactive workload evacuation action cards.
Missing or incorrect.
Partly complete, with documented gaps.
Complete, with reproducible evidence.
Safe execution
Simulation and guidance only; does not send shutdown signals or adjust physical chiller controls.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Process simulated DCGM thermal and power telemetry streams across rack enclosures.
Detect persistent thermal spikes exceeding safe operating margins.
Prompt hosted Claude to synthesize evacuation playbooks and rank recipient target nodes.
Construct an executable migration plan with container checkpointing checkpoints.
Reference links and supporting material
Generate synthetic multi-node DCGM metrics including GPU temperature, memory temperature, and power draw under load. Supplied starter examples are synthetic, not source extracts.
Challenge setup
Configuration summary for this challenge.
Challenge input
Input details are not configured.
Anthropic Claude
LLM for code generation & more
Evaluated output
No output checks are configured.
- Process simulated DCGM thermal and power telemetry streams across rack enclosures. Detect persistent therma...
- Detects thermal anomaly patterns 5 minutes prior to simulated critical thermal shutdown. Generates evacuati...
- Terminal dashboard showing rack-level thermal contours and interactive workload evacuation action cards.
- 2 public reference cases
- 1 configured tool
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- Anthropic Claude · Optional
Evaluation contract
- Complete workflow · Weight 40
- Correctness and evidence · Weight 30
- Review interface · Weight 20
- Safe execution · Weight 10
Reference cases
- 2 public reference cases
Workflow tasks
No workflow tasks are configured.
Recipe freeze marker
No recipe freeze time is recorded.