Build an Accelerator Chassis Thermal Incident Root-Cause Diagnoser — implementation

## Build
Analyze multi-sensor thermal logs during server shutdown events to reconstruct thermal runaway sequences. Generate hardware engineer post-mortem summaries and facility air-balance tasks using hosted Claude.

## User
Datacenter Hardware Reliability Specialist

## Workflow
1. Ingest high-frequency sensor streams tracking intake temperature, exhaust delta, and per-card junction temperatures.
2. Detect abnormal temperature gradient rates and identify the initial failing component.
3. Prompt hosted Claude to synthesize a formal hardware failure post-mortem and inspection steps.
4. Package diagnostic conclusions into a review dossier for facility HVAC and technician teams.

## Interface
Interactive timeline viewer displaying thermal gradient curves, component failure markers, and an audit report preview.

## Data
Create synthetic CSV logs simulating an 8-GPU chassis thermal runaway scenario with exhaust and junction metrics.
Starter examples below are synthetic. They are not downloaded records.

## Source
[NVIDIA DCGM exporter](https://raw.githubusercontent.com/NVIDIA/dcgm-exporter/main/README.md)
Adopts DCGM junction temperature and memory temperature counters to trace localized thermal spikes.

## Tools
anthropic, pydantic-ai, pandas.
anthropic-claude excels at correlating disparate multi-sensor anomalies into concise post-mortem findings.
Hosted Claude API for structured report compilation; pandas locally for metric gradients.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Pinpoint the exact timestamp and GPU index of the initial thermal trip.
- Produce actionable inspection recommendations targeting physical heatsinks and fans.
- Ensure all output dossiers clearly delineate confirmed observations from suggested root causes.

## Starter Examples
```json
[
  {
    "input": {
      "chassis_id": "rack-12-u3",
      "incident_time": "2025-02-10T14:30:00Z",
      "sensor_readings": [
        {
          "gpu_id": 0,
          "junction_temp": 81,
          "memory_temp": 75
        },
        {
          "gpu_id": 1,
          "junction_temp": 105,
          "memory_temp": 99
        },
        {
          "gpu_id": 2,
          "junction_temp": 82,
          "memory_temp": 76
        }
      ]
    },
    "expected_output": {
      "primary_fault_device": "GPU-1",
      "critical_threshold_exceeded": true,
      "hypothesized_cause": "Localized thermal paste degradation or liquid cooling plate blockage on GPU-1",
      "immediate_task": "Reseat GPU-1 cold plate and verify thermal interface material integrity."
    }
  },
  {
    "input": {
      "chassis_id": "rack-12-u4",
      "incident_time": "2025-02-10T15:00:00Z",
      "sensor_readings": [
        {
          "gpu_id": 0,
          "junction_temp": 70,
          "memory_temp": 68
        },
        {
          "gpu_id": 1,
          "junction_temp": 71,
          "memory_temp": 69
        }
      ]
    },
    "expected_output": {
      "primary_fault_device": null,
      "critical_threshold_exceeded": false,
      "hypothesized_cause": "Nominal operations, no thermal runaway detected",
      "immediate_task": "Continue standard telemetry monitoring."
    }
  }
]
```

## Safety
Read-only telemetry analysis; does not interact with datacenter control loops or physical power switches.
Use mock external writes. Require human review before any real deployment.