Build a Thermal Throttling Incident Timeline and Operator Rebalancing Engine — implementation

## Build
Analyze time-series DCGM thermal logs during server throttling events. Hosted Claude reconstructs event sequences and drafts operator chassis airflow mitigation recommendations.

## User
Datacenter Operations Engineer

## Workflow
1. Parse normalized DCGM time-series telemetry records including GPU temperatures, power draw, and throttle flags.
2. Reconstruct the chronological timeline of temperature breaches and clock frequency reductions across affected chassis slots.
3. Send chronological telemetry logs to hosted Claude to isolate thermal runaways and formulate remediation steps.
4. Produce a structured operator inspection runbook containing fan curves and workload throttling mitigations.

## Interface
Web dashboard showing thermal excursion graphs and human operator sign-off panels.

## Data
Construct synthetic DCGM CSV files recording timestamps, device index, temperature, power draw, and throttling duration.
Starter examples below are synthetic. They are not downloaded records.

## Source
[NVIDIA DCGM exporter](https://raw.githubusercontent.com/NVIDIA/dcgm-exporter/main/README.md)
Consumes standardized DCGM exporter metric types such as GPU temperature and throttling reason indicators.

## Tools
mlflow, anthropic, pandas, pydantic.
MLflow tracks thermal event runs, metric parameters, and operational remediation artifacts across historical incidents.
Hosted Claude API for narrative incident root-cause reconstruction and mitigation generation.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Correctly identifies the exact primary throttle timestamp and initial trigger temperature.
- Flags secondary cascading device throttling events occurring within 120 seconds of the primary trigger.
- Produces actionable mitigation runbook with rack slot locations verified by human sign-off.

## Starter Examples
```json
[
  {
    "input": {
      "chassis_id": "rack-4-blade-02",
      "telemetry_stream": [
        {
          "timestamp": "2025-06-01T10:00:00Z",
          "gpu_idx": 0,
          "temp_c": 74,
          "throttle_active": false,
          "clock_mhz": 1800
        },
        {
          "timestamp": "2025-06-01T10:01:00Z",
          "gpu_idx": 0,
          "temp_c": 86,
          "throttle_active": true,
          "clock_mhz": 1150
        },
        {
          "timestamp": "2025-06-01T10:01:30Z",
          "gpu_idx": 1,
          "temp_c": 84,
          "throttle_active": true,
          "clock_mhz": 1200
        }
      ]
    },
    "expected_output": {
      "incident_detected": true,
      "primary_trigger": {
        "timestamp": "2025-06-01T10:01:00Z",
        "gpu_idx": 0,
        "temp_c": 86
      },
      "cascading_count": 1,
      "recommended_action": "INSPECT_BLADE_FAN_BAFFLE"
    }
  },
  {
    "input": {
      "chassis_id": "rack-4-blade-05",
      "telemetry_stream": [
        {
          "timestamp": "2025-06-01T11:00:00Z",
          "gpu_idx": 0,
          "temp_c": 68,
          "throttle_active": false,
          "clock_mhz": 1800
        },
        {
          "timestamp": "2025-06-01T11:01:00Z",
          "gpu_idx": 0,
          "temp_c": 70,
          "throttle_active": false,
          "clock_mhz": 1800
        }
      ]
    },
    "expected_output": {
      "incident_detected": false,
      "primary_trigger": null,
      "cascading_count": 0,
      "recommended_action": "MAINTAIN_CURRENT_COOLING"
    }
  }
]
```

## Safety
Read-only simulated diagnostics. Does not adjust physical fan controllers or alter operational firmware settings.
Use mock external writes. Require human review before any real deployment.