Build a Multi-GPU Interconnect Fabric Degradation Triage Engine — implementation

## Build
Parse normalized DCGM interconnect error counters to diagnose link degradation. Generate technician remediation itineraries and isolation steps using hosted Claude.

## User
Cluster Reliability Engineer

## Workflow
1. Ingest DCGM fabric and link error metrics, including replay counters and uncorrectable error registers.
2. Identify specific PCIe or NVLink lane degradations causing collective communication tail latency.
3. Prompt hosted Claude to interpret error bitmasks and generate precise physical cable reseating instructions.
4. Publish an interconnect triage manifest detailing faulted link endpoints and repair steps.

## Interface
Topology map CLI that highlights faulted interconnects and prints field repair instructions.

## Data
Synthetic node interconnect metrics with normalized DCGM error counters across multi-GPU topological meshes.
Starter examples below are synthetic. They are not downloaded records.

## Source
[NVIDIA DCGM exporter](https://raw.githubusercontent.com/NVIDIA/dcgm-exporter/main/README.md)
Employs DCGM exporter fabric error metric principles (NVLink and PCIe replay counters) to pinpoint link failures.

## Tools
crewai, anthropic.
CrewAI coordinates specialized software agent roles: a telemetry parser agent and a hardware repair synthesis agent.
Hosted a currently available hosted Claude model for fabric diagnostic reasoning; local system processes fabric counter logs.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Detects asymmetric link error distributions between GPU pairs.
- Isolates damaged lanes and physical socket positions.
- Generates hardware technician physical repair checklists via hosted Claude.

## Starter Examples
```json
[
  {
    "input": {
      "node_id": "compute-dgx-12",
      "gpu_source": 1,
      "gpu_target": 2,
      "link_type": "NVLink",
      "replay_errors_per_sec": 45000,
      "uncorrectable_errors": 2
    },
    "expected_output": {
      "link_status": "DEGRADED",
      "impact": "Severe bandwidth throttling on GPU 1-2 communications.",
      "remediation_action": "Schedule node maintenance, drain workloads, and reseat NVLink bridge connecting GPU 1 and GPU 2."
    }
  },
  {
    "input": {
      "node_id": "compute-dgx-04",
      "gpu_source": 0,
      "gpu_target": 1,
      "link_type": "NVLink",
      "replay_errors_per_sec": 0,
      "uncorrectable_errors": 0
    },
    "expected_output": {
      "link_status": "HEALTHY",
      "impact": "Nominal throughput across interconnect mesh.",
      "remediation_action": "No hardware intervention required."
    }
  }
]
```

## Safety
Read-only fabric counter analysis; does not issue live IPMI chassis commands or unbind driver devices.
Use mock external writes. Require human review before any real deployment.