Build an Accelerator Interconnect Diagnosis and Packet Loss Reviewer — implementation

## Build
Analyze multi-GPU node interconnect counters and fabric error flags. Isolate degraded PCIe and NVLink links causing distributed all-reduce slowdowns and propose hardware repair actions.

## User
Cluster Network Engineer

## Workflow
1. Ingest normalized interconnect counter logs, CRC error rates, and replay counts.
2. Correlate interconnect packet drops with collective all-reduce latency degradation.
3. Prompt hosted Claude to inspect fabric topology anomalies and pinpoint failing lanes.
4. Synthesize a hardware maintenance ticket with port isolation and replacement guidance.

## Interface
Network topology graph visualizer highlighting degraded links with click-to-generate repair tickets.

## Data
Construct synthetic interconnect telemetry tables recording link CRC errors, replay events, and NVLink/PCIe port states across 8-GPU nodes.
Starter examples below are synthetic. They are not downloaded records.

## Source
[NVIDIA DCGM exporter](https://raw.githubusercontent.com/NVIDIA/dcgm-exporter/main/README.md)
Extracts DCGM NVLink error metrics, replay counters, and bandwidth saturation indicators from the DCGM exporter counters.

## Tools
anthropic-claude, pydantic-ai.
Interprets multi-hop interconnect error patterns and drafts clear, actionable physical maintenance steps.
Hosted Claude API for topological reasoning and maintenance ticket drafting. No access to proprietary cloud network routing.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Identifies failing interconnect links based on replay threshold exceedance (>1000/hr).
- Correlates link errors with distributed collective step latency degradation.
- Outputs structured repair tickets specifying switch port, GPU index, and replacement part.

## Starter Examples
```json
[
  {
    "input": {
      "node_id": "hpc-gpu-node-12",
      "fabric_errors": [
        {
          "gpu_src": 2,
          "gpu_dst": 3,
          "link_id": "nvlink-1",
          "crc_errors": 45120,
          "replay_count": 8900
        },
        {
          "gpu_src": 0,
          "gpu_dst": 1,
          "link_id": "nvlink-0",
          "crc_errors": 0,
          "replay_count": 0
        }
      ],
      "step_time_ms": 340.5,
      "baseline_step_time_ms": 180
    },
    "expected_output": {
      "fault_detected": true,
      "isolated_component": "GPU 2 -> GPU 3 NVLink 1",
      "degradation_pct": 89.2,
      "repair_action": "Reseat high-speed mezzanine interconnect or replace interposer ribbon cable on slot 2-3",
      "ticket_priority": "urgent"
    }
  },
  {
    "input": {
      "node_id": "hpc-gpu-node-03",
      "fabric_errors": [
        {
          "gpu_src": 0,
          "gpu_dst": 1,
          "link_id": "nvlink-0",
          "crc_errors": 2,
          "replay_count": 1
        }
      ],
      "step_time_ms": 181.2,
      "baseline_step_time_ms": 180
    },
    "expected_output": {
      "fault_detected": false,
      "isolated_component": "None",
      "degradation_pct": 0.6,
      "repair_action": "Maintain routine monitoring; error rate within allowable transient threshold",
      "ticket_priority": "low"
    }
  }
]
```

## Safety
Diagnostic analysis only; does not reset physical PCIe registers or disable cluster switch fabrics.
Use mock external writes. Require human review before any real deployment.