Build a Distributed Interconnect Troubleshooting and Evidence Capture Hub — implementation

## Build
Analyze distributed training fabric error counters and degraded inter-node communication traces on local open clusters. Assemble verifiable link degradation evidence dossiers and hardware replacement tickets.

## User
Cluster Operations Systems Engineer

## Workflow
1. Ingest normalized interconnect telemetry capturing PCIe error rates, NVLink CRC drops, and socket IDs.
2. Isolate degraded physical links causing distributed all-reduce communication stragglers across open-weight runs.
3. Structure anomalous link telemetry into a standardized hardware incident record.
4. Invoke hosted Claude via Anthropic API to generate a datacenter hardware maintenance dispatch.

## Interface
CLI and web portal showing node interconnect topology, link health matrices, and technician tickets.

## Data
Create synthetic normalized fabric telemetry files representing 8-node GPU cluster all-reduce test runs.
Starter examples below are synthetic. They are not downloaded records.

## Source
[NVIDIA DCGM exporter](https://raw.githubusercontent.com/NVIDIA/dcgm-exporter/main/README.md)
Reference DCGM exporter default counter mechanisms for PCIe error rates and interconnect status.

## Tools
pydantic-ai, anthropic, pandas.
pydantic-ai enforces strict structural validation on diagnostic findings before maintenance ticketing.
Hosted Claude API for technical synthesis of hardware fault patterns and technician action plans.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Detects single-bit and multi-bit CRC error spikes across specific node-to-node interconnect lanes.
- Correlates link error spikes with collective all-reduce straggler latency deviations.
- Generates a structured physical maintenance order specifying rack, node, and fabric port IDs.

## Starter Examples
```json
[
  {
    "input": {
      "node_id": "gpu-rack-04-node-02",
      "fabric_interface": "nvlink-3",
      "crc_error_count": 48201,
      "allreduce_latency_us": 14500,
      "baseline_latency_us": 850
    },
    "expected_output": {
      "fault_type": "physical_link_degradation",
      "severity": "CRITICAL",
      "isolated_component": "gpu-rack-04-node-02:nvlink-3",
      "action": "reseat_cable_or_replace_mezzanine"
    }
  },
  {
    "input": {
      "node_id": "gpu-rack-01-node-07",
      "fabric_interface": "pcie-switch-0",
      "crc_error_count": 0,
      "allreduce_latency_us": 870,
      "baseline_latency_us": 850
    },
    "expected_output": {
      "fault_type": "nominal",
      "severity": "INFO",
      "isolated_component": "none",
      "action": "none"
    }
  }
]
```

## Safety
Outputs diagnostic proposals only without issuing actual remote node reboot or fabric reset commands.
Use mock external writes. Require human review before any real deployment.