Build a Distributed Interconnect Troubleshooting and Evidence Capture Hub
Analyze distributed training fabric error counters and degraded inter-node communication traces on local open clusters. Assemble verifiable link degradation evidence dossiers and hardware replacement tickets.
What you are building
The core problem, expected build, and operating context for this challenge.
Analyze distributed training fabric error counters and degraded inter-node communication traces on local open clusters. Assemble verifiable link degradation evidence dossiers and hardware replacement tickets.
Cluster Operations Systems Engineer
Ingest normalized interconnect telemetry capturing PCIe error rates, NVLink CRC drops, and socket IDs.
Isolate degraded physical links causing distributed all-reduce communication stragglers across open-weight runs.
Structure anomalous link telemetry into a standardized hardware incident record.
Invoke hosted Claude via Anthropic API to generate a datacenter hardware maintenance dispatch.
CLI and web portal showing node interconnect topology, link health matrices, and technician tickets.
Create synthetic normalized fabric telemetry files representing 8-node GPU cluster all-reduce test runs. Starter examples below are synthetic. They are not downloaded records.
NVIDIA DCGM exporter Reference DCGM exporter default counter mechanisms for PCIe error rates and interconnect status.
pydantic-ai, anthropic, pandas. pydantic-ai enforces strict structural validation on diagnostic findings before maintenance ticketing. Hosted Claude API for technical synthesis of hardware fault patterns and technician action plans.
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.
Detects single-bit and multi-bit CRC error spikes across specific node-to-node interconnect lanes.
Correlates link error spikes with collective all-reduce straggler latency deviations.
Generates a structured physical maintenance order specifying rack, node, and fabric port IDs.
```json [ { "input": { "node_id": "gpu-rack-04-node-02", "fabric_interface": "nvlink-3", "crc_error_count": 48201, "allreduce_latency_us": 14500, "baseline_latency_us": 850 }, "expected_output": { "fault_type": "physical_link_degradation", "severity": "CRITICAL", "isolated_component": "gpu-rack-04-node-02:nvlink-3", "action": "reseat_cable_or_replace_mezzanine" } }, { "input": { "node_id": "gpu-rack-01-node-07", "fabric_interface": "pcie-switch-0", "crc_error_count": 0, "allreduce_latency_us": 870, "baseline_latency_us": 850 }, "expected_output": { "fault_type": "nominal", "severity": "INFO", "isolated_component": "none", "action": "none" } } ] ```
Outputs diagnostic proposals only without issuing actual remote node reboot or fabric reset commands. Use mock external writes. Require human review before any real deployment.
Participation guide
Outputs diagnostic proposals only without issuing actual remote node reboot or fabric reset commands. Use mock external writes and human review.
How work is evaluated
Detects single-bit and multi-bit CRC error spikes across specific node-to-node interconnect lanes. Correlates link error spikes with collective all-reduce straggler latency deviations. Generates a structured physical maintenance order specifying rack, node, and fabric port IDs.
Runnable source, pinned dependencies, setup commands, working interface, example outputs, recorded demo, tests, and action logs.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
Complete workflow
Ingest normalized interconnect telemetry capturing PCIe error rates, NVLink CRC drops, and socket IDs. Isolate degraded physical links causing distributed all-reduce communication stragglers across open-weight runs. Structure anomalous link telemetry into a standardized hardware incident record. Invoke hosted Claude via Anthropic API to generate a datacenter hardware maintenance dispatch.
Missing or incorrect.
Partly complete, with documented gaps.
Complete, with reproducible evidence.
Correctness and evidence
Detects single-bit and multi-bit CRC error spikes across specific node-to-node interconnect lanes. Correlates link error spikes with collective all-reduce straggler latency deviations. Generates a structured physical maintenance order specifying rack, node, and fabric port IDs.
Missing or incorrect.
Partly complete, with documented gaps.
Complete, with reproducible evidence.
Review interface
CLI and web portal showing node interconnect topology, link health matrices, and technician tickets.
Missing or incorrect.
Partly complete, with documented gaps.
Complete, with reproducible evidence.
Safe execution
Outputs diagnostic proposals only without issuing actual remote node reboot or fabric reset commands.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Ingest normalized interconnect telemetry capturing PCIe error rates, NVLink CRC drops, and socket IDs.
Isolate degraded physical links causing distributed all-reduce communication stragglers across open-weight runs.
Structure anomalous link telemetry into a standardized hardware incident record.
Invoke hosted Claude via Anthropic API to generate a datacenter hardware maintenance dispatch.
Reference links and supporting material
Create synthetic normalized fabric telemetry files representing 8-node GPU cluster all-reduce test runs. Supplied starter examples are synthetic, not source extracts.
Challenge setup
Configuration summary for this challenge.
Challenge input
Input details are not configured.
Pydantic AI
Typed Python agent framework.
Evaluated output
No output checks are configured.
- Ingest normalized interconnect telemetry capturing PCIe error rates, NVLink CRC drops, and socket IDs. Isol...
- Detects single-bit and multi-bit CRC error spikes across specific node-to-node interconnect lanes. Correlat...
- CLI and web portal showing node interconnect topology, link health matrices, and technician tickets.
- 2 public reference cases
- 2 configured tools
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- Pydantic AI · Optional
- Pydantic AI · Optional
Evaluation contract
- Complete workflow · Weight 40
- Correctness and evidence · Weight 30
- Review interface · Weight 20
- Safe execution · Weight 10
Reference cases
- 2 public reference cases
Workflow tasks
No workflow tasks are configured.
Recipe freeze marker
No recipe freeze time is recorded.