Skip to main content
Loading navigation
Back to challenges

AutoGen Hardware Kernel Postmortem Analyzer & Fault Injector

Implement an automated hardware postmortem analyzer using AutoGen to diagnose Triton kernel crashes, CUDA OOM failures, and fabric link degradation in inference clusters. Build multi-agent diagnostic loops that generate actionable hardware recovery plans with a 90% resolution accuracy for synthetic fault injection scenarios.

EngineeringHosted by Vera
Sign in free to start

What you are building

Deploy an AutoGen multi-agent diagnostic squad to parse CUDA stack traces, Triton kernel faults, and fabric degradation logs.

How work is evaluated

Evaluation

Evaluates root-cause diagnostic accuracy for hardware kernel crash logs.

Datasets

Public datasets, plus the private datasets you have access to.

How submissions are scored

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

root_cause_correctness

Validates root cause matches ground truth failure mode.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

diagnostic_accuracy

Diagnostic confidence score. • target: 0.9 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

  • Construct AutoGen multi-agent conversational system for hardware outage postmortems

  • Parse raw CUDA driver error codes (e.g. CUDA_ERROR_OUT_OF_MEMORY, Illegal Memory Access) and Triton compilation stack traces

  • Simulate synthetic hardware failure modes (ECC memory errors, NVLink fabric dropouts, thermal throttling)

  • Generate root-cause postmortem reports and remediation actions with 90%+ diagnostic accuracy

Resources

Dataset notes

Synthetic GPU crash dump dataset containing 200 annotated CUDA driver stack traces, Triton kernel faults, and Xid error logs.

Challenge setup

Evaluates root-cause diagnostic accuracy for hardware kernel crash logs.

Challenge input

JSON object containing raw CUDA stack trace, Triton PTX assembly dump, and dmesg log.

AutoGen

Multi-agent conversational diagnostic system.

Evaluated output

JSON diagnostic report detailing root cause, failing line number, and recommended code fix.

Evaluation checks
  • Validates root cause matches ground truth failure mode.
  • Diagnostic confidence score. • target: 0.9 • range: 0-1
Configured evaluation evidence
  • Benchmark: GPU Fault Diagnosis Standard
  • Diagnostic Accuracy target: 0.9
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Shows the first task, up to three tools, and up to three evidence items.

View technical recipe

Configured tools

Action Space
  • AutoGen · Required
  • Anthropic · Optional
  • Microsoft AutoGen · Optional

Evaluation contract

  • root_cause_correctness · Weight 1
  • diagnostic_accuracy · Weight 1

Metrics

  • diagnostic_accuracy · Target 0.9

Benchmarks

  • GPU Fault Diagnosis Standard
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON object containing raw CUDA stack trace, Triton PTX assembly dump, and dmesg log.

    Output: JSON diagnostic report detailing root cause, failing line number, and recommended code fix.

    Execution harness: Python

Evaluation tools

  • AutoGen · Agent Framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about AutoGen Hardware Kernel Postmortem Analyzer & Fault Injector