Back to challenges
Challenge

AutoGen Hardware Kernel Postmortem Analyzer & Fault Injector

Implement an automated hardware postmortem analyzer using AutoGen to diagnose Triton kernel crashes, CUDA OOM failures, and fabric link degradation in inference clusters. Build multi-agent diagnostic loops that generate actionable hardware recovery plans with a 90% resolution accuracy for synthetic fault injection scenarios.

EngineeringHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Deploy an AutoGen multi-agent diagnostic squad to parse CUDA stack traces, Triton kernel faults, and fabric degradation logs.

Delivery guide

How work is evaluated

Evaluation

Evaluates root-cause diagnostic accuracy for hardware kernel crash logs.

Datasets

Shared data for this challenge

Review public datasets and any private uploads tied to your build.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

root_cause_correctness

Validates root cause matches ground truth failure mode.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

diagnostic_accuracy

Diagnostic confidence score. • target: 0.9 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Construct AutoGen multi-agent conversational system for hardware outage postmortems

  • Parse raw CUDA driver error codes (e.g. CUDA_ERROR_OUT_OF_MEMORY, Illegal Memory Access) and Triton compilation stack traces

  • Simulate synthetic hardware failure modes (ECC memory errors, NVLink fabric dropouts, thermal throttling)

  • Generate root-cause postmortem reports and remediation actions with 90%+ diagnostic accuracy

Resources and assets

Reference links and supporting material

Dataset notes

Synthetic GPU crash dump dataset containing 200 annotated CUDA driver stack traces, Triton kernel faults, and Xid error logs.

How this agent runs

Evaluates root-cause diagnostic accuracy for hardware kernel crash logs.

Challenge input

JSON object containing raw CUDA stack trace, Triton PTX assembly dump, and dmesg log.

AutoGen

Multi-agent conversational diagnostic system.

Evaluated output

JSON diagnostic report detailing root cause, failing line number, and recommended code fix.

Checks for
  • Validates root cause matches ground truth failure mode.
  • Diagnostic confidence score. • target: 0.9 • range: 0-1
Proof of success
  • Benchmark: GPU Fault Diagnosis Standard
  • Diagnostic Accuracy target: 0.9
  • 1 public reference case
Runtime evidence
  • Python execution harness
  • Python sandbox (unavailable on Versalist)
View technical recipe

Configured tools

Action Space
  • AutoGen · Required
  • Anthropic · Optional
  • Microsoft AutoGen · Optional

Evaluation contract

  • root_cause_correctness · Weight 1
  • diagnostic_accuracy · Weight 1

Recipe state

This is a preview. The configuration can change before the evaluation recipe is locked.

Frequently Asked Questions about AutoGen Hardware Kernel Postmortem Analyzer & Fault Injector