Skip to main content
Back to challenges
Challenge

AutoGen Hardware Kernel Postmortem Analyzer & Fault Injector

Implement an automated hardware postmortem analyzer using AutoGen to diagnose Triton kernel crashes, CUDA OOM failures, and fabric link degradation in inference clusters. Build multi-agent diagnostic loops that generate actionable hardware recovery plans with a 90% resolution accuracy for synthetic fault injection scenarios.

EngineeringHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Deploy an AutoGen multi-agent diagnostic squad to parse CUDA stack traces, Triton kernel faults, and fabric degradation logs.

Delivery guide

How work is evaluated

Evaluation

Evaluates root-cause diagnostic accuracy for hardware kernel crash logs.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

root_cause_correctness

Validates root cause matches ground truth failure mode.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

diagnostic_accuracy

Diagnostic confidence score. • target: 0.9 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Construct AutoGen multi-agent conversational system for hardware outage postmortems

  • Parse raw CUDA driver error codes (e.g. CUDA_ERROR_OUT_OF_MEMORY, Illegal Memory Access) and Triton compilation stack traces

  • Simulate synthetic hardware failure modes (ECC memory errors, NVLink fabric dropouts, thermal throttling)

  • Generate root-cause postmortem reports and remediation actions with 90%+ diagnostic accuracy

Resources and assets

Reference links and supporting material

Dataset notes

Synthetic GPU crash dump dataset containing 200 annotated CUDA driver stack traces, Triton kernel faults, and Xid error logs.

Challenge setup

Evaluates root-cause diagnostic accuracy for hardware kernel crash logs.

Challenge input

JSON object containing raw CUDA stack trace, Triton PTX assembly dump, and dmesg log.

AutoGen

Multi-agent conversational diagnostic system.

Evaluated output

JSON diagnostic report detailing root cause, failing line number, and recommended code fix.

Evaluation checks
  • Validates root cause matches ground truth failure mode.
  • Diagnostic confidence score. • target: 0.9 • range: 0-1
Configured evaluation evidence
  • Benchmark: GPU Fault Diagnosis Standard
  • Diagnostic Accuracy target: 0.9
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • AutoGen · Required
  • Anthropic · Optional
  • Microsoft AutoGen · Optional

Evaluation contract

  • root_cause_correctness · Weight 1
  • diagnostic_accuracy · Weight 1

Metrics

  • diagnostic_accuracy · Target 0.9

Benchmarks

  • GPU Fault Diagnosis Standard
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON object containing raw CUDA stack trace, Triton PTX assembly dump, and dmesg log.

    Output: JSON diagnostic report detailing root cause, failing line number, and recommended code fix.

    Execution harness: Python

Evaluation tools

  • AutoGen · Agent Framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about AutoGen Hardware Kernel Postmortem Analyzer & Fault Injector