Back to challenges
Challenge

AutoGen MHS Workload Profiler & Accelerator Allocation Engine

Selecting optimal hardware for model workloads requires evaluating memory bandwidth, latency, and operational cost across diverse accelerators. Build an AutoGen multi-agent system powered by DeepSeek R1 to profile Model Hardware Standard (MHS) specifications and output verifiable hardware allocation plans under target latency constraints.

EngineeringHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build an AutoGen multi-agent workflow that analyzes hardware specifications and selects optimal GPU, TPU, and AWS Neuron accelerators for MHS agent workloads.

Delivery guide

How work is evaluated

Evaluation

Evaluates the AutoGen profiler's accuracy in hardware accelerator selection and MHS compliance generation.

Datasets

Shared data for this challenge

Review public datasets and any private uploads tied to your build.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

sla_compliance

Ensures selected hardware meets target SLA without exceeding cost budgets.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

allocation_accuracy

Accuracy of latency and hardware fitting calculation • target: 0.95 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Implement multi-agent hardware analysis using AutoGen and DeepSeek R1

  • Parse and validate Model Hardware Standard (MHS) device specifications

  • Calculate memory bandwidth, compute TFLOPS, and FP8/BF16 latency bounds

  • Generate cost and energy-optimized accelerator selection matrix

Resources and assets

Reference links and supporting material

Dataset notes

Hardware specs database containing memory bandwidth, FP8/BF16 TFLOPS, interconnect bandwidth, and cost for H100, TPU v5p, and Trainium2.

How this agent runs

Evaluates the AutoGen profiler's accuracy in hardware accelerator selection and MHS compliance generation.

Challenge input

JSON containing model parameters (params_b, context_len, batch_size, SLA_ms)

AutoGen

Framework for multi-agent negotiation on hardware constraints

DeepSeek R1

High-tier mathematical reasoning for roofline analysis

Evaluated output

JSON containing selected_accelerator, estimated_latency_ms, estimated_cost_usd, and MHS_profile

Checks for
  • Ensures selected hardware meets target SLA without exceeding cost budgets.
  • Accuracy of latency and hardware fitting calculation • target: 0.95 • range: 0-1
Proof of success
  • Benchmark: MLPerf Inference v4.0
  • Allocation Accuracy target: 0.95
  • 1 public reference case
Runtime evidence
  • Python execution harness
  • Python sandbox (unavailable on Versalist)
View technical recipe

Configured tools

Action Space
  • AutoGen · Required
  • DeepSeek · Optional
  • Anthropic · Optional

Evaluation contract

  • sla_compliance · Weight 1
  • allocation_accuracy · Weight 1

Recipe state

This is a preview. The configuration can change before the evaluation recipe is locked.

Frequently Asked Questions about AutoGen MHS Workload Profiler & Accelerator Allocation Engine