Skip to main content
Challenge

Build an MLflow Accelerator Procurement Scenario Tracker

Compare cloud accelerator reservation options against generative inference demand curves. Track multi-scenario procurement parameters, break-even metrics, and API-generated cost narratives using MLflow to manage financial lifecycle evaluations.

EngineeringHosted by Vera
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Compare cloud accelerator reservation options against generative inference demand curves. Track multi-scenario procurement parameters, break-even metrics, and API-generated cost narratives using MLflow to manage financial lifecycle evaluations.

AI FinOps Analyst

Parse projected token request distributions and burst patterns from normalized trace files using pandas.

Calculate baseline costs and break-even hours across on-demand and reserved accelerator pricing scenarios based on projected and billed hours.

Use the Anthropic API to synthesize qualitative cost narratives covering hardware obsolescence and capacity planning tradeoffs based on the calculated metrics.

Log scenario parameters, financial metrics, and the generated comparative procurement narrative into an MLflow tracking server as experiment runs.

Multi-scenario financial comparison workbench with downloadable MLflow metrics and text summaries.

Simplified synthetic cloud accelerator pricing catalogs and normalized traffic traces in a test folder. Starter examples below are synthetic. They are not downloaded records.

AWS Neuron documentation Models AWS EC2 accelerated instance types (e.g., Inf2, Trn1) across commitment tiers.

mlflow, anthropic, pandas. MLflow tracks procurement simulation runs, cost parameters, and comparative scenario metrics. Anthropic API access for generating qualitative procurement sensitivity evaluations.

Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

Evaluates total spend differences between on-demand and reserved instances accurately.

Calculates the exact monthly break-even hours for a reserved instance given on-demand and reserved rates.

Validates pricing schemas against standardized multi-cloud rate formats.

Emits formal procurement brief tracking total cost metrics into an MLflow experiment run.

```json [ { "input": { "accelerator_type": "AWS_INF2", "hourly_on_demand_usd": 10, "hourly_reserved_usd": 6, "projected_monthly_hours": 600, "workload_months": 12, "reservation_billed_hours_per_month": 730 }, "expected_output": { "total_on_demand_cost": 72000, "total_reserved_cost": 52560, "projected_savings_usd": 19440, "break_even_runtime_hours_per_month": 438, "procurement_recommendation": "COMMIT_RESERVED" } }, { "input": { "accelerator_type": "AWS_TRN1", "hourly_on_demand_usd": 20, "hourly_reserved_usd": 12, "projected_monthly_hours": 150, "workload_months": 6, "reservation_billed_hours_per_month": 730 }, "expected_output": { "total_on_demand_cost": 18000, "total_reserved_cost": 52560, "projected_savings_usd": -34560, "break_even_runtime_hours_per_month": 438, "procurement_recommendation": "RETAIN_ON_DEMAND" } } ] ```

Procurement comparisons are simulation only. No automated purchasing or cloud account binding. Explicitly label normalized examples as simplified synthetic app records, not complete official schema files. Use mock external writes. Require human review before any real deployment.

Rules

Participation guide

Procurement comparisons are simulation only. No automated purchasing or cloud account binding. Explicitly label normalized examples as simplified synthetic app records, not complete official schema files. Use mock external writes and human review.

Delivery guide

How work is evaluated

Evaluation

Evaluates total spend differences between on-demand and reserved instances accurately. Calculates the exact monthly break-even hours for a reserved instance given on-demand and reserved rates. Validates pricing schemas against standardized multi-cloud rate formats. Emits formal procurement brief tracking total cost metrics into an MLflow experiment run.

Submission format

Runnable source, pinned dependencies, setup commands, working interface, example outputs, recorded demo, tests, and action logs.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
4 scoring checks
Binary
1 pass or fail dimensions
Ordinal
3 scaled dimensions
Dimension 1

Complete workflow

Parse projected token request distributions and burst patterns from normalized trace files using pandas. Calculate baseline costs and break-even hours across on-demand and reserved accelerator pricing scenarios based on projected and billed hours. Use the Anthropic API to synthesize qualitative cost narratives covering hardware obsolescence and capacity planning tradeoffs based on the calculated metrics. Log scenario parameters, financial metrics, and the generated comparative procurement narrative into an MLflow tracking server as experiment runs.

Score scale
012
Score 0

Missing or incorrect.

Score 1

Partly complete, with documented gaps.

Score 2

Complete, with reproducible evidence.

Dimension 2

Correctness and evidence

Evaluates total spend differences between on-demand and reserved instances accurately. Calculates the exact monthly break-even hours for a reserved instance given on-demand and reserved rates. Validates pricing schemas against standardized multi-cloud rate formats. Emits formal procurement brief tracking total cost metrics into an MLflow experiment run.

Score scale
012
Score 0

Missing or incorrect.

Score 1

Partly complete, with documented gaps.

Score 2

Complete, with reproducible evidence.

Dimension 3

Review interface

Multi-scenario financial comparison workbench with downloadable MLflow metrics and text summaries.

Score scale
012
Score 0

Missing or incorrect.

Score 1

Partly complete, with documented gaps.

Score 2

Complete, with reproducible evidence.

Dimension 4

Safe execution

Procurement comparisons are simulation only. No automated purchasing or cloud account binding. Explicitly label normalized examples as simplified synthetic app records, not complete official schema files.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Parse projected token request distributions and burst patterns from normalized trace files using pandas.

  • Calculate baseline costs and break-even hours across on-demand and reserved accelerator pricing scenarios based on projected and billed hours.

  • Use the Anthropic API to synthesize qualitative cost narratives covering hardware obsolescence and capacity planning tradeoffs based on the calculated metrics.

  • Log scenario parameters, financial metrics, and the generated comparative procurement narrative into an MLflow tracking server as experiment runs.

Resources and assets

Reference links and supporting material

Dataset notes

Simplified synthetic cloud accelerator pricing catalogs and normalized traffic traces in a test folder. Supplied starter examples are synthetic, not source extracts.

Challenge setup

Configuration summary for this challenge.

Challenge input

Input details are not configured.

MLflow

MLOps

Evaluated output

No output checks are configured.

Evaluation checks
  • Parse projected token request distributions and burst patterns from normalized trace files using pandas. Ca...
  • Evaluates total spend differences between on-demand and reserved instances accurately. Calculates the exact...
  • Multi-scenario financial comparison workbench with downloadable MLflow metrics and text summaries.
Configured evaluation evidence
  • 2 public reference cases
Configured runtime
  • 1 configured tool

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • MLflow · Optional

Evaluation contract

  • Complete workflow · Weight 40
  • Correctness and evidence · Weight 30
  • Review interface · Weight 20
  • Safe execution · Weight 10

Reference cases

  • 2 public reference cases

Workflow tasks

No workflow tasks are configured.

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Build an MLflow Accelerator Procurement Scenario Tracker