Skip to main content
Back to challenges
Challenge

Advanced Compute Strategy Modeling with LlamaIndex

Create a decision support agent using LlamaIndex to model compute infrastructure strategies for AI developers. Inspired by the reported chip supply ambitions of major labs, this agent will evaluate compute procurement options. By utilizing Fireworks AI for model inference and Edge Impulse for performance simulation, the agent will analyze hypothetical supply chain risks and compute scalability, providing data-driven recommendations without relying on internal document databases.

Business OperationsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build an agentic simulation platform using LlamaIndex and GPT-5.4 Pro to analyze AI infrastructure strategies.

Delivery guide

How work is evaluated

Evaluation

Evaluate the agent's ability to propose optimal infrastructure configurations based on constraints.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

Feasibility Check

Ensure proposed strategy fits within input budget

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

Resource Efficiency

Score of recommended strategy viability • target: 85 • range: 0-100

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Master LlamaIndex agentic abstractions for complex reasoning tasks

  • Configure Fireworks AI as an inference backend for model performance

  • Integrate Edge Impulse APIs for simulated inference latency calculation

  • Design multi-step planning agents using GPT-5.4 Pro and Claude Sonnet 4.6.6

  • Orchestrate comparative strategy modeling between compute hardware providers

  • Build feedback loops between model inference performance and business requirements

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 1 tasks

Challenge setup

Evaluate the agent's ability to propose optimal infrastructure configurations based on constraints.

Challenge input

Constraint map

Llama Index

Data framework for LLM

Yupp AI

LLM evaluation platform

Fireworks AI

Fast inference and fine-tuning platform.

Evaluated output

Infrastructure plan JSON

Evaluation checks
  • Ensure proposed strategy fits within input budget
  • Score of recommended strategy viability • target: 85 • range: 0-100
Configured evaluation evidence
  • Resource Efficiency target: 85
  • 1 public reference case
Configured runtime
  • Python execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Llama Index · Required
  • Yupp AI · Optional
Policy Serving
  • Fireworks AI · Optional
Training Infra
  • Fireworks AI · Optional

Evaluation contract

  • Feasibility Check · Weight 1
  • Resource Efficiency · Weight 1

Metrics

  • Resource Efficiency · Target 85

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: Constraint map

    Output: Infrastructure plan JSON

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Advanced Compute Strategy Modeling with LlamaIndex