Build Model Hardware Standard Accelerator Profiler with Pydantic AI and Qwen 3
Enterprise agent workloads require precise accelerator mapping across GPU, TPU, and AWS Trainium targets based on latency and memory constraints. You will build a type-safe profiling agent using Pydantic AI and Qwen 3 to evaluate model specifications against public hardware limits and output optimal deployment plans. Your solution will be validated by achieving zero schema errors and within 5% cost accuracy on baseline benchmark tests.
What you are building
The core problem, expected build, and operating context for this challenge.
Build a hardware specification profiler and placement optimizer using Pydantic AI validation models and Qwen 3 LLM reasoning.
How work is evaluated
Validates the Pydantic AI profiler agent against multi-accelerator target constraints and workload specs.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
schema_conformance
Ensures agent output strictly parses into MHSPlacementDecision Pydantic model
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
cost_estimation_accuracy
Percentage accuracy of calculated cost relative to reference hardware benchmark • target: 0.95 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Define type-safe Pydantic models representing Model Hardware Standard (MHS) memory, memory bandwidth, and compute constraints
Integrate Qwen 3 via Pydantic AI agent dependencies for hardware allocation reasoning
Calculate compute-to-memory ratios (arithmetic intensity) for LLM inference workloads across H100, TPU v5e, and Trainium2 targets
Generate auditable placement specifications with predicted latency and cost trade-offs
Reference links and supporting material
Synthetic target workload traces representing 7B to 70B parameter models under varying concurrency and sequence lengths.
Challenge setup
Validates the Pydantic AI profiler agent against multi-accelerator target constraints and workload specs.
Challenge input
JSON containing model parameters (params_b, context_len, concurrency, max_latency_ms) and target accelerators list
Pydantic AI
Ensures type safety and structured outputs for hardware placement.
Qwen 3
Provides accurate multi-step hardware constraint analysis.
Evaluated output
JSON matching Pydantic schema with recommended_accelerator, estimated_latency_ms, estimated_cost_per_1m_tokens, and justification
- Ensures agent output strictly parses into MHSPlacementDecision Pydantic model
- Percentage accuracy of calculated cost relative to reference hardware benchmark • target: 0.95 • range: 0-1
- Benchmark: vLLM-Hardware-Bench
- Cost Estimation Accuracy target: 0.95
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- Pydantic AI · Required
- Anthropic · Optional
- Qwen 3 · Optional
- Pydantic AI · Required
Evaluation contract
- schema_conformance · Weight 1
- cost_estimation_accuracy · Weight 1
Metrics
- cost_estimation_accuracy · Target 0.95
Benchmarks
- vLLM-Hardware-Bench
- general
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON containing model parameters (params_b, context_len, concurrency, max_latency_ms) and target accelerators list
Output: JSON matching Pydantic schema with recommended_accelerator, estimated_latency_ms, estimated_cost_per_1m_tokens, and justification
Execution harness: Python
Evaluation tools
- Pydantic AI · Agent Framework
- Qwen 3 · LLM Engine
Recipe freeze marker
No recipe freeze time is recorded.