Build Model Hardware Standard Accelerator Profiler with Pydantic AI and Qwen 3
Enterprise agent workloads require precise accelerator mapping across GPU, TPU, and AWS Trainium targets based on latency and memory constraints. You will build a type-safe profiling agent using Pydantic AI and Qwen 3 to evaluate model specifications against public hardware limits and output optimal deployment plans. Your solution will be validated by achieving zero schema errors and within 5% cost accuracy on baseline benchmark tests.
What you are building
Build a hardware specification profiler and placement optimizer using Pydantic AI validation models and Qwen 3 LLM reasoning.
How work is evaluated
Validates the Pydantic AI profiler agent against multi-accelerator target constraints and workload specs.
Datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
schema_conformance
Ensures agent output strictly parses into MHSPlacementDecision Pydantic model
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
cost_estimation_accuracy
Percentage accuracy of calculated cost relative to reference hardware benchmark • target: 0.95 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Learning goals
Define type-safe Pydantic models representing Model Hardware Standard (MHS) memory, memory bandwidth, and compute constraints
Integrate Qwen 3 via Pydantic AI agent dependencies for hardware allocation reasoning
Calculate compute-to-memory ratios (arithmetic intensity) for LLM inference workloads across H100, TPU v5e, and Trainium2 targets
Generate auditable placement specifications with predicted latency and cost trade-offs
Resources
Synthetic target workload traces representing 7B to 70B parameter models under varying concurrency and sequence lengths.
Challenge setup
Validates the Pydantic AI profiler agent against multi-accelerator target constraints and workload specs.
Challenge input
JSON containing model parameters (params_b, context_len, concurrency, max_latency_ms) and target accelerators list
Pydantic AI
Ensures type safety and structured outputs for hardware placement.
Qwen 3
Provides accurate multi-step hardware constraint analysis.
Evaluated output
JSON matching Pydantic schema with recommended_accelerator, estimated_latency_ms, estimated_cost_per_1m_tokens, and justification
- Ensures agent output strictly parses into MHSPlacementDecision Pydantic model
- Percentage accuracy of calculated cost relative to reference hardware benchmark • target: 0.95 • range: 0-1
- Benchmark: vLLM-Hardware-Bench
- Cost Estimation Accuracy target: 0.95
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Shows the first task, up to three tools, and up to three evidence items.
View technical recipe
Configured tools
- Pydantic AI · Required
- Anthropic · Optional
- Qwen 3 · Optional
- Pydantic AI · Required
Evaluation contract
- schema_conformance · Weight 1
- cost_estimation_accuracy · Weight 1
Metrics
- cost_estimation_accuracy · Target 0.95
Benchmarks
- vLLM-Hardware-Bench
- general
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON containing model parameters (params_b, context_len, concurrency, max_latency_ms) and target accelerators list
Output: JSON matching Pydantic schema with recommended_accelerator, estimated_latency_ms, estimated_cost_per_1m_tokens, and justification
Execution harness: Python
Evaluation tools
- Pydantic AI · Agent Framework
- Qwen 3 · LLM Engine
Recipe freeze marker
No recipe freeze time is recorded.