Triton Kernel Quantization Benchmark Agent with Pydantic AI and Together AI
Build an automated Triton kernel quantization benchmark workflow using Pydantic AI and Together AI to profile FP8 versus INT4 matrix operations. Evaluate throughput gains and perplexity tradeoffs to select optimal kernel configurations achieving a minimum 1.8x speedup.
What you are building
The core problem, expected build, and operating context for this challenge.
Utilize Pydantic AI's type-safe agent framework and Together AI inference to benchmark Triton quantization kernels.
How work is evaluated
Evaluates speedup multiplier and model quality preservation for benchmarked kernels.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
Minimum Speedup Met
Kernel speedup factor over baseline FP16 must be at least 1.8x.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Perplexity Delta
Increase in perplexity introduced by quantization. • target: 0.05 • range: 0-0.15
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Implement type-safe quantization benchmark workflows with Pydantic AI.
Integrate Together AI inference endpoints for perplexity comparison baseline.
Profile Triton FP8 and INT4 GEMM kernels across matrix sizes.
Determine automated kernel dispatch rules based on accuracy/throughput Pareto frontiers.
Reference links and supporting material
Synthetic matrix shape test cases and calibration datasets for LLM weight quantization.
Challenge setup
Evaluates speedup multiplier and model quality preservation for benchmarked kernels.
Challenge input
JSON with matrix dimensions and quantization format
Pydantic AI
Provides type-safe structured output handling and agent state management
Together AI
Provides fast model endpoints for baseline perplexity verification
Evaluated output
JSON with speedup_factor and perplexity_delta
- Kernel speedup factor over baseline FP16 must be at least 1.8x.
- Increase in perplexity introduced by quantization. • target: 0.05 • range: 0-0.15
- Benchmark: Triton Kernel Performance Benchmark
- Perplexity Delta target: 0.05
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- Pydantic AI · Optional
- Anthropic · Optional
- Together AI · Required
- Together AI · Required
- Pydantic AI · Optional
Evaluation contract
- Minimum Speedup Met · Weight 1
- Perplexity Delta · Weight 1
Metrics
- Perplexity Delta · Target 0.05
Benchmarks
- Triton Kernel Performance Benchmark
- general
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON with matrix dimensions and quantization format
Output: JSON with speedup_factor and perplexity_delta
Execution harness: Python
Evaluation tools
- Pydantic AI · Agent Framework
- Together AI · Inference Runtime
Recipe freeze marker
No recipe freeze time is recorded.