Triton Kernel Quantization Benchmark Agent with Pydantic AI and Together AI
Build an automated Triton kernel quantization benchmark workflow using Pydantic AI and Together AI to profile FP8 versus INT4 matrix operations. Evaluate throughput gains and perplexity tradeoffs to select optimal kernel configurations achieving a minimum 1.8x speedup.
What you are building
The core problem, expected build, and operating context for this challenge.
Utilize Pydantic AI's type-safe agent framework and Together AI inference to benchmark Triton quantization kernels.
How work is evaluated
Evaluates speedup multiplier and model quality preservation for benchmarked kernels.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
Minimum Speedup Met
Kernel speedup factor over baseline FP16 must be at least 1.8x.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Perplexity Delta
Increase in perplexity introduced by quantization. • target: 0.05 • range: 0-0.15
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Implement type-safe quantization benchmark workflows with Pydantic AI.
Integrate Together AI inference endpoints for perplexity comparison baseline.
Profile Triton FP8 and INT4 GEMM kernels across matrix sizes.
Determine automated kernel dispatch rules based on accuracy/throughput Pareto frontiers.
Reference links and supporting material
Synthetic matrix shape test cases and calibration datasets for LLM weight quantization.
How this agent runs
Evaluates speedup multiplier and model quality preservation for benchmarked kernels.
Challenge input
JSON with matrix dimensions and quantization format
Pydantic AI
Provides type-safe structured output handling and agent state management
Together AI
Provides fast model endpoints for baseline perplexity verification
Evaluated output
JSON with speedup_factor and perplexity_delta
- Kernel speedup factor over baseline FP16 must be at least 1.8x.
- Increase in perplexity introduced by quantization. • target: 0.05 • range: 0-0.15
- Benchmark: Triton Kernel Performance Benchmark
- Perplexity Delta target: 0.05
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
View technical recipe
Configured tools
- Pydantic AI · Optional
- Anthropic · Optional
- Together AI · Required
- Together AI · Required
- Pydantic AI · Optional
Evaluation contract
- Minimum Speedup Met · Weight 1
- Perplexity Delta · Weight 1
Recipe state
This is a preview. The configuration can change before the evaluation recipe is locked.