Hardware Quantization Quality and Speed Evaluator with LangChain and LangGraph
Quantizing frontier models down to FP8 or INT4 significantly reduces HBM bandwidth constraints but can introduce accuracy degradation. You will build a stateful evaluation agent with LangChain and LangGraph to run quantization benchmarks across multiple dynamic precision formats, reporting perplexity degradation alongside token throughput gains. The solution will be validated by accurately predicting optimal precision per task context.
What you are building
The core problem, expected build, and operating context for this challenge.
Build a stateful evaluation workflow with LangChain and LangGraph to evaluate model quality vs hardware latency across FP16, FP8, and INT4 quantizations.
How work is evaluated
Evaluates LangGraph workflow output on recommending optimal quantization for specific tasks.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
tolerance_compliance
Ensures selected precision does not breach maximum allowed accuracy drop
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
pareto_optimal_selection
Score indicating whether chosen precision lies on optimal Pareto frontier • target: 1 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Build a stateful LangGraph graph managing benchmark execution stages (Load -> Evaluate -> Compare -> Report)
Integrate LangChain evaluators to measure answer quality degradation across precision levels
Measure hardware memory footprint reductions across FP16, FP8, and INT4 dynamic formats
Produce recommendations balancing perplexity loss against tokens per second gains
Reference links and supporting material
Benchmarking results measuring FP16, FP8, and INT4 quantization output on coding and math tasks.
Challenge setup
Evaluates LangGraph workflow output on recommending optimal quantization for specific tasks.
Challenge input
JSON containing task_type, accuracy_tolerance_pct, and precision_metrics (FP16, FP8, INT4)
LangChain
Provides standard prompt templates and chain evaluators.
Evaluated output
JSON detailing recommended_precision, memory_saved_gb, throughput_gain_multiplier, and accuracy_delta_pct
- Ensures selected precision does not breach maximum allowed accuracy drop
- Score indicating whether chosen precision lies on optimal Pareto frontier • target: 1 • range: 0-1
- Benchmark: Quantization-Quality-Bench
- Pareto Optimal Selection target: 1
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- LangGraph · Required
- Langchain · Optional
- LangChain · Optional
- LangGraph · Required
- Langchain · Optional
- LangChain · Optional
Evaluation contract
- tolerance_compliance · Weight 1
- pareto_optimal_selection · Weight 1
Metrics
- pareto_optimal_selection · Target 1
Benchmarks
- Quantization-Quality-Bench
- general
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON containing task_type, accuracy_tolerance_pct, and precision_metrics (FP16, FP8, INT4)
Output: JSON detailing recommended_precision, memory_saved_gb, throughput_gain_multiplier, and accuracy_delta_pct
Execution harness: Python
Evaluation tools
- LangChain · Agent Framework
Recipe freeze marker
No recipe freeze time is recorded.