Build a Local Open-Weight Quantization Precision Tradeoff Explorer

Profile open-weight models across INT4, INT8, and FP8 formats on test hardware. Synthesize memory savings and degradation trade-offs using hosted Claude.

ML Platforms Engineer

Ingest local open-weight model parameter definitions and candidate quantization profiles (FP8, INT8, INT4).

Compute theoretical memory footprint reductions and simulated memory bandwidth requirement metrics.

Query hosted Claude via Anthropic API to analyze accuracy retention trade-offs and recommended target workloads.

Output an executive precision trade-off dossier with hardware memory occupancy curves.

Web dashboard displaying memory compression charts and Claude-generated deployment recommendations.

Synthetic open-weight model architecture specs (Qwen, Llama variants) paired with mock benchmark compression ratios. Starter examples below are synthetic. They are not downloaded records.

vLLM open-model serving Leverages vLLM quantization format definitions including FP8, INT8, and INT4 for open-weight serving optimization.

mlflow, anthropic. MLflow tracks quantization experiment runs, model artifact sizes, and throughput metrics across local test configurations. Hosted a currently available hosted Claude model for trade-off evaluation; local execution handles open-weight configuration parsing.

Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

Accurately calculates memory compression factors across FP8, INT8, and INT4.

Records experiment runs into MLflow tracking tracking server.

Produces hosted Claude rationales for production quantization deployment readiness.

```json [ { "input": { "base_model": "Llama-3-8B", "base_precision": "FP16", "target_precision": "INT4", "parameter_count_b": 8, "benchmark_eval_loss_delta": 0.08 }, "expected_output": { "original_size_gb": 16, "quantized_size_gb": 4.5, "memory_reduction_pct": 71.8, "deployment_verdict": "Suitable for edge deployment or high-throughput batching where 0.08 loss increase is tolerable." } }, { "input": { "base_model": "Llama-3-70B", "base_precision": "FP16", "target_precision": "FP8", "parameter_count_b": 70, "benchmark_eval_loss_delta": 0.01 }, "expected_output": { "original_size_gb": 140, "quantized_size_gb": 70, "memory_reduction_pct": 50, "deployment_verdict": "Recommended for enterprise deployment. Halves memory requirements with negligible loss degradation." } } ] ```

Processes local open-weight model metadata only; no remote execution of unverified binaries. Use mock external writes. Require human review before any real deployment.