Build a Local Open-Weight Model Quantization Profiling Workspace

Benchmark open-weight models across FP8, INT8, and INT4 precisions on local developer test rigs. Evaluate memory footprint and perplexity tradeoffs to recommend target deployment configs.

Edge Deployment Engineer

Profile open-weight model weights using local Ollama or vLLM open runtimes.

Measure VRAM consumption, memory bandwidth saturation, and tokens-per-second.

Call hosted Claude to evaluate quantization degradation reports and operational limits.

Output an optimized deployment recommendation manifest for edge hardware.

Local web UI showing precision comparison radar charts, latency distributions, and exportable config manifests.

Use open-weight small models (e.g. Qwen-1.5B or Llama-3-8B) with standard synthetic calibration texts. Starter examples below are synthetic. They are not downloaded records.

vLLM open-model serving Applies vLLM quantization format specifications (FP8, INT8, INT4, AWQ) to structure memory benchmark comparisons.

ollama, anthropic-claude, pydantic-ai. Provides a clean local execution environment to benchmark open-weight model quantization variants without cloud dependencies. Hosted Claude API for cross-precision degradation analysis. Local execution strictly for open-weight test models.

Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

Profiles at least two quantization precisions of an open-weight model on local test hardware.

Measures token latency and active memory footprint accurately.

Generates structured JSON hardware deployment specifications via Claude API.

```json [ { "input": { "model_name": "Qwen2.5-7B", "available_vram_gb": 12, "benchmark_results": [ { "precision": "FP16", "vram_used_gb": 14.2, "tps": 22.1, "eval_loss": 2.14 }, { "precision": "INT4_AWQ", "vram_used_gb": 5.8, "tps": 48.5, "eval_loss": 2.21 } ] }, "expected_output": { "selected_precision": "INT4_AWQ", "fit_status": "fits_with_headroom", "kv_cache_headroom_gb": 6.2, "performance_gain_pct": 119.4, "deployment_flag": "--quantization awq --gpu-memory-utilization 0.9" } }, { "input": { "model_name": "Llama3-8B", "available_vram_gb": 4, "benchmark_results": [ { "precision": "INT4_AWQ", "vram_used_gb": 5.6, "tps": 32, "eval_loss": 2.3 } ] }, "expected_output": { "selected_precision": "none", "fit_status": "insufficient_vram", "kv_cache_headroom_gb": 0, "performance_gain_pct": 0, "deployment_flag": "Requires minimum 8GB VRAM or CPU offload" } } ] ```

Operates only on open-weight models in local sandboxes; never attempts to load or quantize proprietary Claude weights. Use mock external writes. Require human review before any real deployment.