Build a Local Open-Weight Model Quantization Profiling Workspace — implementation

## Build
Benchmark open-weight models across FP8, INT8, and INT4 precisions on local developer test rigs. Evaluate memory footprint and perplexity tradeoffs to recommend target deployment configs.

## User
Edge Deployment Engineer

## Workflow
1. Profile open-weight model weights using local Ollama or vLLM open runtimes.
2. Measure VRAM consumption, memory bandwidth saturation, and tokens-per-second.
3. Call hosted Claude to evaluate quantization degradation reports and operational limits.
4. Output an optimized deployment recommendation manifest for edge hardware.

## Interface
Local web UI showing precision comparison radar charts, latency distributions, and exportable config manifests.

## Data
Use open-weight small models (e.g. Qwen-1.5B or Llama-3-8B) with standard synthetic calibration texts.
Starter examples below are synthetic. They are not downloaded records.

## Source
[vLLM open-model serving](https://raw.githubusercontent.com/vllm-project/vllm/main/README.md)
Applies vLLM quantization format specifications (FP8, INT8, INT4, AWQ) to structure memory benchmark comparisons.

## Tools
ollama, anthropic-claude, pydantic-ai.
Provides a clean local execution environment to benchmark open-weight model quantization variants without cloud dependencies.
Hosted Claude API for cross-precision degradation analysis. Local execution strictly for open-weight test models.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Profiles at least two quantization precisions of an open-weight model on local test hardware.
- Measures token latency and active memory footprint accurately.
- Generates structured JSON hardware deployment specifications via Claude API.

## Starter Examples
```json
[
  {
    "input": {
      "model_name": "Qwen2.5-7B",
      "available_vram_gb": 12,
      "benchmark_results": [
        {
          "precision": "FP16",
          "vram_used_gb": 14.2,
          "tps": 22.1,
          "eval_loss": 2.14
        },
        {
          "precision": "INT4_AWQ",
          "vram_used_gb": 5.8,
          "tps": 48.5,
          "eval_loss": 2.21
        }
      ]
    },
    "expected_output": {
      "selected_precision": "INT4_AWQ",
      "fit_status": "fits_with_headroom",
      "kv_cache_headroom_gb": 6.2,
      "performance_gain_pct": 119.4,
      "deployment_flag": "--quantization awq --gpu-memory-utilization 0.9"
    }
  },
  {
    "input": {
      "model_name": "Llama3-8B",
      "available_vram_gb": 4,
      "benchmark_results": [
        {
          "precision": "INT4_AWQ",
          "vram_used_gb": 5.6,
          "tps": 32,
          "eval_loss": 2.3
        }
      ]
    },
    "expected_output": {
      "selected_precision": "none",
      "fit_status": "insufficient_vram",
      "kv_cache_headroom_gb": 0,
      "performance_gain_pct": 0,
      "deployment_flag": "Requires minimum 8GB VRAM or CPU offload"
    }
  }
]
```

## Safety
Operates only on open-weight models in local sandboxes; never attempts to load or quantize proprietary Claude weights.
Use mock external writes. Require human review before any real deployment.