Build a Server Firmware Regression Detector and Change Approval Pipeline — implementation

## Build
Analyze performance benchmark diffs and driver logs across server firmware canary releases. Hosted Claude evaluates throughput degradations to approve or reject fleet-wide firmware deployment.

## User
Firmware Systems Qualification Engineer

## Workflow
1. Ingest benchmark execution logs comparing baseline firmware against newly deployed canary firmware versions.
2. Extract compute throughput, kernel launch latency, and PCIe error counters from normalized test runs.
3. Submit regression comparison tables to hosted Claude to determine whether performance deviations breach sign-off thresholds.
4. Output a structured firmware qualification scorecard and automated change ticket disposition.

## Interface
Web qualification dashboard displaying regression delta charts and change approval sign-off toggles.

## Data
Create synthetic JSON files logging canary benchmark metrics, firmware versions, PCIe transaction retries, and latency deltas.
Starter examples below are synthetic. They are not downloaded records.

## Source
[NVIDIA DCGM exporter](https://raw.githubusercontent.com/NVIDIA/dcgm-exporter/main/README.md)
Extracts hardware performance counters and PCIe error telemetry patterns as described in DCGM documentation.

## Tools
langgraph, anthropic, pydantic, pandas.
LangGraph manages stateful multi-step review gates for processing, evaluating, and human-signing firmware qualification records.
Hosted Claude API for technical regression analysis and qualification report drafting.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Flags canary builds showing greater than 3% throughput degradation compared to baseline.
- Identifies non-zero PCIe replay or error counters introduced during canary execution.
- Generates clear qualification verdicts: APPROVED_FOR_ROLLOUT, REJECTED_REGRESSION, or MANUAL_REVIEW.

## Starter Examples
```json
[
  {
    "input": {
      "target_firmware": "v555.42.02",
      "baseline_firmware": "v550.54.14",
      "benchmark_metrics": {
        "baseline_throughput_tps": 1250,
        "canary_throughput_tps": 1245,
        "pcie_replay_errors": 0,
        "kernel_launch_latency_us": 12.1
      }
    },
    "expected_output": {
      "qualification_verdict": "APPROVED_FOR_ROLLOUT",
      "throughput_delta_pct": -0.4,
      "pcie_error_flag": false,
      "approval_memo": "Performance delta within 3% tolerance and zero PCIe bus errors."
    }
  },
  {
    "input": {
      "target_firmware": "v555.50.01",
      "baseline_firmware": "v550.54.14",
      "benchmark_metrics": {
        "baseline_throughput_tps": 1250,
        "canary_throughput_tps": 1120,
        "pcie_replay_errors": 42,
        "kernel_launch_latency_us": 18.5
      }
    },
    "expected_output": {
      "qualification_verdict": "REJECTED_REGRESSION",
      "throughput_delta_pct": -10.4,
      "pcie_error_flag": true,
      "approval_memo": "Significant throughput degradation (-10.4%) and active PCIe replay errors detected."
    }
  }
]
```

## Safety
Evaluation and review only. Does not flash firmware or modify production cluster nodes.
Use mock external writes. Require human review before any real deployment.