Build a Multi-Tenant GPU Power Capping and Quota Manager — implementation

## Build
Analyze DCGM power draw telemetry across clusters to enforce multi-tenant energy budgets. Detect workloads operating below thermal design power and generate dynamic power cap adjustments for heterogeneous workloads using hosted Claude.

## User
Datacenter Operations Engineer

## Workflow
1. Ingest normalized time-series DCGM power usage (Watts) and active tenant mappings per GPU.
2. Detect workloads operating below their thermal design power (TDP) that could safely run under a lowered power limit.
3. Invoke hosted Claude to analyze power efficiency profiles and suggest dynamic software power caps.
4. Export an executable bash script containing nvidia-smi commands to enforce the updated power budgets.

## Interface
Command-line utility and reporting dashboard for cluster administrators to review and apply power budget adjustments per tenant.

## Data
Generate simplified synthetic app records in JSON modeling DCGM GPU metrics (average power draw, utilization percentages, tenant mappings) with varying workload intensity profiles.
Starter examples below are synthetic. They are not downloaded records.

## Source
[NVIDIA DCGM exporter](https://raw.githubusercontent.com/NVIDIA/dcgm-exporter/main/README.md)
References DCGM power metrics (power draw and utilization) as exposed by dcgm-exporter for accurate cluster profiling.

## Tools
anthropic-claude, pandas, pydantic.
anthropic-claude processes complex multi-metric power timelines to generate precise, natural-language rationales and actionable scripts.
Hosted Claude API for power cap reasoning; local metric parsing utilities.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Correctly isolates average power consumption per tenant over the specified timeframe.
- Produces safe power cap reductions only for memory-bound workloads not utilizing full SM power.
- Ensures all adjustments respect the hardware's minimum and maximum allowed power limits.
- Validates that output commands are syntactically correct.

## Starter Examples
```json
[
  {
    "input": {
      "node_id": "gpu-node-04",
      "gpu_max_power_w": 400,
      "gpu_min_power_w": 100,
      "telemetry": [
        {
          "tenant": "nlp-batch",
          "gpu_id": 0,
          "avg_power_draw_w": 185,
          "utilization_pct": 98
        },
        {
          "tenant": "vision-train",
          "gpu_id": 1,
          "avg_power_draw_w": 380,
          "utilization_pct": 99
        }
      ]
    },
    "expected_output": {
      "power_optimizations_found": true,
      "proposed_caps": [
        {
          "gpu_id": 0,
          "new_limit_w": 250,
          "rationale": "Memory-bound workload maxing at 185W; safe to cap at 250W to save node power budget."
        }
      ],
      "executable_script": "nvidia-smi -i 0 -pl 250"
    }
  },
  {
    "input": {
      "node_id": "gpu-node-05",
      "gpu_max_power_w": 400,
      "gpu_min_power_w": 100,
      "telemetry": [
        {
          "tenant": "rl-train",
          "gpu_id": 0,
          "avg_power_draw_w": 395,
          "utilization_pct": 100
        }
      ]
    },
    "expected_output": {
      "power_optimizations_found": false,
      "proposed_caps": [],
      "executable_script": ""
    }
  }
]
```

## Safety
Avoids executing live server power adjustments directly, relying on generated scripts for manual administrator review.
Use mock external writes. Require human review before any real deployment.