Build an Accelerator Quota Allocation and Over-Subscription Planner — implementation

## Build
Model multi-team GPU reservation quotas against historical compute usage traces to identify over-subscription capacity. Generate team quota adjustment proposals with hosted Claude without risking OOM evictions.

## User
Compute Infrastructure Manager

## Workflow
1. Ingest team allocation quotas alongside historical 30-day GPU time-slice usage logs.
2. Calculate concurrency peaks, idle time-blocks, and safe over-subscription ratios.
3. Use hosted Claude to draft quota restructuring recommendations and SLA impact analyses.
4. Publish an over-subscription plan with scheduled reclamation guardrails for cluster admins.

## Interface
Quota planning portal featuring historical demand duration curves and adjustable over-subscription sliders.

## Data
Construct synthetic JSON traces showing hourly GPU reservation claims versus actual compute activity per team.
Starter examples below are synthetic. They are not downloaded records.

## Source
[NVIDIA DCGM exporter](https://raw.githubusercontent.com/NVIDIA/dcgm-exporter/main/README.md)
Applies GPU compute and memory utilization metrics as defined in standard DCGM monitoring systems.

## Tools
optuna, anthropic-claude, pydantic-ai.
optuna optimizes numerical over-subscription ratios to minimize unused capacity while capping eviction probability.
Hosted Claude API for narrative justification; local numerical optimization via Optuna.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Calculate the risk score of queue starvation under simulated over-subscription levels.
- Synthesize team-by-team quota adjustment cards with transparent historical rationale.
- Enforce strict priority rules preserving critical training workloads over batch eval jobs.

## Starter Examples
```json
[
  {
    "input": {
      "cluster_total_gpus": 64,
      "allocations": [
        {
          "team": "ad-ranking",
          "assigned_gpus": 32,
          "peak_concurrent_gpus": 18,
          "avg_utilization_pct": 45
        },
        {
          "team": "speech-ai",
          "assigned_gpus": 32,
          "peak_concurrent_gpus": 20,
          "avg_utilization_pct": 50
        }
      ]
    },
    "expected_output": {
      "current_effective_headroom_gpus": 26,
      "recommended_over_subscription_ratio": 1.35,
      "proposed_reallocations": [
        {
          "team": "ad-ranking",
          "new_guaranteed_gpus": 20,
          "burst_gpus": 12
        },
        {
          "team": "speech-ai",
          "new_guaranteed_gpus": 22,
          "burst_gpus": 10
        }
      ],
      "unlocked_free_pool_gpus": 22
    }
  },
  {
    "input": {
      "cluster_total_gpus": 16,
      "allocations": [
        {
          "team": "core-nlp",
          "assigned_gpus": 16,
          "peak_concurrent_gpus": 16,
          "avg_utilization_pct": 95
        }
      ]
    },
    "expected_output": {
      "current_effective_headroom_gpus": 0,
      "recommended_over_subscription_ratio": 1,
      "proposed_reallocations": [
        {
          "team": "core-nlp",
          "new_guaranteed_gpus": 16,
          "burst_gpus": 0
        }
      ],
      "unlocked_free_pool_gpus": 0
    }
  }
]
```

## Safety
Analytical quota modeling tool only; does not modify scheduler cgroups or evict running jobs.
Use mock external writes. Require human review before any real deployment.