Build a Multi-Tenant Accelerator Chargeback and Idle Waste Auditor — implementation

## Build
Correlate allocated GPU tenancy windows with active kernel compute metrics to audit idle capacity costs. Construct transparent chargeback adjustments and team reclaim notices using hosted Claude.

## User
AI Infrastructure Operations Lead

## Workflow
1. Collect multi-tenant job scheduling logs alongside high-watermark GPU memory and tensor core telemetry.
2. Isolate reservations where memory was reserved without active compute execution.
3. Prompt hosted Claude to synthesize an executive chargeback audit explaining preventable idle expenditure.
4. Deliver an interactive reconciliation report with itemized penalty and credit recommendations.

## Interface
Executive audit console showing departmental cost attribution, wasted dollar metrics, and appeal workflows.

## Data
Generate synthetic JSON event logs mapping pod tenant tags to DCGM compute activity records.
Starter examples below are synthetic. They are not downloaded records.

## Source
[NVIDIA DCGM exporter](https://raw.githubusercontent.com/NVIDIA/dcgm-exporter/main/README.md)
Extracts GPU tensor core utilization and active memory footprint metrics defined in DCGM exporter.

## Tools
mlflow, anthropic-claude, pydantic-ai.
mlflow tracks multi-run financial experiments, chargeback rules, and cost-recovery metric logs.
Hosted Claude API for stakeholder communication and policy audit summaries.

## Deliverables
Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

## Acceptance
- Distinguish between genuine compute idle time and model initialization periods.
- Calculate unrecovered financial spend based on per-accelerator hourly internal cost models.
- Provide verifiable audit traces mapping job IDs to specific wasted capacity intervals.

## Starter Examples
```json
[
  {
    "input": {
      "tenant_id": "analytics-org",
      "billing_rate_per_gpu_hr": 4,
      "jobs": [
        {
          "job_id": "job-891",
          "allocated_gpus": 8,
          "duration_hours": 10,
          "active_compute_hours": 1.5
        }
      ]
    },
    "expected_output": {
      "tenant_id": "analytics-org",
      "total_charged_usd": 320,
      "idle_waste_usd": 272,
      "waste_percentage": 85,
      "audit_ruling": "Job reserved 8 GPUs for 10 hours with 8.5 hours of unutilized memory holding.",
      "recommended_action": "Apply idle hold surcharge of $272.00 or enforce auto-reclaim after 30 minutes inactivity."
    }
  },
  {
    "input": {
      "tenant_id": "recommendations-team",
      "billing_rate_per_gpu_hr": 4,
      "jobs": [
        {
          "job_id": "job-104",
          "allocated_gpus": 4,
          "duration_hours": 5,
          "active_compute_hours": 4.8
        }
      ]
    },
    "expected_output": {
      "tenant_id": "recommendations-team",
      "total_charged_usd": 80,
      "idle_waste_usd": 3.2,
      "waste_percentage": 4,
      "audit_ruling": "High efficiency usage observed; exceeds 90% active utilization target.",
      "recommended_action": "Approve standard chargeback without penalties."
    }
  }
]
```

## Safety
Analytical accounting and billing reconciliation only; produces no real ledger modifications.
Use mock external writes. Require human review before any real deployment.