Skip to main content
Back to challenges
Challenge

Sovereign AI Model Evaluation & Regulatory Compliance Framework

Joint bilateral digital and AI infrastructure deployments require strict compliance benchmarking against national AI governance frameworks. Build an automated model evaluation harness using Deepspeed and Prometheus to assess LLM safety, bias, and latency across regional data centers. Achieve reproducible benchmark reports conforming to ASEAN AI Ethics guidelines.

Workflow AutomationHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build a sovereign AI governance model evaluation harness enforcing regional ethical compliance standards.

Delivery guide

How work is evaluated

Evaluation

Evaluates accuracy of toxicity detection and completeness of Prometheus metric outputs.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

safety_refusal_check

Ensures illicit requests trigger clean model refusals.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

eval_throughput_prompts_per_sec

Evaluation pipeline execution speed. • target: 25 • range: 0-100

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Construct automated benchmark harnesses evaluating LLMs against hallucination metrics

  • Implement multi-lingual prompt safety probes matching ASEAN AI Governance Frameworks

  • Expose metric collection pipelines integrating with Prometheus and Grafana dashboards

  • Generate automated pass/fail compliance certification cards for enterprise model deployment

Resources and assets

Reference links and supporting material

Dataset notes

Synthetic prompt injection and safety benchmark probes customized for ASEAN regulatory contexts.

Challenge setup

Evaluates accuracy of toxicity detection and completeness of Prometheus metric outputs.

Challenge input

JSON model prompt and completion pair

Prometheus Client

Standard library for publishing metrics to Prometheus monitoring server.

Evaluated output

JSON compliance score card

Evaluation checks
  • Ensures illicit requests trigger clean model refusals.
  • Evaluation pipeline execution speed. • target: 25 • range: 0-100
Configured evaluation evidence
  • Benchmark: AIGov-Safety-Bench
  • Eval Throughput Prompts Per Sec target: 25
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Aim · Required
  • DeepSpeed · Optional
  • Cognition Devin · Optional
Observation
  • Cognition Devin · Optional
Orchestration
  • Cognition Devin · Optional

Evaluation contract

  • safety_refusal_check · Weight 1
  • eval_throughput_prompts_per_sec · Weight 1

Metrics

  • eval_throughput_prompts_per_sec · Target 25

Benchmarks

  • AIGov-Safety-Bench
  • security

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON model prompt and completion pair

    Output: JSON compliance score card

    Execution harness: Python

Evaluation tools

  • Prometheus Client · Telemetry Metric Exporter

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Sovereign AI Model Evaluation & Regulatory Compliance Framework