Build a National AI Research Benchmark Evaluator with Google ADK
India's growing ecosystem of featured AI startups requires standardized benchmark evaluation. In this challenge, you will implement an automated model benchmarking pipeline using Google ADK and Vertex AI Evaluation services. The evaluator tests emerging AI startup models across Indic translation accuracy, reasoning capabilities, and hallucination safety benchmarks, producing standardized leaderboard report cards.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an automated model benchmarking pipeline using Google ADK and Vertex AI evaluation metrics.
How work is evaluated
Evaluates benchmark score aggregation accuracy for tested AI model outputs.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
tier_ranking_test
Ensures high performance model (BLEU > 30, acc > 0.8) qualifies as TIER_1_ENTERPRISE.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
evaluation_consistency
Deterministic scoring consistency across repeated benchmark runs. • target: 1 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Configure Google ADK benchmarking agents with standard metric suites.
Execute automated evaluation runs across Indic language test sets.
Compute hallucination rate, BLEU/ROUGE scores, and latency metrics.
Publish formatted markdown and JSON model leaderboard summaries.
Reference links and supporting material
Standardized evaluation prompt suite covering Indic translation, coding, and mathematical reasoning.
Challenge setup
Evaluates benchmark score aggregation accuracy for tested AI model outputs.
Challenge input
JSON with model_name, bleu_score, reasoning_acc, latency_ms.
Google ADK
Required framework for Vertex AI evaluation orchestration.
Evaluated output
JSON with overall_score, tier_ranking, production_ready.
- Ensures high performance model (BLEU > 30, acc > 0.8) qualifies as TIER_1_ENTERPRISE.
- Deterministic scoring consistency across repeated benchmark runs. • target: 1 • range: 0-1
- Benchmark: LLM Leaderboard Benchmark
- Evaluation Consistency target: 1
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- Google ADK · Required
- Google · Optional
- Exa · Optional
Evaluation contract
- tier_ranking_test · Weight 1
- evaluation_consistency · Weight 1
Metrics
- evaluation_consistency · Target 1
Benchmarks
- LLM Leaderboard Benchmark
- general
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON with model_name, bleu_score, reasoning_acc, latency_ms.
Output: JSON with overall_score, tier_ranking, production_ready.
Execution harness: Python
Evaluation tools
- Google ADK · Agent Framework
Recipe freeze marker
No recipe freeze time is recorded.