Hybrid Reasoning AI Evaluation Engine
This challenge tasks developers with building a transparent and robust AI evaluation engine. This system will rigorously benchmark and verify LLM outputs to ensure integrity and prevent misleading performance claims. It will employ a hybrid reasoning approach, combining instant checks with deep analytical dives, and leverage MCP-enabled tool integration to access benchmark datasets securely. Participants will utilize DSPy for programmatic optimization of evaluation pipelines, LMDeploy for efficiently serving and swapping multiple models (e.g., Llama variants, OpenAI 5.2), and Gemini 3 Pro for its advanced deep reasoning capabilities. The goal is to create an auditable evaluation framework that can detect subtle inconsistencies and biases in model performance.
What you are building
Develop a robust AI evaluation engine leveraging DSPy for programmatic optimization, LMDeploy for efficient model serving, and Gemini 3 Pro for hybrid instant/deep reasoning.
How work is evaluated
The evaluation module will assess the engine's ability to accurately benchmark LLM outputs against known datasets, detect deliberate inconsistencies or 'fudging', and provide a clear, auditable report.
Datasets
Public datasets, plus the private datasets you have access to.
Learning goals
Design MCP-enabled tool integration for benchmark dataset access.
Utilize LMDeploy for dynamic serving and comparison of multiple LLMs.
Build RAG pipelines for factual verification of evaluation criteria.
Implement hybrid instant/deep reasoning modes for comprehensive model assessment.
Master DSPy for programmatically optimizing LLM evaluation pipelines.
Resources
Sample data for 2 tasks
Challenge setup
The evaluation module will assess the engine's ability to accurately benchmark LLM outputs against known datasets, detect deliberate inconsistencies or 'fudging', and provide a clear, auditable report.
Challenge input
{ 'llm_output': 'string', 'question': 'string', 'ground_truth_url': 'string' }
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{ 'is_factual': 'boolean', 'reasoning': 'string', 'confidence_score': 'float' }
- At least 90% of factual claims must be correctly verified against ground truth.
- The system must correctly identify discrepancies larger than the 'expected_deviation' in benchmark consiste...
- DeepReasoningActivationRate target: 75%
- Python execution harness
Shows the first task, up to three tools, and up to three evidence items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- At least 90% of factual claims must be correctly verified against ground truth.
- The system must correctly identify discrepancies larger than the 'expected_deviation' in benchmark consistency checks.
Metrics
- DeepReasoningActivationRate · Target 0.75
- EvaluationThroughput · Target 50
- ConfidenceScoreAccuracy · Target 0.85
Workflow tasks
- Task 1
Input: { 'llm_output': 'string', 'question': 'string', 'ground_truth_url': 'string' }
Output: { 'is_factual': 'boolean', 'reasoning': 'string', 'confidence_score': 'float' }
Execution harness: Python
- Task 2
Input: { 'model_name': 'string', 'benchmark_results': { 'subset_A': 'float', 'subset_B': 'float' }, 'expected_deviation': 'float' }
Output: { 'consistent': 'boolean', 'deviation_detected': 'float', 'analysis': 'string', 'hybrid_reasoning_trace': 'array' }
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.