Hybrid Reasoning AI Evaluation Engine
This challenge tasks developers with building a transparent and robust AI evaluation engine. This system will rigorously benchmark and verify LLM outputs to ensure integrity and prevent misleading performance claims. It will employ a hybrid reasoning approach, combining instant checks with deep analytical dives, and leverage MCP-enabled tool integration to access benchmark datasets securely. Participants will utilize DSPy for programmatic optimization of evaluation pipelines, LMDeploy for efficiently serving and swapping multiple models (e.g., Llama variants, OpenAI 5.2), and Gemini 3 Pro for its advanced deep reasoning capabilities. The goal is to create an auditable evaluation framework that can detect subtle inconsistencies and biases in model performance.
What you are building
The core problem, expected build, and operating context for this challenge.
Develop a robust AI evaluation engine leveraging DSPy for programmatic optimization, LMDeploy for efficient model serving, and Gemini 3 Pro for hybrid instant/deep reasoning.
How work is evaluated
The evaluation module will assess the engine's ability to accurately benchmark LLM outputs against known datasets, detect deliberate inconsistencies or 'fudging', and provide a clear, auditable report.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
What you should walk away with
Design MCP-enabled tool integration for benchmark dataset access.
Utilize LMDeploy for dynamic serving and comparison of multiple LLMs.
Build RAG pipelines for factual verification of evaluation criteria.
Implement hybrid instant/deep reasoning modes for comprehensive model assessment.
Master DSPy for programmatically optimizing LLM evaluation pipelines.
Reference links and supporting material
Sample data for 2 tasks
How this agent runs
The evaluation module will assess the engine's ability to accurately benchmark LLM outputs against known datasets, detect deliberate inconsistencies or 'fudging', and provide a clear, auditable report.
Challenge input
{ 'llm_output': 'string', 'question': 'string', 'ground_truth_url': 'string' }
Agent execution
The configured agent processes the input under the challenge policy.
Evaluated output
{ 'is_factual': 'boolean', 'reasoning': 'string', 'confidence_score': 'float' }
- At least 90% of factual claims must be correctly verified against ground truth.
- The system must correctly identify discrepancies larger than the 'expected_deviation' in benchmark consiste...
- DeepReasoningActivationRate target: 75%
- Python execution harness
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- The evaluation module defines the checks.
Recipe state
This is a preview. The configuration can change before the evaluation recipe is locked.