Back to evaluations
Public evaluation

FactVerification

The evaluation module will assess the engine's ability to accurately benchmark LLM outputs against known datasets, detect deliberate inconsistencies or 'fudging', and provide a clear, auditable report.

Evaluation type
task based
Challenge
Hybrid Reasoning AI Evaluation Engine
Difficulty
Advanced
Rigor
Unspecified
Open linked challenge

Evaluation overview

How the linked challenge is judged: tasks, benchmarks, and criteria count.

Tasks
2
Benchmarks
0
Criteria
0

Task templates

Inputs and expected outputs.

Task 1

FactVerification

Evaluates the engine's ability to verify factual claims made by an LLM against a ground truth dataset.

Input format

{ 'llm_output': 'string', 'question': 'string', 'ground_truth_url': 'string' }

Output format

{ 'is_factual': 'boolean', 'reasoning': 'string', 'confidence_score': 'float' }

Task 2

BenchmarkConsistencyCheck

Assesses if the engine can detect discrepancies in LLM performance across different benchmark subsets, simulating 'fudged' results.

Input format

{ 'model_name': 'string', 'benchmark_results': { 'subset_A': 'float', 'subset_B': 'float' }, 'expected_deviation': 'float' }

Output format

{ 'consistent': 'boolean', 'deviation_detected': 'float', 'analysis': 'string', 'hybrid_reasoning_trace': 'array' }