Public evaluation

FactualVerification

The evaluation will assess the CrewAI system's ability to accurately identify factual errors, logical inconsistencies, and 'AI slop' characteristics in generated scientific text, providing clear justifications.

Evaluation type

task based

Challenge

Orchestrate Scientific Integrity Agent Crew

Difficulty

Advanced

Rigor

Unspecified

Open linked challenge

Evaluation overview

How the linked challenge is judged: tasks, benchmarks, and criteria count.

Tasks

Benchmarks

Criteria

Task templates

Inputs and expected outputs.

Task 1

FactualVerification

Evaluate if the agent crew correctly identifies and flags factual inaccuracies in a provided abstract.

Input format

{'abstract': '...', 'known_errors': [{'phrase': '...', 'reason': '...'}]}

Output format

{'flagged_issues': [{'phrase': '...', 'finding': '...', 'justification': '...'}], 'overall_verdict': 'accurate|inaccurate'}

Task 2

AISlopDetection

Assess the agent crew's ability to identify stylistic patterns or vague language indicative of AI generation.

Input format

{'abstract': '... (potentially AI-generated)', 'slop_indicators': ['vague language', 'repetitive phrasing']}

Output format

{'slop_flags': [{'indicator': '...', 'justification': '...'}], 'ai_likelihood': 'low|medium|high'}