Neutrality Score for Bias Detection & Fact-Checking
Inspired by discussions around content neutrality, this challenge focuses on building an advanced AI agent system capable of analyzing text for bias, factual inaccuracies, and neutrality standards. You will use LangGraph to design a Directed Acyclic Graph (DAG) workflow, orchestrating several specialized agents. Gemini 2.5 Pro (leveraging its Deep Think mode for nuanced analysis) will be central for identifying subtle biases and performing robust factual verification. OpenAI GPT 5 will provide alternative phrasing or counter-arguments to assess different perspectives. The system must implement the A2A (Agent-to-Agent) Protocol for seamless and secure communication between agents during cross-verification processes, ensuring claims are independently assessed. Hybrid instant/deep reasoning will allow agents to quickly triage simple facts while engaging in thorough, multi-step analysis for complex or contentious statements. The output should include a neutrality score and suggested revisions.
What you are building
Develop a LangGraph-based multi-agent system using Gemini 2.5 Pro and OpenAI GPT 5 for real-time content neutrality analysis, bias detection, and fact-checking, featuring A2A protocol.
How work is evaluated
The evaluation will focus on the system's ability to accurately detect bias, verify facts, generate meaningful neutrality scores, and propose effective revisions. Agent collaboration via A2A and the use of hybrid reasoning will also be assessed.
Datasets
Public datasets, plus the private datasets you have access to.
Learning goals
Orchestrate hybrid instant/deep reasoning for varied analysis depths.
Design a DAG-based agent workflow using LangGraph for content analysis.
Implement A2A Protocol for secure agent-to-agent cross-verification.
Integrate Gemini 2.5 Pro Deep Think mode for advanced bias and factual detection.
Develop RAG capabilities for referencing reliable external sources.
Resources
Sample data for 1 tasks
Challenge setup
The evaluation will focus on the system's ability to accurately detect bias, verify facts, generate meaningful neutrality scores, and propose effective revisions. Agent collaboration via A2A and the use of hybrid reas...
Challenge input
{ "text_passage": "string", "context_sources": [{ "title": "string", "url": "string", "content": "string" }] }
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{ "overall_neutrality_score": "float (0.0 - 1.0)", "biased_statements_detected": [{"statement": "string", "bias_type": "string", "reasoning": "stri...
- Verifies that the output is valid JSON and matches the specified schema.
- Confirms that the 'agent_collaboration_log' is populated, indicating successful agent-to-agent communication.
- Bias Detection Accuracy target: 4
- Python execution harness
Shows the first task, up to three tools, and up to three evidence items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Verifies that the output is valid JSON and matches the specified schema.
- Confirms that the 'agent_collaboration_log' is populated, indicating successful agent-to-agent communication.
Metrics
- Bias Detection Accuracy · Target 4
- Factual Verification Precision · Target 4
- Quality of Revisions · Target 4
Workflow tasks
- Task 1
Input: { "text_passage": "string", "context_sources": [{ "title": "string", "url": "string", "content": "string" }] }
Output: { "overall_neutrality_score": "float (0.0 - 1.0)", "biased_statements_detected": [{"statement": "string", "bias_type": "string", "reasoning": "string", "suggested_revision": "string"}], "factual_errors_detected": [{"statement": "string", "error_details": "string", "corrected_info": "string", "verified_source": "string"}], "agent_collaboration_log": [{"agent_id": "string", "action": "string", "message_to_agent": "string", "response_from_agent": "string"}] }
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.