Challenge

Editorial Compliance & Content Neutrality System Agent

Build a sophisticated multi-agent system using LangChain with LangGraph for orchestrating an autonomous content review process focused on editorial compliance and neutrality. Inspired by recent disputes over media independence and the need for rigorous content vetting, this system will analyze articles, reports, or social media content for bias, factual inconsistencies, and adherence to predefined editorial guidelines. The LangGraph framework will define a stateful workflow, where specialized agents (e.g., 'Content Ingester', 'Bias Analyzer', 'Fact Checker', 'Compliance Reporter') collaborate and pass information dynamically. GPT-5 Pro will serve as the primary reasoning engine for nuanced content analysis and complex policy interpretation, leveraging its advanced understanding of context and subtle language cues. Gemini 3 Flash will be leveraged for rapid factual lookups against external knowledge bases or for generating alternative, neutral phrasing of contentious statements. AI21 Studio will be utilized for deploying and managing GPT-5 Pro inference, while Together AI will handle the serving of Gemini 3 Flash, ensuring high availability, optimized performance, and cost-efficiency for real-time content assessment workflows.

Workflow AutomationHosted by Vera
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build a sophisticated multi-agent system using LangChain with LangGraph for orchestrating an autonomous content review process focused on editorial compliance and neutrality. Inspired by recent disputes over media independence and the need for rigorous content vetting, this system will analyze articles, reports, or social media content for bias, factual inconsistencies, and adherence to predefined editorial guidelines. The LangGraph framework will define a stateful workflow, where specialized agents (e.g., 'Content Ingester', 'Bias Analyzer', 'Fact Checker', 'Compliance Reporter') collaborate and pass information dynamically. GPT-5 Pro will serve as the primary reasoning engine for nuanced content analysis and complex policy interpretation, leveraging its advanced understanding of context and subtle language cues. Gemini 3 Flash will be leveraged for rapid factual lookups against external knowledge bases or for generating alternative, neutral phrasing of contentious statements. AI21 Studio will be utilized for deploying and managing GPT-5 Pro inference, while Together AI will handle the serving of Gemini 3 Flash, ensuring high availability, optimized performance, and cost-efficiency for real-time content assessment workflows.

Datasets

Shared data for this challenge

Review public datasets and any private uploads tied to your build.

Loading datasets...
Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
6 scoring checks
Binary
6 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

CorrectBiasTypeDetection

The system correctly identifies the primary type of bias present in the article, matching the 'ground_truth_bias_type'.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

AllFactualErrorsIdentified

The system identifies all critical factual errors listed in 'ground_truth_factual_errors' within the input.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 3

GuidelineComplianceCheck

The 'compliance_score' accurately reflects the degree of adherence to the provided 'editorial_guidelines'.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 4

Bias Detection Accuracy

The F1-score for identifying specific bias types compared to ground truth. • target: 0.9 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 5

Factual Error Recall

The percentage of 'ground_truth_factual_errors' that are correctly identified by the system. • target: 0.85 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 6

Neutral Phrasing Quality Score

A subjective score (1-5) for the quality, neutrality, and grammatical correctness of 'suggested_neutral_phrasing'. • target: 4 • range: 1-5

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Master LangChain's LangGraph for designing and implementing complex, stateful multi-agent workflows with conditional routing and persistent state management.

  • Orchestrate advanced reasoning pipelines by integrating multiple cutting-edge LLMs, specifically using GPT-5 Pro for in-depth analysis and Gemini 3 Flash for rapid fact-checking.

  • Implement robust content analysis algorithms within agents to detect partisan bias, verify factual accuracy, and ensure strict adherence to predefined editorial compliance guidelines.

  • Leverage AI21 Studio and Together AI for efficient and scalable deployment and inference management of large language models, optimizing for latency and throughput in real-time content assessment.

  • Design agent-to-agent communication protocols and data exchange formats within LangGraph for seamless task handoffs and collaborative problem-solving among specialized agents.

  • Develop custom tools for agents to interact with external databases, public knowledge graphs, or fact-checking APIs to augment their analytical capabilities.

  • Implement comprehensive logging and observability within the LangGraph workflow to trace agent decisions, model outputs, and compliance outcomes for auditability.

How this agent runs

Evaluate the LangGraph system's ability to accurately identify bias, factual errors, and compliance violations in provided text content, and its efficiency in processing complex articles.

Preview configuration

Challenge input

JSON object with 'article_text' (string, full content of the article), 'ground_truth_bias_type' (string, e.g., 'negative_framing', 'omission_of_fac...

LangChain

Framework for building LLM applications

Google Gemini

Google's multimodal AI model

Evaluated output

JSON object with 'detected_bias_type' (string), 'bias_reasoning' (string), 'identified_factual_errors' (array of {statement, detected_error_explana...

Checks for
  • The system correctly identifies the primary type of bias present in the article, matching the 'ground_truth...
  • The system identifies all critical factual errors listed in 'ground_truth_factual_errors' within the input.
  • The 'compliance_score' accurately reflects the degree of adherence to the provided 'editorial_guidelines'.
Proof of success
  • Bias Detection Accuracy target: 0.9
  • 1 public reference case
Runtime evidence
  • Python execution harness
View technical recipe

Configured tools

Action Space
  • LangChain · Required
  • Langchain · Optional
  • Google Gemini · Optional
Orchestration
  • LangChain · Required
  • Langchain · Optional

Evaluation contract

  • CorrectBiasTypeDetection · Weight 1
  • AllFactualErrorsIdentified · Weight 1
  • GuidelineComplianceCheck · Weight 1
  • Bias Detection Accuracy · Weight 1
  • Factual Error Recall · Weight 1
  • Neutral Phrasing Quality Score · Weight 1

Recipe state

This is a preview. The configuration can change before the evaluation recipe is locked.

Run this agent on your dataset

Versalist can run this agent on your behalf with your data. Tell us about your dataset and the result you need.

Discuss your dataset
Start from your terminal
$npx -y @versalist/cli start editorial-compliance-content-neutrality-system-agent

[ok] Wrote CHALLENGE.md

[ok] Wrote .versalist.json

[ok] Wrote eval/examples.json

Requires VERSALIST_API_KEY. Works with any MCP-aware editor.

Docs
Manage API keys
Explore

Find another challenge

Jump to a random challenge when you want a fresh benchmark or a different problem space.

Useful when you want to pressure-test your workflow on a new dataset, new constraints, or a new evaluation rubric.

Frequently Asked Questions about Editorial Compliance & Content Neutrality System Agent