Agentic News Headline Generator with Fact-Checking and Bias Detection
Following Google's experience with AI-generated headlines sometimes being inaccurate or misleading, this challenge focuses on developing a robust agentic system to generate news headlines. The system, built with the Claude Agents SDK, will emphasize factual accuracy, detect potential biases, and ensure relevance to the source article. Your agent will act as an editorial assistant, using Claude Opus 4.1 for sophisticated reasoning and content generation, combined with external tools for fact-checking and validation. This project highlights the critical role of AI governance, evaluation, and responsible AI practices in content generation workflows.
What you are building
The core problem, expected build, and operating context for this challenge.
Develop a Claude Agents SDK agent that generates news headlines with automated fact-checking and bias detection using Claude Opus 4.1 and external validation tools.
How work is evaluated
The evaluation will focus on the factual accuracy, relevance, and absence of bias in the AI-generated headlines, as well as the agent's ability to use tools effectively for validation.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
No factual inaccuracies
Generated headlines must not contain any verifiable factual errors relative to the source article.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Guardrails AI policy enforcement
Generated headlines must adhere to all predefined Guardrails AI policies.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Factual Accuracy Score
Score reflecting the factual correctness of the generated headline (0-1). • target: 0.95 • range: 0.8-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Relevance to Article
Semantic similarity score between headline and article core content (0-1). • target: 0.9 • range: 0.75-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Bias Detection Score (HeyBoss AI)
Measure of how successfully potential biases are detected and mitigated (0-1, lower is better for bias). • target: 0.1 • range: 0-0.3
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master the Claude Agents SDK for constructing multi-turn, tool-using agents, focusing on defining tools, implementing tool handlers, and managing conversational state with Claude Opus 4.1.
Design and implement custom tools for the Claude agent to perform external lookups, such as simulating an API call to a 'fact-checking service' or a 'bias analysis engine'.
Utilize Claude Opus 4.1's advanced reasoning capabilities to analyze source articles, identify key facts, and generate multiple headline options that are concise, accurate, and engaging.
Integrate HeyBoss AI to monitor and evaluate the agent's generated headlines for potential factual errors, tone issues, or unintended biases, providing real-time feedback.
Build an MLOps pipeline using ZenML to orchestrate the entire headline generation and evaluation workflow, including data ingestion, agent execution, and storing evaluation results.
Implement Guardrails AI to enforce strict output policies on generated headlines, ensuring they meet length constraints, avoid specific keywords, and adhere to a desired factual confidence score.
Reference links and supporting material
Sample data for 2 tasks
Challenge setup
The evaluation will focus on the factual accuracy, relevance, and absence of bias in the AI-generated headlines, as well as the agent's ability to use tools effectively for validation.
Challenge input
{ "article_text": "string", "contextual_keywords": ["string"] }
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{ "generated_headline": "string", "fact_check_report": { "score": float, "issues": ["string"] }, "relevance_score": float }
- Generated headlines must not contain any verifiable factual errors relative to the source article.
- Generated headlines must adhere to all predefined Guardrails AI policies.
- Score reflecting the factual correctness of the generated headline (0-1). • target: 0.95 • range: 0.8-1
- Factual Accuracy Score target: 0.95
- 2 public reference cases
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- No factual inaccuracies · Weight 1
- Guardrails AI policy enforcement · Weight 1
- Factual Accuracy Score · Weight 1
- Relevance to Article · Weight 1
- Bias Detection Score (HeyBoss AI) · Weight 1
Metrics
- Factual Accuracy Score · Target 0.95
- Relevance to Article · Target 0.9
- Bias Detection Score (HeyBoss AI) · Target 0.1
Reference cases
- 2 public reference cases
Workflow tasks
- Task 1
Input: { "article_text": "string", "contextual_keywords": ["string"] }
Output: { "generated_headline": "string", "fact_check_report": { "score": float, "issues": ["string"] }, "relevance_score": float }
Execution harness: Python
- Task 2
Input: { "article_text": "string", "headline_policies": { "max_length": int, "forbidden_words": ["string"] } }
Output: { "generated_headline": "string", "bias_analysis": { "score": float, "indicators": ["string"] }, "policy_violations": ["string"] }
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.