AI Policy Argument Generation Agent
Develop an advanced AI agent system using the Claude Agents SDK to assist in complex policy negotiations, drawing inspiration from the SAG-AFTRA talks regarding an 'AI tax' on synthetic actors. This challenge requires building a system capable of analyzing diverse policy documents, legal texts, and economic data to generate well-reasoned arguments and counter-arguments for specific stakeholders, such as labor unions and production studios. The core of the solution will be a multi-agent workflow orchestrated by the Claude Agents SDK, leveraging Claude 3.5 Sonnet for its robust reasoning and document understanding capabilities. The agents will be tasked with identifying key points of contention, forecasting potential impacts of proposed policies, and synthesizing persuasive rhetoric. Evaluation of the generated arguments will be conducted using Gentrace, focusing on logical coherence, factual accuracy, and persuasive strength. The system will rely on Azure Blob Storage and Azure Cognitive Search for efficient storage and retrieval of relevant policy documents and background information.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an AI agent system with Claude Agents SDK to generate policy arguments for negotiations, utilizing Claude 3.5 Sonnet, Gentrace for evaluation, and Azure for data management.
How work is evaluated
Evaluation will focus on the quality, factual accuracy, logical coherence, and persuasive strength of the arguments generated by the agent system for specified policy scenarios. Gentrace will be used to track and score these metrics.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
FactualAccuracyCheck
Verifies that generated arguments do not contain factual inaccuracies based on provided documents.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
StakeholderAlignment
Ensures arguments are consistently aligned with the specified stakeholder's interests.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
LogicalCoherence_Score
A score (1-5) representing the logical flow and consistency of the argument. • target: 4 • range: 1-5
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Persuasiveness_Rating
A human-rated score (1-5) for how persuasive the argument is. • target: 4 • range: 1-5
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Completeness_of_Coverage
Percentage of relevant points from documents covered by the argument. • target: 0.8 • range: 0.5-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master the Claude Agents SDK for constructing sophisticated multi-step agentic workflows, incorporating tool use and computer interaction capabilities.
Implement advanced document understanding techniques with Claude 3.5 Sonnet to parse complex legal, contractual, and policy texts, extracting key clauses and implications.
Design a robust data pipeline utilizing Azure Blob Storage for raw document storage and Azure Cognitive Search for semantic retrieval of relevant information for agents.
Develop strategies for generating balanced and persuasive arguments, including identifying pros and cons, potential impacts, and counter-arguments for different stakeholder perspectives.
Integrate Gentrace into the agent workflow to evaluate the logical coherence, factual grounding, and rhetorical effectiveness of generated policy arguments.
Build tool-use capabilities within Claude Agents SDK to interact with external databases, search engines, and custom analytics functions for comprehensive data gathering.
Reference links and supporting material
Sample data for 2 tasks
Challenge setup
Evaluation will focus on the quality, factual accuracy, logical coherence, and persuasive strength of the arguments generated by the agent system for specified policy scenarios. Gentrace will be used to track and scor...
Challenge input
{"stakeholder": "SAG-AFTRA", "topic": "AI Tax on Synthetic Actors", "documents": [{"title": "string", "content": "string"}]}
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{"argument_text": "string", "key_points": ["string"], "potential_impacts": ["string"]}
- Verifies that generated arguments do not contain factual inaccuracies based on provided documents.
- Ensures arguments are consistently aligned with the specified stakeholder's interests.
- A score (1-5) representing the logical flow and consistency of the argument. • target: 4 • range: 1-5
- LogicalCoherence Score target: 4
- 2 public reference cases
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- FactualAccuracyCheck · Weight 1
- StakeholderAlignment · Weight 1
- LogicalCoherence_Score · Weight 1
- Persuasiveness_Rating · Weight 1
- Completeness_of_Coverage · Weight 1
Metrics
- LogicalCoherence_Score · Target 4
- Persuasiveness_Rating · Target 4
- Completeness_of_Coverage · Target 0.8
Reference cases
- 2 public reference cases
Workflow tasks
- Task 1
Input: {"stakeholder": "SAG-AFTRA", "topic": "AI Tax on Synthetic Actors", "documents": [{"title": "string", "content": "string"}]}
Output: {"argument_text": "string", "key_points": ["string"], "potential_impacts": ["string"]}
Execution harness: Python
- Task 2
Input: {"stakeholder": "Studio Executives", "topic": "AI Tax on Synthetic Actors", "documents": [{"title": "string", "content": "string"}]}
Output: {"argument_text": "string", "key_points": ["string"], "potential_impacts": ["string"]}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.