Back to challenges
Challenge

Vaccine Safety Literature Synthesis Agent with Claude Agents SDK

Federal advisory committees need automated synthesis of public vaccine safety research and funding strategic plans. Developers will implement a Claude Agents SDK system with extended thinking to parse epidemiological literature, evaluate vaccine safety claims, and construct transparent policy strategy summaries with verifiable source citations.

Data ScienceHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build an extended-thinking agent with Claude Agents SDK to synthesize vaccine safety literature and support federal policy planning.

Delivery guide

How work is evaluated

Evaluation

Evaluates claim verification accuracy, citation correctness, and reasoning trace clarity.

Datasets

Shared data for this challenge

Review public datasets and any private uploads tied to your build.

Loading datasets...
Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

citation_verification_test

Verifies that retrieved PMIDs match the input documents.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

factual_grounding_score

Score measuring absence of hallucinations relative to input abstracts. • target: 0.98 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Initialize a Claude Agents SDK project with extended thinking enabled.

  • Extract scientific consensus findings from epidemiological research literature.

  • Audit public claims against empirical vaccine safety evidence databases.

  • Generate structured strategic research recommendations for advisory committees.

Resources and assets

Reference links and supporting material

Dataset notes

Curated collection of peer-reviewed epidemiological study abstracts on vaccine safety and advisory committee transcripts.

How this agent runs

Evaluates claim verification accuracy, citation correctness, and reasoning trace clarity.

Challenge input

JSON containing topic query and list of study abstract documents.

Claude Agents SDK

Offers native tool use and deep extended thinking reasoning modes.

Evaluated output

JSON containing consensus conclusion, evidence quality grade, and citations.

Checks for
  • Verifies that retrieved PMIDs match the input documents.
  • Score measuring absence of hallucinations relative to input abstracts. • target: 0.98 • range: 0-1
Proof of success
  • Benchmark: Biomedical Literature QA Benchmark
  • Factual Grounding Score target: 0.98
  • 1 public reference case
Runtime evidence
  • Python execution harness
  • Python sandbox (unavailable on Versalist)
View technical recipe

Configured tools

Action Space
  • Anthropic Claude · Required
  • Anthropic · Optional
  • Warp · Optional

Evaluation contract

  • citation_verification_test · Weight 1
  • factual_grounding_score · Weight 1

Recipe state

This is a preview. The configuration can change before the evaluation recipe is locked.

Frequently Asked Questions about Vaccine Safety Literature Synthesis Agent with Claude Agents SDK