Skip to main content
Back to challenges
Challenge

Clinical Trial Inclusion Criteria Extractor with Claude Agents SDK

Building on STAT news reporting regarding Capricor Therapeutics' trial hurdles, this challenge asks developers to build an inclusion/exclusion eligibility extraction agent. Using the Claude Agents SDK with extended thinking, participants will construct an agent that ingests unstructured clinical trial documents, parses eligibility rules into executable boolean logic expressions, and matches incoming candidate patient records.

Data ScienceHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build a trial matching eligibility extractor using the Claude Agents SDK with extended thinking and structured output validation.

Delivery guide

How work is evaluated

Evaluation

Evaluates patient eligibility matching precision and accuracy.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

check_eligibility_bool

Ensure is_eligible is boolean

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

matching_accuracy

Accuracy of patient eligibility determination against ground truth • target: 0.92 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Parse unstructured clinical trial protocol criteria into formal logic expressions

  • Leverage Claude Agents SDK for step-by-step logic translation

  • Evaluate candidate patient FHIR data against executable trial criteria

  • Generate transparent inclusion/exclusion audit summaries for trial coordinators

Resources and assets

Reference links and supporting material

Dataset notes

Synthetic clinical trial inclusion criteria texts and corresponding patient EHR profiles.

Challenge setup

Evaluates patient eligibility matching precision and accuracy.

Challenge input

JSON containing protocol_criteria_text and patient_record

Claude Agents SDK

Mandatory Claude SDK

Evaluated output

JSON containing is_eligible (boolean) and failed_criteria (list of strings)

Evaluation checks
  • Ensure is_eligible is boolean
  • Accuracy of patient eligibility determination against ground truth • target: 0.92 • range: 0-1
Configured evaluation evidence
  • Benchmark: TrialMatch-Bench
  • Matching Accuracy target: 0.92
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Upstage · Required
  • GPT-Engineer · Optional
  • LastMile AI · Optional

Evaluation contract

  • check_eligibility_bool · Weight 1
  • matching_accuracy · Weight 1

Metrics

  • matching_accuracy · Target 0.92

Benchmarks

  • TrialMatch-Bench
  • biotech

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON containing protocol_criteria_text and patient_record

    Output: JSON containing is_eligible (boolean) and failed_criteria (list of strings)

    Execution harness: Python

Evaluation tools

  • Claude Agents SDK · Agent Framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Clinical Trial Inclusion Criteria Extractor with Claude Agents SDK