Skip to main content
Back to challenges
Challenge

Public Health Misinformation Triage Agent with LangChain and Instructor

Public health communications require structured extraction of scientific claims and empathetic counter-messaging. You will build a LangChain agent using Instructor to validate public health claims against medical literature, outputting structured JSON with empathy scores and accuracy metrics achieving 90% validation accuracy.

Data ScienceHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Implement a LangChain graph workflow using Instructor for structured output validation against scientific literature databases.

Delivery guide

How work is evaluated

Evaluation

Evaluates structured JSON extraction accuracy and claim validation performance against gold standard public health claims.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

valid_json_schema

Output must match expected JSON schema exactly

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

validation_accuracy

Accuracy of claim factual classification • target: 0.9 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Build structured LLM outputs using Instructor and Pydantic schema validation

  • Construct multi-step claim verification chains using LangChain and LangGraph

  • Evaluate empathetic messaging tone alongside scientific factual accuracy

Resources and assets

Reference links and supporting material

Dataset notes

Dataset of 500 vaccine-related public health claims with annotated factual correctness and empathy ratings.

Challenge setup

Evaluates structured JSON extraction accuracy and claim validation performance against gold standard public health claims.

Challenge input

JSON object with claim_text string

LangChain

Required framework for building LLM stateful chains.

Instructor

Enforces strict Pydantic JSON outputs from LLM responses.

Evaluated output

JSON object with is_factual boolean, confidence float, and citations array

Evaluation checks
  • Output must match expected JSON schema exactly
  • Accuracy of claim factual classification • target: 0.9 • range: 0-1
Configured evaluation evidence
  • Benchmark: HealthFact-Eval
  • Validation Accuracy target: 0.9
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Instructor · Required
  • Langchain · Optional
Policy Serving
  • Pangu 3.0 · Optional
Orchestration
  • Langchain · Optional

Evaluation contract

  • valid_json_schema · Weight 1
  • validation_accuracy · Weight 1

Metrics

  • validation_accuracy · Target 0.9

Benchmarks

  • HealthFact-Eval
  • biotech

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON object with claim_text string

    Output: JSON object with is_factual boolean, confidence float, and citations array

    Execution harness: Python

Evaluation tools

  • LangChain · Agent Orchestration Framework
  • Instructor · Structured Extraction Library

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Public Health Misinformation Triage Agent with LangChain and Instructor