Skip to main content
Back to challenges
Challenge

Vaccine Empathy and Science Communication Sentiment Classifier with HuggingFace Transformers

Public health advocacy requires detecting empathetic vs confrontational language tone in online scientific discourse. You will fine-tune a Transformer classification model using PyTorch and HuggingFace Transformers to rate vaccine dialogue empathy levels with an F1-score of 0.88.

Data ScienceHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Fine-tune a DistilRoBERTa model using HuggingFace Transformers to score empathetic communication tone in public health dialogues.

Delivery guide

How work is evaluated

Evaluation

Evaluates empathy score classification F1-score on public health communication dialogue test sets.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

high_empathy_check

Ensure validating empathetic language is classified as HIGH_EMPATHY

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

macro_f1

Macro F1-score across empathy categories • target: 0.88 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Fine-tune a pretrained language model (DistilRoBERTa/BioBERT) for multi-label text classification

  • Tokenize and pre-process science communication conversational datasets using HuggingFace `tokenizers`

  • Quantify empathy vs judgment indicators in public health response text

Resources and assets

Reference links and supporting material

Dataset notes

Science communication dataset containing 1,500 annotated vaccine discourse exchanges.

Challenge setup

Evaluates empathy score classification F1-score on public health communication dialogue test sets.

Challenge input

JSON object with text string

HuggingFace Transformers

Standard library for fine-tuning transformer architectures.

PyTorch

Backend framework for neural network training.

Evaluated output

JSON object with empathy_category string and score float

Evaluation checks
  • Ensure validating empathetic language is classified as HIGH_EMPATHY
  • Macro F1-score across empathy categories • target: 0.88 • range: 0-1
Configured evaluation evidence
  • Benchmark: EmpathyNLP-Bench
  • Macro F1 target: 0.88
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Transformers · Required
  • Hugging Face · Optional
Policy Serving
  • Claude 4 Opus · Optional

Evaluation contract

  • high_empathy_check · Weight 1
  • macro_f1 · Weight 1

Metrics

  • macro_f1 · Target 0.88

Benchmarks

  • EmpathyNLP-Bench
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON object with text string

    Output: JSON object with empathy_category string and score float

    Execution harness: Python

Evaluation tools

  • HuggingFace Transformers · NLP Modeling Framework
  • PyTorch · Deep Learning Engine

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Vaccine Empathy and Science Communication Sentiment Classifier with HuggingFace Transformers