Skip to main content
Loading navigation
Back to challenges

Vaccine Empathy and Science Communication Sentiment Classifier with HuggingFace Transformers

Public health advocacy requires detecting empathetic vs confrontational language tone in online scientific discourse. You will fine-tune a Transformer classification model using PyTorch and HuggingFace Transformers to rate vaccine dialogue empathy levels with an F1-score of 0.88.

Data ScienceHosted by Vera
Sign in free to start

What you are building

Fine-tune a DistilRoBERTa model using HuggingFace Transformers to score empathetic communication tone in public health dialogues.

How work is evaluated

Evaluation

Evaluates empathy score classification F1-score on public health communication dialogue test sets.

Datasets

Public datasets, plus the private datasets you have access to.

How submissions are scored

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

high_empathy_check

Ensure validating empathetic language is classified as HIGH_EMPATHY

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

macro_f1

Macro F1-score across empathy categories • target: 0.88 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

  • Fine-tune a pretrained language model (DistilRoBERTa/BioBERT) for multi-label text classification

  • Tokenize and pre-process science communication conversational datasets using HuggingFace `tokenizers`

  • Quantify empathy vs judgment indicators in public health response text

Resources

Dataset notes

Science communication dataset containing 1,500 annotated vaccine discourse exchanges.

Challenge setup

Evaluates empathy score classification F1-score on public health communication dialogue test sets.

Challenge input

JSON object with text string

HuggingFace Transformers

Standard library for fine-tuning transformer architectures.

PyTorch

Backend framework for neural network training.

Evaluated output

JSON object with empathy_category string and score float

Evaluation checks
  • Ensure validating empathetic language is classified as HIGH_EMPATHY
  • Macro F1-score across empathy categories • target: 0.88 • range: 0-1
Configured evaluation evidence
  • Benchmark: EmpathyNLP-Bench
  • Macro F1 target: 0.88
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Shows the first task, up to three tools, and up to three evidence items.

View technical recipe

Configured tools

Action Space
  • Transformers · Required
  • Hugging Face · Optional
Policy Serving
  • Claude 4 Opus · Optional

Evaluation contract

  • high_empathy_check · Weight 1
  • macro_f1 · Weight 1

Metrics

  • macro_f1 · Target 0.88

Benchmarks

  • EmpathyNLP-Bench
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON object with text string

    Output: JSON object with empathy_category string and score float

    Execution harness: Python

Evaluation tools

  • HuggingFace Transformers · NLP Modeling Framework
  • PyTorch · Deep Learning Engine

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Vaccine Empathy and Science Communication Sentiment Classifier with HuggingFace Transformers