Skip to main content
Back to challenges
Challenge

AI-Powered Misinformation Detection for Children

Develop an AI system to detect and flag potentially harmful content in online chatbots and social media aimed at children. The system should analyze text, images, and potentially audio to identify grooming attempts, misinformation, or content promoting self-harm. The challenge focuses on creating a robust, explainable AI model that balances sensitivity with accuracy, minimizing false positives while ensuring high recall for genuinely harmful content. This is crucial given the recent warning from US AGs about the responsibility of tech companies in protecting children online.

Special Purpose AgentsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Create an AI system to detect harmful content in online platforms targeting children.

Delivery guide

How work is evaluated

Evaluation

Evaluation will be based on accuracy, precision, recall, F1-score, and the explainability of the model's predictions.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Learning goals

What you should walk away with

  • Design a system adaptable to different platforms and content types.

  • Build a multi-modal AI model for harmful content detection.

  • Develop an explainable AI system to understand model decisions.

  • Minimize false positives to avoid unnecessary censorship.

  • Achieve high accuracy and recall in identifying harmful content.

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 1 tasks

Challenge setup

Evaluation will be based on accuracy, precision, recall, F1-score, and the explainability of the model's predictions.

Challenge input

JSON: {'text': '...', 'image': '...', 'audio': '...'}

Agent execution

No agent tool is listed for this challenge.

Evaluated output

JSON: {'prediction': 'harmful'/'not harmful', 'explanation': '...'}

Evaluation checks
  • No evaluation checks are configured.
Configured evaluation evidence
  • Accuracy target: 0.9
Configured runtime
  • Docker execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

No tool records are attached.

Evaluation contract

  • No evaluation checks are configured.

Metrics

  • Accuracy · Target 0.9
  • Precision · Target 0.85
  • Recall · Target 0.85
  • F1-Score · Target 0.85

Workflow tasks

  1. Task 1

    Input: JSON: {'text': '...', 'image': '...', 'audio': '...'}

    Output: JSON: {'prediction': 'harmful'/'not harmful', 'explanation': '...'}

    Execution harness: Docker

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about AI-Powered Misinformation Detection for Children