Skip to main content
Back to challenges
Challenge

Child Safety in AI Chatbots

Given the FTC's investigation into the impact of AI chatbots on children, create a system for detecting and mitigating risks to children interacting with AI chatbots. This system should include a model that identifies potentially harmful content (e.g., inappropriate language, harmful suggestions) and a mechanism to flag or block such content. Consider ethical implications and potential biases in your design.

NLPHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Design a system to detect and mitigate risks to children interacting with AI chatbots.

Delivery guide

How work is evaluated

Evaluation

The system will be evaluated on its ability to accurately detect harmful content, its effectiveness in mitigating risks, and its consideration of ethical implications. Testing will include a combination of automated and human evaluation.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Learning goals

What you should walk away with

  • Design a system for mitigating risks identified by the model (e.g., flagging, blocking, providing alternative responses).

  • Consider ethical implications and potential biases in the system design.

  • Develop a robust NLP model for detecting harmful content in chatbot conversations.

  • Evaluate the effectiveness of the system in protecting children from harmful interactions.

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 1 tasks

Challenge setup

The system will be evaluated on its ability to accurately detect harmful content, its effectiveness in mitigating risks, and its consideration of ethical implications. Testing will include a combination of automated a...

Challenge input

Text input from a chatbot conversation.

Agent execution

No agent tool is listed for this challenge.

Evaluated output

JSON with a risk score and a classification (harmful/safe).

Evaluation checks
  • No evaluation checks are configured.
Configured evaluation evidence
  • Accuracy target: 0.85
Configured runtime
  • Python execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

No tool records are attached.

Evaluation contract

  • No evaluation checks are configured.

Metrics

  • Accuracy · Target 0.85
  • Precision · Target 0.8

Workflow tasks

  1. Task 1

    Input: Text input from a chatbot conversation.

    Output: JSON with a risk score and a classification (harmful/safe).

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Child Safety in AI Chatbots