Skip to main content
Challenge

Build a Watchlist False Positive Whitelist Decay Auditor

Analyze historically whitelisted screening matches against fresh yente FollowTheMoney entity updates. Flag previously cleared entities that have accumulated new risk identifiers requiring re-adjudication.

CybersecurityHosted by Vera
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Analyze historically whitelisted screening matches against fresh yente FollowTheMoney entity updates. Flag previously cleared entities that have accumulated new risk identifiers requiring re-adjudication.

Senior Sanctions Compliance Officer

Ingest historical screening whitelists containing cleared FollowTheMoney entities and original analyst rationales.

Query a simulated yente service for the latest entity properties and sanction listing history for each whitelisted target.

Compare historical entity profiles against current properties to detect newly added aliases, passports, or risk tags.

Generate a re-adjudication queue prioritizing whitelisted matches with significant material changes in their underlying risk profile.

Browser-based compliance dashboard showing reviewer adjudication diffs, source lineage provenance trees, and dispute escalation triggers.

Use open-source yente FollowTheMoney entity format definitions and generate synthetic screening alert files containing contradictory reviewer dispositions. Starter examples below are synthetic. They are not downloaded records.

OpenSanctions data toolkit Extracts FollowTheMoney entity and match structures defined in the yente open-source data toolkit documentation to model screening hits and watchlist provenance.

llamaindex, pydantic, fastapi. LlamaIndex indexes reviewer adjudication notes, discrepancy criteria, and entity schemas to retrieve relevant precedents during arbitration synthesis. Fast hosted instruction-tuned LLM accessed via API with at least 8k context window for reasoning over multi-analyst narratives.

Submit runnable source, pinned dependencies, setup commands, a working interface, example outputs, and a recorded end-to-end demo. Include tests and an action log.

Successfully retrieves updated entity properties from a simulated yente service.

Identifies material profile changes that invalidate previous false positive rationales.

Outputs a prioritized re-screening queue in JSON format.

```json [ { "input": { "whitelist_id": "WL-992", "target_id": "ftm-per-123", "cleared_date": "2023-01-15", "original_rationale": "Different nationality (Spain vs target's unknown).", "current_yente_profile": { "schema": "Person", "properties": { "nationality": [ "Spain", "Russia" ], "sanctions": [ "Updated 2024-02: Added Spanish dual citizenship" ] } } }, "expected_output": { "whitelist_id": "WL-992", "target_id": "ftm-per-123", "status": "DECAYED_REQUIRES_REVIEW", "material_changes_detected": [ "nationality", "sanctions" ], "reason": "New dual citizenship matches previously cleared discrepancy." } }, { "input": { "whitelist_id": "WL-993", "target_id": "ftm-per-456", "cleared_date": "2023-05-10", "original_rationale": "Distinct date of birth.", "current_yente_profile": { "schema": "Person", "properties": { "birthDate": [ "1980-01-01" ] } } }, "expected_output": { "whitelist_id": "WL-993", "target_id": "ftm-per-456", "status": "VALID", "material_changes_detected": [], "reason": "No material changes to entity profile since clearance." } } ] ```

Processes mock screening decisions only; no live external sanction list updates or financial blocking actions are executed. Use mock external writes. Require human review before any real deployment.

Rules

Participation guide

Processes mock screening decisions only; no live external sanction list updates or financial blocking actions are executed. Use mock external writes and human review.

Delivery guide

How work is evaluated

Evaluation

Successfully retrieves updated entity properties from a simulated yente service. Identifies material profile changes that invalidate previous false positive rationales. Outputs a prioritized re-screening queue in JSON format.

Submission format

Runnable source, pinned dependencies, setup commands, working interface, example outputs, recorded demo, tests, and action logs.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
4 scoring checks
Binary
1 pass or fail dimensions
Ordinal
3 scaled dimensions
Dimension 1

Complete workflow

Ingest historical screening whitelists containing cleared FollowTheMoney entities and original analyst rationales. Query a simulated yente service for the latest entity properties and sanction listing history for each whitelisted target. Compare historical entity profiles against current properties to detect newly added aliases, passports, or risk tags. Generate a re-adjudication queue prioritizing whitelisted matches with significant material changes in their underlying risk profile.

Score scale
012
Score 0

Missing or incorrect.

Score 1

Partly complete, with documented gaps.

Score 2

Complete, with reproducible evidence.

Dimension 2

Correctness and evidence

Successfully retrieves updated entity properties from a simulated yente service. Identifies material profile changes that invalidate previous false positive rationales. Outputs a prioritized re-screening queue in JSON format.

Score scale
012
Score 0

Missing or incorrect.

Score 1

Partly complete, with documented gaps.

Score 2

Complete, with reproducible evidence.

Dimension 3

Review interface

Browser-based compliance dashboard showing reviewer adjudication diffs, source lineage provenance trees, and dispute escalation triggers.

Score scale
012
Score 0

Missing or incorrect.

Score 1

Partly complete, with documented gaps.

Score 2

Complete, with reproducible evidence.

Dimension 4

Safe execution

Processes mock screening decisions only; no live external sanction list updates or financial blocking actions are executed.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Ingest historical screening whitelists containing cleared FollowTheMoney entities and original analyst rationales.

  • Query a simulated yente service for the latest entity properties and sanction listing history for each whitelisted target.

  • Compare historical entity profiles against current properties to detect newly added aliases, passports, or risk tags.

  • Generate a re-adjudication queue prioritizing whitelisted matches with significant material changes in their underlying risk profile.

Resources and assets

Reference links and supporting material

Dataset notes

Use open-source yente FollowTheMoney entity format definitions and generate synthetic screening alert files containing contradictory reviewer dispositions. Supplied starter examples are synthetic, not source extracts.

Challenge setup

Configuration summary for this challenge.

Challenge input

Input details are not configured.

LlamaIndex

Data framework for LLM applications

Evaluated output

No output checks are configured.

Evaluation checks
  • Ingest historical screening whitelists containing cleared FollowTheMoney entities and original analyst rati...
  • Successfully retrieves updated entity properties from a simulated yente service. Identifies material profil...
  • Browser-based compliance dashboard showing reviewer adjudication diffs, source lineage provenance trees, an...
Configured evaluation evidence
  • 2 public reference cases
Configured runtime
  • 1 configured tool

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • LlamaIndex · Optional

Evaluation contract

  • Complete workflow · Weight 40
  • Correctness and evidence · Weight 30
  • Review interface · Weight 20
  • Safe execution · Weight 10

Reference cases

  • 2 public reference cases

Workflow tasks

No workflow tasks are configured.

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Build a Watchlist False Positive Whitelist Decay Auditor