Skip to main content
Back to challenges
Challenge

BIS Regulatory Compliance Knowledge Graph with LlamaIndex

Central bank digital currency and cross-border innovation projects require precise indexing of evolving international supervisory guidance and technical standards. You will build a graph RAG pipeline that indexes BIS regulatory frameworks and answers complex regulatory query suites with full attribution and high extraction precision.

CybersecurityHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build an enterprise Graph RAG compliance engine using LlamaIndex to query complex BIS Innovation Hub regulatory publications with full source attribution.

Delivery guide

How work is evaluated

Evaluation

Evaluates precision and source citation fidelity of answers generated from BIS regulatory publication corpus.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

has_valid_citations

Ensures generated response includes explicit source file citations

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

retrieval_precision

Precision of retrieved graph triplets for query relevance • target: 0.96 • range: 0.88-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Construct Knowledge Graph indexes using LlamaIndex (`KnowledgeGraphIndex`)

  • Extract structured entities and relations from complex central bank regulatory publications

  • Implement hybrid vector and knowledge graph retriever pipelines

  • Achieve accurate regulatory query answering with verifiable source attribution links

Resources and assets

Reference links and supporting material

Dataset notes

Corpus of 50 official BIS Innovation Hub project publications and cross-border payment regulatory reports.

Challenge setup

Evaluates precision and source citation fidelity of answers generated from BIS regulatory publication corpus.

Challenge input

JSON containing natural language regulatory question

LlamaIndex

Enterprise data framework providing Knowledge Graph Index and Graph RAG capabilities.

Evaluated output

JSON containing answer, cited_documents, and extracted_triplets

Evaluation checks
  • Ensures generated response includes explicit source file citations
  • Precision of retrieved graph triplets for query relevance • target: 0.96 • range: 0.88-1
Configured evaluation evidence
  • Benchmark: RegRAG-Bench
  • Retrieval Precision target: 0.96
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Llama Index · Required
  • LlamaIndex · Optional
  • Guidance · Optional

Evaluation contract

  • has_valid_citations · Weight 1
  • retrieval_precision · Weight 1

Metrics

  • retrieval_precision · Target 0.96

Benchmarks

  • RegRAG-Bench
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON containing natural language regulatory question

    Output: JSON containing answer, cited_documents, and extracted_triplets

    Execution harness: Python

Evaluation tools

  • LlamaIndex · agent_framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about BIS Regulatory Compliance Knowledge Graph with LlamaIndex