Skip to main content
Back to challenges
Challenge

Build a Clearinghouse Real-Time Risk Exposure RAG Engine with LlamaIndex

Securities clearinghouses handling billions in daily trade volume need continuous vector-indexed risk policy enforcement. Build an advanced RAG query and tool-calling agent using LlamaIndex to query margin liquidity rules and assess broker-dealer clearing exposure. Achieve sub-second retrieval with 96% context relevance across complex clearinghouse handbooks.

CybersecurityHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Construct an advanced LlamaIndex RAG agent to evaluate broker-dealer clearing collateral exposure against clearinghouse rulebooks.

Delivery guide

How work is evaluated

Evaluation

Evaluate retrieval precision and margin deficit calculation accuracy.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

Correct Deficit Calculation

Ensure margin_call_issued triggers when required margin exceeds cash collateral.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

Context Precision

Precision of retrieved regulatory text context. • target: 0.96 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Build a hierarchical vector index using LlamaIndex over complex clearinghouse rulebooks

  • Implement LlamaIndex QueryEngine tools to calculate dynamic margin maintenance calls

  • Validate broker-dealer clearing account capital adequacy under high-volatility stress scenarios

  • Ensure context retrieval accuracy with zero hallucinated policy citations

Resources and assets

Reference links and supporting material

Dataset notes

Synthetic dataset of 300 clearinghouse risk manual sections paired with 1,000 broker capital telemetry frames.

Challenge setup

Evaluate retrieval precision and margin deficit calculation accuracy.

Challenge input

JSON containing broker_id, cash_collateral, equity_exposure, and market_volatility_index.

LlamaIndex

Offers specialized indexing and retrieval primitives for complex structured documents.

Evaluated output

JSON containing margin_required, margin_call_issued (boolean), and cited_rule_section.

Evaluation checks
  • Ensure margin_call_issued triggers when required margin exceeds cash collateral.
  • Precision of retrieved regulatory text context. • target: 0.96 • range: 0-1
Configured evaluation evidence
  • Benchmark: FinRAGBench
  • Context Precision target: 0.96
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Llama Index · Required
  • LlamaIndex · Optional
  • Llama 3.2 · Optional

Evaluation contract

  • Correct Deficit Calculation · Weight 1
  • Context Precision · Weight 1

Metrics

  • Context Precision · Target 0.96

Benchmarks

  • FinRAGBench
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON containing broker_id, cash_collateral, equity_exposure, and market_volatility_index.

    Output: JSON containing margin_required, margin_call_issued (boolean), and cited_rule_section.

    Execution harness: Python

Evaluation tools

  • LlamaIndex · Data & RAG Framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Build a Clearinghouse Real-Time Risk Exposure RAG Engine with LlamaIndex