Skip to main content
Back to challenges
Challenge

Home Improvement Loan Collateral Risk RAG Engine with LlamaIndex

Institutional investors acquiring home improvement loan portfolios require detailed automated risk audits on borrower property equity and lien positions. Build an advanced RAG and document parsing pipeline using LlamaIndex to process loan agreement filings, title searches, and property appraisal documents. Achieve 92% accuracy in flags raised for unrecorded senior liens.

Business OperationsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Construct a document analysis and loan risk engine using LlamaIndex to audit home improvement collateral and lien priorities.

Delivery guide

How work is evaluated

Evaluation

Evaluates accuracy of extracted loan CLTV and junior lien risk flags.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

risk_flag_match

Checks correct identification of junior lien risk.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

cltv_mae

Mean Absolute Error on calculated CLTV percentage. • target: 0.02 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Implement LlamaIndex index structures (VectorStoreIndex, SummaryIndex) over mortgage legal documents

  • Build custom LlamaIndex Query Engines for extracting junior vs senior lien status

  • Calculate Combined Loan-To-Value (CLTV) ratios from appraisal text and loan balance figures

  • Flag high-risk collateral accounts with inadequate remaining equity

Resources and assets

Reference links and supporting material

Dataset notes

Dataset of synthetic home improvement loan underwriting files, appraisal reports, and title search abstracts.

Challenge setup

Evaluates accuracy of extracted loan CLTV and junior lien risk flags.

Challenge input

JSON object with dossier_text, primary_mortgage_bal, improvement_loan_amt, appraised_val

LlamaIndex

Mandatory assigned framework for document retrieval and indexing.

Evaluated output

JSON object with computed_cltv, junior_lien_risk (boolean)

Evaluation checks
  • Checks correct identification of junior lien risk.
  • Mean Absolute Error on calculated CLTV percentage. • target: 0.02 • range: 0-1
Configured evaluation evidence
  • Benchmark: UnderwritingRAGBench
  • Cltv Mae target: 0.02
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Llama Index · Required
  • RAI · Optional
  • aiXplain · Optional

Evaluation contract

  • risk_flag_match · Weight 1
  • cltv_mae · Weight 1

Metrics

  • cltv_mae · Target 0.02

Benchmarks

  • UnderwritingRAGBench
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON object with dossier_text, primary_mortgage_bal, improvement_loan_amt, appraised_val

    Output: JSON object with computed_cltv, junior_lien_risk (boolean)

    Execution harness: Python

Evaluation tools

  • LlamaIndex · RAG Framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Home Improvement Loan Collateral Risk RAG Engine with LlamaIndex