Skip to main content
Back to challenges
Challenge

Build an E2E Payment Rail Recovery RAG Agent with LlamaIndex

Payment failures across complex end-to-end routing networks require instant root-cause analysis and mitigation strategies. Develop an enterprise knowledge and recovery RAG agent using LlamaIndex that indexes error logs and dynamically suggests routing repairs.

CybersecurityHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build an enterprise payment operations agent using LlamaIndex with dynamic query engines to diagnose payment routing failure logs.

Delivery guide

How work is evaluated

Evaluation

Evaluates accuracy of error diagnosis and correctness of reroute action recommendations.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

test_no_retry_closed_account

Ensure closed account code prevents useless automated retry

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

diagnosis_accuracy

Accuracy of root cause diagnosis • target: 0.96 • range: 0.88-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Index ISO 20022 error codes and acquirer specification documentation using LlamaIndex.

  • Create dynamic agent query engines to diagnose transactional error trajectories.

  • Deliver sub-second root cause diagnosis and automated rerun actions.

Resources and assets

Reference links and supporting material

Dataset notes

Synthetic payment failure logs with ISO 20022 rejection codes and gateway technical guides.

Challenge setup

Evaluates accuracy of error diagnosis and correctness of reroute action recommendations.

Challenge input

JSON payment reject message containing raw status code and acquiring bank response

LlamaIndex

Framework requirement

Evaluated output

JSON containing root_cause, recovery_action, retry_recommended

Evaluation checks
  • Ensure closed account code prevents useless automated retry
  • Accuracy of root cause diagnosis • target: 0.96 • range: 0.88-1
Configured evaluation evidence
  • Benchmark: RAG Diagnostics Benchmark v1
  • Diagnosis Accuracy target: 0.96
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Llama Index · Required
  • LlamaIndex · Optional
  • Llama 3.2 · Optional

Evaluation contract

  • test_no_retry_closed_account · Weight 1
  • diagnosis_accuracy · Weight 1

Metrics

  • diagnosis_accuracy · Target 0.96

Benchmarks

  • RAG Diagnostics Benchmark v1
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON payment reject message containing raw status code and acquiring bank response

    Output: JSON containing root_cause, recovery_action, retry_recommended

    Execution harness: Python

Evaluation tools

  • LlamaIndex · agent_framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Build an E2E Payment Rail Recovery RAG Agent with LlamaIndex