Cross-Border Financial API Security Scanner using AutoGen and Garak
As Singapore and India integrate PayNow-UPI fast payment networks, cybersecurity risks such as prompt injection against AI support tools, schema manipulation, and API vulnerability exploitation become paramount concerns for central banking authorities. In this challenge, you will construct a multi-agent security scanner using AutoGen combined with Garak (the LLM vulnerability scanner). The AutoGen multi-agent team consists of a Red-Team Adversary Agent, a Defensive Inspector Agent, and a Patch Validation Agent. The system automatically launches security probes using Garak against payment gateway API endpoints and verifies whether defensive guardrails withstand zero-day injection attacks.
What you are building
Build an automated security testing framework for financial APIs using AutoGen multi-agent collaboration and Garak LLM vulnerability scanning.
How work is evaluated
Evaluates ability of the AutoGen/Garak system to uncover injected vulnerabilities and auto-generate valid remediations.
Datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
vulnerability_detection_rate
Must identify 100% of injected seed vulnerabilities.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
remediation_resilience_score
Resilience percentage of patched prompt against second-pass Garak scan • target: 95 • range: 0-100
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Learning goals
Orchestrate a multi-agent conversation loop using AutoGen to simulate adversary-defender scenarios
Incorporate Garak security probes to test financial API guardrails against prompt injections
Generate structured threat mitigation reports aligned with MAS Cyber Hygiene Guidelines
Automate patch validation testing for LLM system prompts
Resources
Dataset of central bank API specifications, vulnerable system prompts, and benchmark prompt injection attacks.
Challenge setup
Evaluates ability of the AutoGen/Garak system to uncover injected vulnerabilities and auto-generate valid remediations.
Challenge input
JSON with system prompt under test and target API schema
AutoGen
Required framework for orchestrating adversary and defender interactions.
Garak
Required tool for LLM vulnerability probes.
Evaluated output
JSON vulnerability report detailing detected breaches and verified remediations
- Must identify 100% of injected seed vulnerabilities.
- Resilience percentage of patched prompt against second-pass Garak scan • target: 95 • range: 0-100
- Benchmark: SecLLM-Bench
- Remediation Resilience Score target: 95
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Shows the first task, up to three tools, and up to three evidence items.
View technical recipe
Configured tools
- AutoGen · Required
- TPOT · Optional
- Garak · Optional
- Garak · Optional
Evaluation contract
- vulnerability_detection_rate · Weight 1
- remediation_resilience_score · Weight 1
Metrics
- remediation_resilience_score · Target 95
Benchmarks
- SecLLM-Bench
- security
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON with system prompt under test and target API schema
Output: JSON vulnerability report detailing detected breaches and verified remediations
Execution harness: Python
Evaluation tools
- AutoGen · Multi-Agent Framework
- Garak · Security Scanner
Recipe freeze marker
No recipe freeze time is recorded.