Skip to main content
Back to challenges
Challenge

Automated Merchant Risk Scoring Agent with Pydantic AI and OpenRouter

Rapid proliferation of e-commerce AI agents requires real-time risk assessment for autonomous consumer payment actions. Build a type-safe merchant risk analyzer using Pydantic AI and OpenRouter to validate transaction intent, verify merchant legitimacy, and enforce dynamic spending controls. Maintain zero false positives on verified low-risk merchant checkouts while flagging fraudulent intent.

CybersecurityHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Develop a real-time, type-safe merchant risk evaluation engine using Pydantic AI and OpenRouter to screen autonomous consumer agent transactions.

Delivery guide

How work is evaluated

Evaluation

Evaluates accuracy of merchant threat assessment and validity of structured response models.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

high_risk_flag_test

Verifies suspicious URLs are blocked or flagged for step-up MFA

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

risk_precision

Precision of risk decision classifications • target: 0.95 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Define strict Pydantic AI output models for merchant verification and transaction limits

  • Configure OpenRouter dynamic LLM fallback chains for low-latency scoring

  • Implement domain validation and merchant history lookup tools

  • Enforce step-up authentication when merchant risk parameters trigger medium-risk alerts

Resources and assets

Reference links and supporting material

Dataset notes

Dataset of 400 transaction checkout payloads with synthetic domain characteristics and merchant trust signals.

Challenge setup

Evaluates accuracy of merchant threat assessment and validity of structured response models.

Challenge input

JSON containing checkout details, merchant URL, user ID, and intent payload

Pydantic AI

Type-safe modeling framework

OpenRouter

Multi-provider model routing

Evaluated output

Pydantic validated JSON with risk_score, decision, and rationale

Evaluation checks
  • Verifies suspicious URLs are blocked or flagged for step-up MFA
  • Precision of risk decision classifications • target: 0.95 • range: 0-1
Configured evaluation evidence
  • Benchmark: FraudGuard-Eval
  • Risk Precision target: 0.95
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Pydantic AI · Required
Policy Serving
  • OpenRouter · Optional
Orchestration
  • Pydantic AI · Required
  • OpenRouter · Optional

Evaluation contract

  • high_risk_flag_test · Weight 1
  • risk_precision · Weight 1

Metrics

  • risk_precision · Target 0.95

Benchmarks

  • FraudGuard-Eval
  • security

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON containing checkout details, merchant URL, user ID, and intent payload

    Output: Pydantic validated JSON with risk_score, decision, and rationale

    Execution harness: Python

Evaluation tools

  • Pydantic AI · Agent Framework
  • OpenRouter · LLM Gateway

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Automated Merchant Risk Scoring Agent with Pydantic AI and OpenRouter