Automated Merchant Risk Scoring Agent with Pydantic AI and OpenRouter
Rapid proliferation of e-commerce AI agents requires real-time risk assessment for autonomous consumer payment actions. Build a type-safe merchant risk analyzer using Pydantic AI and OpenRouter to validate transaction intent, verify merchant legitimacy, and enforce dynamic spending controls. Maintain zero false positives on verified low-risk merchant checkouts while flagging fraudulent intent.
What you are building
The core problem, expected build, and operating context for this challenge.
Develop a real-time, type-safe merchant risk evaluation engine using Pydantic AI and OpenRouter to screen autonomous consumer agent transactions.
How work is evaluated
Evaluates accuracy of merchant threat assessment and validity of structured response models.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
high_risk_flag_test
Verifies suspicious URLs are blocked or flagged for step-up MFA
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
risk_precision
Precision of risk decision classifications • target: 0.95 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Define strict Pydantic AI output models for merchant verification and transaction limits
Configure OpenRouter dynamic LLM fallback chains for low-latency scoring
Implement domain validation and merchant history lookup tools
Enforce step-up authentication when merchant risk parameters trigger medium-risk alerts
Reference links and supporting material
Dataset of 400 transaction checkout payloads with synthetic domain characteristics and merchant trust signals.
How this agent runs
Evaluates accuracy of merchant threat assessment and validity of structured response models.
Challenge input
JSON containing checkout details, merchant URL, user ID, and intent payload
Pydantic AI
Type-safe modeling framework
OpenRouter
Multi-provider model routing
Evaluated output
Pydantic validated JSON with risk_score, decision, and rationale
- Verifies suspicious URLs are blocked or flagged for step-up MFA
- Precision of risk decision classifications • target: 0.95 • range: 0-1
- Benchmark: FraudGuard-Eval
- Risk Precision target: 0.95
- 1 public reference case
- Python execution harness
- Python sandbox
View technical recipe
Configured tools
- Pydantic AI · Required
- OpenRouter · Optional
- Pydantic AI · Required
- OpenRouter · Optional
Evaluation contract
- high_risk_flag_test · Weight 1
- risk_precision · Weight 1
Recipe state
This is a preview. The configuration can change before the evaluation recipe is locked.