Skip to main content
Back to challenges
Challenge

Automated Web & Mobile Accessibility Auditing Agent

Smart Nation Singapore initiatives require digital public service products to adhere strictly to WCAG 2.2 AAA accessibility standards. Construct an automated accessibility testing agent using Playwright and Axe-core to inspect web interfaces. Generate structured remediation summaries for web developers automatically.

Workflow AutomationHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build an automated web accessibility audit agent using Playwright and Axe-core for public service portals.

Delivery guide

How work is evaluated

Evaluation

Evaluates DOM accessibility issue detection recall against known flawed HTML samples.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

violation_recall_check

Ensures all seeded accessibility errors are identified.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

audit_recall

Recall percentage of known WCAG defects. • target: 1 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Automate headless browser navigation using Playwright

  • Integrate Axe-core engine to perform WCAG 2.2 ruleset validation

  • Extract DOM node selectors corresponding to contrast, ARIA label, and keyboard focus defects

  • Format actionable markdown/JSON remediation reports for developer pipelines

Resources and assets

Reference links and supporting material

Dataset notes

Synthetic HTML page components containing pre-seeded WCAG accessibility defects.

Challenge setup

Evaluates DOM accessibility issue detection recall against known flawed HTML samples.

Challenge input

HTML snippet string

Playwright

Standard browser testing framework.

Evaluated output

JSON violation summary

Evaluation checks
  • Ensures all seeded accessibility errors are identified.
  • Recall percentage of known WCAG defects. • target: 1 • range: 0-1
Configured evaluation evidence
  • Benchmark: WebA11y-Audit-Bench
  • Audit Recall target: 1
  • 1 public reference case
Configured runtime
  • JavaScript execution harness
  • javascript sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Metaflow · Optional
  • Smolagents · Optional
Policy Serving
  • OpenRouter · Required
Orchestration
  • OpenRouter · Required

Evaluation contract

  • violation_recall_check · Weight 1
  • audit_recall · Weight 1

Metrics

  • audit_recall · Target 1

Benchmarks

  • WebA11y-Audit-Bench
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: HTML snippet string

    Output: JSON violation summary

    Execution harness: JavaScript

Evaluation tools

  • Playwright · Browser Automation Engine

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Automated Web & Mobile Accessibility Auditing Agent