Skip to main content
Back to challenges
Challenge

Build an Advanced Visual Web Agent with OpenAI Agents SDK and GPT-5 Pro

Develop a sophisticated multi-agent system designed to interact with web interfaces visually, mimicking human browser usage without relying on direct HTML parsing. This challenge leverages OpenAI Agents SDK for orchestrating agent teams, enabling them to collaboratively perform complex tasks. Agents, powered by the advanced reasoning capabilities of GPT-5 Pro, will execute high-level planning and decision-making. BrowserUse will be utilized to accelerate the development of custom browser automation tools (e.g., using Playwright or Selenium), allowing agents to perform precise visual interactions. Gentrace provides critical evaluation and observability pipelines to monitor and refine agent performance, while Sarvam AI enables intuitive voice-activated commands for controlling the agent system, making it highly accessible for real-world applications such as subscriber analysis on a simulated Beehiiv-like platform and SEO optimization.

Special Purpose AgentsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Design and implement a visual web agent using OpenAI Agents SDK, GPT-5 Pro, and specialized tools for complex browser automation and voice control.

Delivery guide

How work is evaluated

Evaluation

Evaluation will focus on the agent's ability to accurately perform visual web navigation, extract information, and execute tasks based on natural language instructions, with a strong emphasis on tool utilization and error handling.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
5 scoring checks
Binary
5 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

CorrectInformationExtraction

Verifies that the agent extracts correct numerical and textual information from the simulated web interface.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

ToolExecutionSuccess

Checks if all required browser automation tools were successfully invoked and completed their actions without errors.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 3

VoiceCommandResponsiveness

Assesses if the agent correctly interprets and responds to specified voice commands.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 4

TaskCompletionRate

Percentage of tasks successfully completed end-to-end. • target: 90 • range: 0-100

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 5

ExecutionTime

Average time taken to complete a task in seconds. • target: 60 • range: 0-300

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Master OpenAI Agents SDK for building and managing autonomous agents with function calling, tool use, and multi-turn conversational capabilities

  • Design and implement robust visual web interaction tools using modern browser automation frameworks (e.g., Playwright) to enable agents to navigate and interact with web UIs without direct HTML parsing

  • Integrate Portia AI for defining agent roles, managing their configurations, and overseeing their lifecycle within a multi-agent system

  • Leverage BrowserUse for accelerated development and refinement of custom tool code, enhancing agent capabilities for specific web automation tasks

  • Establish comprehensive evaluation and observability pipelines using Gentrace to monitor agent decision-making, tool execution, and overall task completion accuracy

  • Implement voice-activated command processing using Sarvam AI to provide a natural language interface for directing and monitoring the visual web agent system

  • Build extended reasoning pipelines with GPT-5 Pro to enable advanced planning, problem-solving, and adaptive strategy formulation for unforeseen web scenarios

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 2 tasks

Challenge setup

Evaluation will focus on the agent's ability to accurately perform visual web navigation, extract information, and execute tasks based on natural language instructions, with a strong emphasis on tool utilization and e...

Challenge input

{ 'dashboard_url': 'simulated_beehiiv_url', 'analysis_period': 'last 30 days' }

GPT-5 Pro

Models · Large Language Models

OpenAI

OpenAI AI model provider

Portia AI

Open source framework for predictable,

Evaluated output

{ 'subscriber_count': int, 'growth_percentage': float, 'key_trends_summary': 'string' }

Evaluation checks
  • Verifies that the agent extracts correct numerical and textual information from the simulated web interface.
  • Checks if all required browser automation tools were successfully invoked and completed their actions witho...
  • Assesses if the agent correctly interprets and responds to specified voice commands.
Configured evaluation evidence
  • TaskCompletionRate target: 90
  • 2 public reference cases
Configured runtime
  • Python execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • GPT-5 Pro · Required
  • OpenAI · Optional
  • Portia AI · Optional

Evaluation contract

  • CorrectInformationExtraction · Weight 1
  • ToolExecutionSuccess · Weight 1
  • VoiceCommandResponsiveness · Weight 1
  • TaskCompletionRate · Weight 1
  • ExecutionTime · Weight 1

Metrics

  • TaskCompletionRate · Target 90
  • ExecutionTime · Target 60

Reference cases

  • 2 public reference cases

Workflow tasks

  1. Task 1

    Input: { 'dashboard_url': 'simulated_beehiiv_url', 'analysis_period': 'last 30 days' }

    Output: { 'subscriber_count': int, 'growth_percentage': float, 'key_trends_summary': 'string' }

    Execution harness: Python

  2. Task 2

    Input: { 'target_url': 'https://example.com/blog', 'target_keyword': 'generative AI trends' }

    Output: { 'missing_alt_tags': int, 'low_density_keywords': ['keyword1', 'keyword2'], 'suggested_improvements': ['string'] }

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Build an Advanced Visual Web Agent with OpenAI Agents SDK and GPT-5 Pro