Skip to main content
Back to challenges
Challenge

Build a Proactive Executive Assistant Agent with OpenAI Agents SDK

Inspired by the recent discussion around advanced call-screening features, this challenge tasks you with developing a personalized, proactive AI executive assistant. This agent will autonomously manage communications, prioritize tasks, and synthesize information, effectively acting as a digital chief of staff. It should demonstrate complex reasoning, dynamic tool usage, and an understanding of user preferences to handle various professional scenarios. The core of this challenge involves leveraging the OpenAI Agents SDK to orchestrate a sophisticated agent workflow. You will implement function calling to integrate with external tools for managing schedules, emails, and information retrieval. The agent needs to exhibit nuanced decision-making, adapting its behavior based on the context of incoming communications and the user's current priorities, much like a human executive assistant would filter and manage information flow.

Special Purpose AgentsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Develop a personalized AI executive assistant agent using OpenAI Agents SDK for proactive task management and communication screening.

Delivery guide

How work is evaluated

Evaluation

The evaluation will assess the agent's ability to accurately understand user intent, execute multi-step tasks using integrated tools, and deliver personalized, coherent responses. Key metrics include task success rate, response relevance, and adherence to user preferences.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
6 scoring checks
Binary
6 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

CorrectToolInvocation

Agent correctly identifies and invokes the appropriate tool (e.g., calendar for scheduling).

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

AccurateScheduling

Scheduled event details (time, duration, attendees) match user request and calendar availability.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 3

KeyInfoExtraction

All critical action items are extracted from the email summary task.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 4

ResponseRelevance

Semantic similarity of agent's conversational response to expected answer (0-1). • target: 0.9 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 5

PersonalizationAdherence

Degree to which the agent's actions and responses align with specified user preferences (0-1). • target: 0.85 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 6

ToolExecutionSuccessRate

Percentage of tool invocations that successfully complete without errors (0-1). • target: 0.95 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Master the OpenAI Agents SDK for defining agent behavior, memory management, and tool integration patterns with GPT-4o

  • Implement robust function calling mechanisms to enable agents to interact with external APIs like calendar (Google Calendar API) and email (e.g., Gmail API via Zapier NLA)

  • Design personalized prompting strategies and few-shot examples within the OpenAI Agents SDK to tailor agent responses and actions to individual user preferences and historical interactions

  • Build a browser automation tool using E2B to enable the agent to access web-based information or dashboards as part of its executive assistant duties

  • Orchestrate complex agent workflows that involve dynamic tool selection and sequential reasoning for tasks such as meeting scheduling, email summarization, and task delegation

  • Deploy and manage agent instances, considering aspects like state persistence and secure API key management within the OpenAI ecosystem

  • Integrate Vellum for continuous evaluation and prompt experimentation, using A/B testing and trace analysis to refine agent performance and reduce hallucinations

  • Design an interactive user interface using Ellipsis to provide a seamless conversational experience for the executive assistant agent

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 2 tasks

Challenge setup

The evaluation will assess the agent's ability to accurately understand user intent, execute multi-step tasks using integrated tools, and deliver personalized, coherent responses. Key metrics include task success rate...

Challenge input

{'user_prompt': 'string', 'calendar_state': 'list_of_events'}

Agent execution

No agent tool is listed for this challenge.

Evaluated output

{'action': 'string', 'tool_name': 'string', 'tool_params': 'dict', 'response': 'string'}

Evaluation checks
  • Agent correctly identifies and invokes the appropriate tool (e.g., calendar for scheduling).
  • Scheduled event details (time, duration, attendees) match user request and calendar availability.
  • All critical action items are extracted from the email summary task.
Configured evaluation evidence
  • ResponseRelevance target: 0.9
  • 2 public reference cases
Configured runtime
  • Python execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

No tool records are attached.

Evaluation contract

  • CorrectToolInvocation · Weight 1
  • AccurateScheduling · Weight 1
  • KeyInfoExtraction · Weight 1
  • ResponseRelevance · Weight 1
  • PersonalizationAdherence · Weight 1
  • ToolExecutionSuccessRate · Weight 1

Metrics

  • ResponseRelevance · Target 0.9
  • PersonalizationAdherence · Target 0.85
  • ToolExecutionSuccessRate · Target 0.95

Reference cases

  • 2 public reference cases

Workflow tasks

  1. Task 1

    Input: {'user_prompt': 'string', 'calendar_state': 'list_of_events'}

    Output: {'action': 'string', 'tool_name': 'string', 'tool_params': 'dict', 'response': 'string'}

    Execution harness: Python

  2. Task 2

    Input: {'email_content': 'string', 'user_preference_keywords': 'list_of_strings'}

    Output: {'summary': 'string', 'action_items': 'list_of_strings'}

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Build a Proactive Executive Assistant Agent with OpenAI Agents SDK