Back to evaluations
Public evaluation

ProactiveScenarioHandling

The assistant will be evaluated on its ability to understand proactive user scenarios, accurately delegate tasks to specialized agents via A2A Protocol, and correctly execute commands through simulated iOS APIs, demonstrating proactive and context-aware assistance.

Evaluation type
task based
Challenge
Proactive iOS Digital Assistant
Difficulty
Advanced
Rigor
Unspecified
Open linked challenge

Evaluation overview

How the linked challenge is judged: tasks, benchmarks, and criteria count.

Tasks
2
Benchmarks
0
Criteria
0

Task templates

Inputs and expected outputs.

Task 1

ProactiveScenarioHandling

Assistant must identify a proactive opportunity based on simulated user context and suggest/execute an appropriate action.

Input format

{ 'user_context': { 'calendar_events': ['Meeting at 3 PM today'], 'location': 'home', 'current_time': '2:30 PM', 'message_alerts': [] }, 'user_preferences': { 'travel_time_alert': true } }

Output format

{ 'proactive_suggestion': 'Traffic is heavy, leaving now for your 3 PM meeting is recommended.', 'executed_actions': ['Send notification'] }

Task 2

A2ACommunicationAndToolUse

Central agent delegates a task requiring specific iOS service interaction to a specialized agent via A2A Protocol, which then uses a simulated API.

Input format

{ 'request': 'Schedule a call with John Doe for tomorrow at 10 AM', 'contact_info': { 'John Doe': '123-456-7890' } }

Output format

{ 'delegation_status': 'success', 'receiving_agent': 'Calendar Agent', 'tool_output': { 'calendar_event_id': 'evt123', 'status': 'scheduled' } }