Skip to main content
Back to challenges
Challenge

Build Multimodal Adversarial Benchmarking Agents

This challenge focuses on developing an advanced agent system to perform rigorous, adversarial evaluation of cutting-edge multimodal Large Language Models (LLMs). Participants will design and implement a multi-agent system capable of generating complex, multimodal prompts (text, image, audio) and then evaluating the responses from different LLMs for accuracy, coherence, and robustness against adversarial attacks. The system will leverage the A2A (Agent-to-Agent) Protocol for seamless communication between an adversarial prompt generation agent and an evaluation agent. DSPy will be used to programmatically optimize the adversarial prompt generation and response analysis, ensuring the benchmark is adaptive and thorough. This challenge emphasizes the practical application of agentic AI for quality assurance and objective comparison of next-generation generative models.

Special Purpose AgentsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Develop an A2A multi-agent system using DSPy to generate adversarial multimodal prompts and benchmark generative AI models.

Delivery guide

How work is evaluated

Evaluation

The evaluation will assess the system's ability to generate diverse, challenging multimodal benchmarks and accurately evaluate LLM responses. Key areas include the robustness of the A2A communication, the effectiveness of DSPy in optimizing evaluation logic, and the insights gained from comparing GPT 5.1 Pro and Gemini 2.5 Pro.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Learning goals

What you should walk away with

  • Utilize DSPy for programmatic optimization of multimodal prompt generation and evaluation logic.

  • Compare the robustness and performance of Ernie 5.0 against Gemini 2.5 Pro on generated benchmarks.

  • Develop techniques for generating and evaluating complex multimodal content (text, image, audio).

  • Design and implement an A2A Protocol-enabled multi-agent system for adversarial benchmarking.

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 2 tasks

Challenge setup

The evaluation will assess the system's ability to generate diverse, challenging multimodal benchmarks and accurately evaluate LLM responses. Key areas include the robustness of the A2A communication, the effectivenes...

Challenge input

{ 'target_model': 'Ernie 5.0' }

Agent execution

No agent tool is listed for this challenge.

Evaluated output

{ 'prompts': [{'id': 'p1', 'text': '...', 'image_url': '...', 'audio_url': '...'}], 'adversarial_strategy': '...' }

Evaluation checks
  • Verify that agents successfully communicate and exchange multimodal data via A2A Protocol without errors.
  • Confirm that DSPy is correctly used to optimize prompt generation or evaluation logic, demonstrating improv...
  • Ensure the system can process and generate references to image and audio data correctly.
Configured evaluation evidence
  • Adversarial Prompt Diversity target: 0.8
Configured runtime
  • Python execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

No tool records are attached.

Evaluation contract

  • Verify that agents successfully communicate and exchange multimodal data via A2A Protocol without errors.
  • Confirm that DSPy is correctly used to optimize prompt generation or evaluation logic, demonstrating improved performance over naive prompting.
  • Ensure the system can process and generate references to image and audio data correctly.

Metrics

  • Adversarial Prompt Diversity · Target 0.8
  • Evaluation Metric Accuracy · Target 0.75

Workflow tasks

  1. Task 1

    Input: { 'target_model': 'Ernie 5.0' }

    Output: { 'prompts': [{'id': 'p1', 'text': '...', 'image_url': '...', 'audio_url': '...'}], 'adversarial_strategy': '...' }

    Execution harness: Python

  2. Task 2

    Input: { 'prompt': {'id': 'p1', 'text': '...', 'image_url': '...', 'audio_url': '...'}, 'model_response': {'text_output': '...', 'generated_image_url': '...', 'generated_audio_url': '...'} }

    Output: { 'eval_scores': {'accuracy': 0.8, 'coherence': 0.7, 'safety': 0.9}, 'justification': '...' }

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Build Multimodal Adversarial Benchmarking Agents