Public evaluation

BrandSentimentAnalysis

The evaluation will assess the system's ability to accurately identify brand mentions, analyze sentiment, and generate coherent reports based on simulated prompt data. It will also test the functionality of tool integrations and the Voiceflow interface.

Evaluation type

task based

Challenge

Agents for Prompt-Driven Brand Sentiment & Affinity

Difficulty

Advanced

Rigor

Unspecified

Open linked challenge

Evaluation overview

How the linked challenge is judged: tasks, benchmarks, and criteria count.

Tasks

Benchmarks

Criteria

Task templates

Inputs and expected outputs.

Task 1

BrandSentimentAnalysis

Evaluate the multi-agent system's ability to process a batch of prompts, extract brand mentions, determine sentiment, and flag high-affinity or low-affinity instances.

Input format

JSON array of strings, each string representing an AI-generated prompt.

Output format

JSON object with brand names as keys, each containing an array of identified sentiments (e.g., 'positive', 'negative', 'neutral') and associated prompt snippets.

Task 2

WorkflowAutomationTrigger

Verify that the Kore.ai integrated workflow is triggered correctly when specific conditions are met (e.g., negative sentiment spike for a tracked brand).

Input format

Simulated event data indicating a negative sentiment threshold breach for a brand.

Output format

Confirmation of Kore.ai workflow execution (e.g., webhook log entry, specific API call made).

Task 3

VoiceflowQueryResponse

Test the Voiceflow interface's ability to respond to natural language queries about brand sentiment and system status.

Input format

String representing a natural language query (e.g., 'What's the current sentiment for Product X?').

Output format

String response from the Voiceflow agent, containing relevant sentiment data or status.

Linked challenge

Agents for Prompt-Driven Brand Sentiment & Affinity

This challenge focuses on building a sophisticated multi-agent system using CrewAI to analyze brand mentions and sentiment within a stream of AI-generated prompts. The system will orchestrate specialized agents to monitor, categorize, and report on brand affinity. The core idea is to simulate an intelligent monitoring platform that provides actionable insights into brand perception and recommendation patterns from diverse data sources, leveraging CrewAI's role-playing architecture. Developers will design a collaborative agent team, where each agent has a distinct role, tools, and goals. For instance, a 'Prompt Ingestor' agent, a 'Brand Analyst' agent, and a 'Report Generator' agent will work in concert. The system will use DeepSeek R1 for its advanced reasoning capabilities to accurately interpret nuanced sentiment and complex brand associations. Integration with Zapier Interfaces will enable seamless data ingestion from various sources, while tool can facilitate custom workflow automation for alert triggers and data processing. Voiceflow will provide a conversational interface for real-time query and status updates, making the system highly interactive. Blaxel will serve as the underlying agent orchestration backbone, ensuring robust agent management.

Open challenge