Autonomous Market Strategy Team
This challenge involves constructing an autonomous, multi-agent market strategy team using the CrewAI framework. The team will be tasked with analyzing the competitive landscape for emerging generative AI technologies, such as new video generation models or large language models like DeepSeek. The goal is to provide strategic recommendations for businesses looking to innovate or compete in this rapidly evolving sector. The core intelligence of the agent team will be driven by DeepSeek R1, selected for its advanced reasoning capabilities and cost-efficiency, dynamically routed via OpenRouter for optimal performance across various analytical tasks. Crucially, the agents will leverage MemGPT for long-term memory, allowing them to recall past analyses, adapt to evolving market trends, and maintain conversational context over extended interactions. This enables more sophisticated and consistent strategic planning. To support the development and interaction with this complex system, GitHub Copilot will serve as an AI engineering assistant for creating custom tools and refining agent logic, while Bito AI will provide a chat interface for users to query the team, review reports, and provide feedback, fostering a human-in-the-loop oversight.
What you are building
The core problem, expected build, and operating context for this challenge.
Develop a role-based CrewAI agent team for market strategy, utilizing DeepSeek R1, MemGPT for long-term memory, OpenRouter for model routing, GitHub Copilot, and Bito AI.
How work is evaluated
Evaluation will assess the CrewAI team's ability to autonomously generate accurate, comprehensive, and actionable market strategy reports. Key metrics will include the quality of agent collaboration, effective utilization of long-term memory, and the precision of analytical outputs.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
Report Completeness
Ensure all required sections (summary, findings, recommendations) are present in the market strategy report.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Contextual Recall Accuracy
Verify that the follow-up query response correctly leverages information from the initial query.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Tool Invocation Success
Confirm that all necessary custom tools were successfully invoked by agents during tasks.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Strategic Recommendation Quality
Manual rating (1-5) of the actionability and creativity of strategic recommendations. • target: 4 • range: 1-5
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
DeepSeek R1 Utilization Rate
Percentage of complex analytical tasks processed by DeepSeek R1 via OpenRouter. • target: 0.98 • range: 0.9-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Agent Collaboration Score
Metric (0-1) reflecting efficient and non-redundant task distribution among agents. • target: 0.85 • range: 0.7-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master CrewAI framework for defining agent roles, tasks, and processes, enabling seamless collaboration for strategic analysis.
Implement MemGPT for agents to maintain long-term memory, allowing recall of prior market analyses, company profiles, and strategic discussions.
Design and integrate custom tools for CrewAI agents to perform market research, competitor analysis, financial modeling, and report generation.
Leverage OpenRouter to dynamically select and route requests to DeepSeek R1 or other models based on task complexity, cost, or specific capabilities.
Build a conversational interface using Bito AI to allow stakeholders to interact with the CrewAI team, pose questions, and receive strategic reports.
Utilize GitHub Copilot as an AI pair programmer to accelerate the development of agent tools, task definitions, and complex Python logic within the CrewAI setup.
Develop robust evaluation criteria for the CrewAI team's generated market strategies, focusing on accuracy, completeness, and actionable insights.
Reference links and supporting material
Sample data for 2 tasks
Challenge setup
Evaluation will assess the CrewAI team's ability to autonomously generate accurate, comprehensive, and actionable market strategy reports. Key metrics will include the quality of agent collaboration, effective utiliza...
Challenge input
{ "topic": "string", "industry": "string", "report_format": "json|markdown" }
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{ "report_title": "string", "executive_summary": "string", "key_findings": ["string"], "strategic_recommendations": ["string"] }
- Ensure all required sections (summary, findings, recommendations) are present in the market strategy report.
- Verify that the follow-up query response correctly leverages information from the initial query.
- Confirm that all necessary custom tools were successfully invoked by agents during tasks.
- Strategic Recommendation Quality target: 4
- 2 public reference cases
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Report Completeness · Weight 1
- Contextual Recall Accuracy · Weight 1
- Tool Invocation Success · Weight 1
- Strategic Recommendation Quality · Weight 1
- DeepSeek R1 Utilization Rate · Weight 1
- Agent Collaboration Score · Weight 1
Metrics
- Strategic Recommendation Quality · Target 4
- DeepSeek R1 Utilization Rate · Target 0.98
- Agent Collaboration Score · Target 0.85
Reference cases
- 2 public reference cases
Workflow tasks
- Task 1
Input: { "topic": "string", "industry": "string", "report_format": "json|markdown" }
Output: { "report_title": "string", "executive_summary": "string", "key_findings": ["string"], "strategic_recommendations": ["string"] }
Execution harness: Python
- Task 2
Input: { "initial_query": "string", "follow_up_query": "string" }
Output: { "initial_response_summary": "string", "follow_up_response": "string", "recalled_context_evidence": "string" }
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.