Agentic SaaS Competitive Intelligence
This challenge focuses on building a sophisticated multi-agent system to provide competitive intelligence and strategic recommendations for a SaaS company facing market pressures from new agentic AI tools. Leveraging the structured output capabilities of Pydantic AI and the advanced reasoning of Gemini 3 Pro, developers will design and implement a team of specialized agents. These agents will autonomously research market trends, analyze competitor offerings (especially new AI-powered solutions), and evaluate internal performance metrics to identify vulnerabilities and opportunities. The system will emphasize data quality and integrity, using Cleanlab for pre-processing and validating research inputs. Agent interactions will be orchestrated to ensure a coherent analysis, culminating in actionable strategic insights. Observability and evaluation are paramount, with Arize AI integrated to monitor agent performance, output quality, and decision-making processes, ensuring the system provides reliable and impactful intelligence for business leaders.
What you are building
The core problem, expected build, and operating context for this challenge.
Build a multi-agent competitive intelligence system using Pydantic AI for structured analysis, orchestrated by Shakudo, with Gemini 3 Pro for reasoning and Arize AI for observability.
How work is evaluated
The evaluation will assess the system's ability to accurately analyze market trends, identify competitive threats from agentic AI tools, and generate actionable, structured strategic recommendations. Emphasis will be placed on the robustness of Pydantic AI models, the coherence of agent orchestration, and the quality of observed outcomes via Arize AI.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
Structured Output Adherence
Outputs from agents strictly adhere to defined Pydantic models.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Recommendation Actionability
Generated recommendations are clear, concise, and actionable for a business.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Analysis Depth
Score based on the comprehensiveness and insightfulness of the market and competitor analysis. • target: 4 • range: 0-5
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Agent Orchestration Efficiency
Measures how effectively agents collaborate to complete tasks without redundancy or conflict. • target: 4 • range: 0-5
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master Pydantic AI for building type-safe agents that generate and consume structured data models, ensuring robust data integrity throughout the agent workflow.
Orchestrate hierarchical multi-agent workflows using Shakudo, defining complex task dependencies and inter-agent communication protocols for a cohesive intelligence gathering process.
Leverage Gemini 3 Pro's advanced reasoning capabilities for nuanced market analysis, trend identification, and strategic recommendation generation within a Pydantic AI agent.
Implement data validation and cleaning pipelines using Cleanlab to ensure high-quality, reliable input data for agent processing, minimizing noise and bias.
Design and integrate observability into the multi-agent system using Arize AI to track agent execution, evaluate output quality, and monitor overall system performance and decision validity.
Build tool-calling agents with Pydantic AI to interface with external APIs for real-time market data retrieval and competitor analysis, returning structured results.
Reference links and supporting material
Sample data for 2 tasks
Challenge setup
The evaluation will assess the system's ability to accurately analyze market trends, identify competitive threats from agentic AI tools, and generate actionable, structured strategic recommendations. Emphasis will be...
Challenge input
JSON object with 'company_profile' and 'market_context' fields.
Agent execution
No agent tool is listed for this challenge.
Evaluated output
JSON object with 'trends': [{'name': str, 'description': str, 'impact': str}], 'recommendations': [str]
- Outputs from agents strictly adhere to defined Pydantic models.
- Generated recommendations are clear, concise, and actionable for a business.
- Score based on the comprehensiveness and insightfulness of the market and competitor analysis. • target: 4...
- Analysis Depth target: 4
- 2 public reference cases
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Structured Output Adherence · Weight 1
- Recommendation Actionability · Weight 1
- Analysis Depth · Weight 1
- Agent Orchestration Efficiency · Weight 1
Metrics
- Analysis Depth · Target 4
- Agent Orchestration Efficiency · Target 4
Reference cases
- 2 public reference cases
Workflow tasks
- Task 1
Input: JSON object with 'company_profile' and 'market_context' fields.
Output: JSON object with 'trends': [{'name': str, 'description': str, 'impact': str}], 'recommendations': [str]
Execution harness: Python
- Task 2
Input: JSON object with 'competitor_name' and 'competitor_features': {'ai_features': [str], 'pricing_model': str}
Output: JSON object with 'analysis': {'strengths': [str], 'weaknesses': [str], 'strategic_implications': [str]}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.