Orchestrating Research Workflow with Prefect and CrewAI

implementationChallenge

Prompt Content

Define a Prefect flow that orchestrates the entire CrewAI research process for a given technical topic. The flow should include:
1. A `Researcher` task to perform initial web searches and store key findings in ChromaDB.
2. An `Analyst` task to retrieve findings from ChromaDB, synthesize them, and store structured insights back into ChromaDB.
3. A `Technical Writer` task to retrieve all relevant structured insights from ChromaDB and generate a comprehensive markdown report. This task should correspond to the `ResearchReportGeneration` evaluation task.

```python
# ... (previous agent and tool definitions)

from crewai import Task
from prefect import flow, task

# Define Tasks
task_research_topic = Task(
    description=(
        "Conduct comprehensive web research on the topic: '{topic}'. "
        "Identify key historical milestones, major contributors, and technological advancements. "
        "Store all relevant findings in ChromaDB using the 'store_research' tool." 
    ),
    expected_output='A summary of key research findings stored in ChromaDB.',
    agent=researcher
)

task_analyze_findings = Task(
    description=(
        "Retrieve all research findings related to '{topic}' from ChromaDB. "
        "Synthesize the information, identify patterns, and structure it into an outline. "
        "Store the structured insights back into ChromaDB using the 'store_research' tool." 
    ),
    expected_output='A structured outline of insights related to the topic, stored in ChromaDB.',
    agent=analyst
)

task_write_report = Task(
    description=(
        "Retrieve all structured insights and research data related to '{topic}' from ChromaDB. "
        "Based on this, write a comprehensive, well-structured technical report in markdown format. "
        "The report should include sections like Introduction, Key Concepts, Historical Context, Current State, Future Outlook, and Conclusion." 
    ),
    expected_output='A final markdown-formatted technical research report.',
    agent=writer
)

@flow(name="Technical Research Flow")
def technical_research_flow(topic: str):
    # Instantiate the CrewAI crew with the topic as context
    tech_research_crew = Crew(
        agents=[researcher, analyst, writer],
        tasks=[task_research_topic, task_analyze_findings, task_write_report],
        verbose=True,
        process="sequential" # Or 'hierarchical' if using hierarchical agents
    )

    print(f"Starting research on: {topic}")
    crew_result = tech_research_crew.kickoff(inputs={'topic': topic})
    print(f"CrewAI research completed. Final report:\n{crew_result}")
    return crew_result

# To run this flow in Prefect:
# 1. Ensure Prefect server is running
# 2. python your_script.py
# 3. technical_research_flow("The future of AI ethics and external audits.")
```

Try this prompt

Open the workspace to execute this prompt with free credits, or use your own API keys for unlimited usage.

Usage Tips

Copy the prompt and paste it into your preferred AI tool (Claude, ChatGPT, Gemini)

Customize placeholder values with your specific requirements and context

For best results, provide clear examples and test different variations