Skip to main content
Loading navigation
Back to prompt library

Orchestrating Research Workflow with Prefect and CrewAI

Linked challenge: Collaborative Technical Research Crew

Prompt source

implementation54 lines9 sectionsNo variables1 code block
Define a Prefect flow that orchestrates the entire CrewAI research process for a given technical topic. The flow should include:
1. A `Researcher` task to perform initial web searches and store key findings in ChromaDB.
2. An `Analyst` task to retrieve findings from ChromaDB, synthesize them, and store structured insights back into ChromaDB.
3. A `Technical Writer` task to retrieve all relevant structured insights from ChromaDB and generate a comprehensive markdown report. This task should correspond to the `ResearchReportGeneration` evaluation task.

```python
# ... (previous agent and tool definitions)

from crewai import Task
from prefect import flow, task

# Define Tasks
task_research_topic = Task(
    description=(
        "Conduct comprehensive web research on the topic: '{topic}'. "
        "Identify key historical milestones, major contributors, and technological advancements. "
        "Store all relevant findings in ChromaDB using the 'store_research' tool." 
    ),
    expected_output='A summary of key research findings stored in ChromaDB.',
    agent=researcher
)

task_analyze_findings = Task(
    description=(
        "Retrieve all research findings related to '{topic}' from ChromaDB. "
        "Synthesize the information, identify patterns, and structure it into an outline. "
        "Store the structured insights back into ChromaDB using the 'store_research' tool." 
    ),
    expected_output='A structured outline of insights related to the topic, stored in ChromaDB.',
    agent=analyst
)

task_write_report = Task(
    description=(
        "Retrieve all structured insights and research data related to '{topic}' from ChromaDB. "
        "Based on this, write a comprehensive, well-structured technical report in markdown format. "
        "The report should include sections like Introduction, Key Concepts, Historical Context, Current State, Future Outlook, and Conclusion." 
    ),
    expected_output='A final markdown-formatted technical research report.',
    agent=writer
)

@flow(name="Technical Research Flow")
def technical_research_flow(topic: str):
    # Instantiate the CrewAI crew with the topic as context
    tech_research_crew = Crew(
        agents=[researcher, analyst, writer],
        tasks=[task_research_topic, task_analyze_findings, task_write_report],
        verbose=True,
        process="sequential" # Or 'hierarchical' if using hierarchical agents
    )

    print(f"Starting research on: {topic}")
    crew_result = tech_research_crew.kickoff(inputs={'topic': topic})
    print(f"CrewAI research completed. Final report:\n{crew_result}")
    return crew_result

# To run this flow in Prefect:
# 1. Ensure Prefect server is running
# 2. python your_script.py
# 3. technical_research_flow("The future of AI ethics and external audits.")
```