Automated Generative AI Model Evaluation
This challenge focuses on building a sophisticated, automated evaluation harness for generative AI models. Instead of relying solely on quantitative metrics, the system will employ a team of AI agents, powered by CrewAI and Claude Opus 4.5, to qualitatively assess diverse generative outputs like creative writing, complex code snippets, or synthetic media descriptions. The system needs to generate comprehensive evaluation reports, track experiments, and orchestrate evaluation workflows for continuous benchmarking.
What you are building
The core problem, expected build, and operating context for this challenge.
Develop an automated generative AI evaluation system using a CrewAI team with Claude Opus 4.5 for qualitative assessment, tracked with MLflow and orchestrated by Prefect.
How work is evaluated
The evaluation will focus on the effectiveness of the agent team in providing nuanced qualitative assessments, the correctness of `MLflow` tracking, and the robustness of the `Prefect` orchestration. The system's ability to handle diverse generative model outputs and produce actionable insights will be key.
Challenge datasets
Public datasets, plus the private datasets you have access to.
What you should walk away with
Integrate experiment tracking for evaluation results and model versions.
Design and implement a multi-agent system for automated generative AI evaluation.
Orchestrate complex evaluation pipelines for continuous benchmarking and analysis.
Leverage advanced LLMs for nuanced qualitative assessment of generated content.
Reference links and supporting material
Sample data for 2 tasks
Challenge setup
The evaluation will focus on the effectiveness of the agent team in providing nuanced qualitative assessments, the correctness of `MLflow` tracking, and the robustness of the `Prefect` orchestration. The system's abil...
Challenge input
{'model_id': 'gpt-3.5-turbo', 'generated_text': 'A short story about a detective solving a mystery in a magical city.'}
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{'qualitative_report': {'creativity_score': 4.5, 'coherence_score': 4.0, 'originality_score': 4.2, 'summary': 'Detailed feedback...'}, 'mlflow_run_...
- Ensures the CrewAI agent team executes the evaluation task without errors and produces a structured report.
- Verifies that the `mlflow_run_id` is present and corresponds to a logged run in MLflow.
- Confirms that the agent system successfully stores or retrieves evaluation context from ChromaDB (evaluated...
- QualitativeReportCompleteness target: 0.9
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Ensures the CrewAI agent team executes the evaluation task without errors and produces a structured report.
- Verifies that the `mlflow_run_id` is present and corresponds to a logged run in MLflow.
- Confirms that the agent system successfully stores or retrieves evaluation context from ChromaDB (evaluated via internal logs or a mock API).
Metrics
- QualitativeReportCompleteness · Target 0.9
- ClaudeOpusReasoningQuality · Target 0.8
- PrefectWorkflowReliability · Target 0.95
Workflow tasks
- Task 1
Input: {'model_id': 'gpt-3.5-turbo', 'generated_text': 'A short story about a detective solving a mystery in a magical city.'}
Output: {'qualitative_report': {'creativity_score': 4.5, 'coherence_score': 4.0, 'originality_score': 4.2, 'summary': 'Detailed feedback...'}, 'mlflow_run_id': '...'}
Execution harness: Python
- Task 2
Input: {'model_id': 'gpt-4', 'code_snippet': 'def factorial(n): if n == 0: return 1 else: return n * factorial(n-1)'}
Output: {'code_review': {'correctness': 'Pass', 'efficiency_rating': 'Good', 'best_practices': 'Recursive, but clean.', 'issues': [], 'recommendations': []}, 'mlflow_run_id': '...'}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.