Skip to main content
Back to challenges
Challenge

Architecting Autonomous Refactoring Pipelines with CrewAI and Claude Opus 4.6.6

Leverage the power of Claude Opus 4.6.6 and CrewAI to simulate the architectural refactoring of complex codebases. Inspired by the rapid migration of legacy systems to memory-safe environments, this challenge requires you to build a team of agents that analyze structural dependencies, propose modernization strategies, and validate refactored code modules against strict performance requirements. You will utilize Pydantic AI for structured data validation across the agent team and incorporate Lakera to ensure that code-generation prompts are hardened against injection attacks. Evaluation is integrated directly into the development cycle using Galileo to monitor agent performance and drift, ensuring that the autonomous migration remains consistent with your initial codebase standards. By integrating All Hands AI as a development interface, you will provide a seamless environment for the agents to interact with live repositories, ensuring the refactoring process is iterative and measurable.

Workflow AutomationHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Orchestrate a role-based CrewAI agent team to perform automated code refactoring and migration using Claude Opus 4.6.6 and Pydantic AI for structured communication.

Delivery guide

How work is evaluated

Evaluation

Validation of code structure integrity and security policy compliance.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

injection_scan

Ensure Lakera flags zero vulnerabilities

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

refactor_accuracy

Success rate of syntax-valid migrations • target: 0.95 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Master CrewAI role-based orchestration to decompose monolith refactoring tasks into granular assignments

  • Implement Pydantic AI models to guarantee data integrity in agent-to-agent communication

  • Deploy Lakera scanners to audit prompt-response cycles for security vulnerabilities during the refactoring process

  • Integrate Galileo observability to track latency and quality metrics for autonomous agent cycles

  • Configure All Hands AI as the interactive agent execution environment for multi-turn coding sessions

  • Design task-specific goal metrics to optimize Claude Opus 4.6.6 reasoning outputs for complex migration logic

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 1 tasks

Challenge setup

Validation of code structure integrity and security policy compliance.

Challenge input

Source codebase snapshot

CrewAI

Framework for orchestrating

Galileo

Generative AI eval and observability platform.

Evaluated output

Refactored module and architectural plan

Evaluation checks
  • Ensure Lakera flags zero vulnerabilities
  • Success rate of syntax-valid migrations • target: 0.95 • range: 0-1
Configured evaluation evidence
  • Refactor Accuracy target: 0.95
  • 1 public reference case
Configured runtime
  • Python execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • CrewAI · Required
  • crewAI · Optional
Observation
  • Galileo · Optional
Reward / Eval
  • Galileo · Optional

Evaluation contract

  • injection_scan · Weight 1
  • refactor_accuracy · Weight 1

Metrics

  • refactor_accuracy · Target 0.95

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: Source codebase snapshot

    Output: Refactored module and architectural plan

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Architecting Autonomous Refactoring Pipelines with CrewAI and Claude Opus 4.6.6