BACKBACK

What is CrewAI? How it works and how to evaluate it

Build agents
Updated
HoneyHive Engineering

CrewAI is an open-source Python framework for multi-agent systems, pairing Crews of role-based agents with Flows that control state and steps.

CrewAI is an open-source Python framework for building multi-agent systems. Its docs call it “the leading open-source framework for orchestrating autonomous AI agents and building complex workflows,” and it has two layers: Crews, teams of role-based agents, and Flows, which control state and the order of steps (introduction).

Crews and Flows

The docs describe a Flow as “the ‘manager’ or the ‘process definition’ of your application,” and Crews as “the ‘teams’ that do the heavy lifting” (introduction).

Layer What it does Use it for
Flow State across steps, event-driven execution, conditionals, loops and branching The structure of the application
Crew Agents with roles and goals that delegate and collaborate on tasks Work that needs autonomy and collaboration

A Flow starts the process and holds state, hands a complex task to a Crew, then continues from the result. For production applications, the docs advise starting with a Flow and calling a Crew inside a step when a task needs agent teamwork. Simple automation can be a Flow with plain Python tasks alone (introduction).

What a Crew is made of

  • Agents: each has a role, goal and backstory. Optional limits include max_iter (default 20, after which the agent must give its best answer) and max_execution_time. allow_delegation defaults to False, and respect_context_window (default True) summarizes messages to stay under the context window (agents).
  • Tasks: a description, an expected_output, the responsible agent, and optional context from other tasks, output_pydantic for structured output, async_execution and human_input for human review of the final answer (tasks).
  • Process: sequential by default, where “tasks are executed one after another.” A hierarchical process adds a manager agent that delegates tasks and validates outcomes, and it requires a manager_llm or manager_agent (crews).
  • Crew settings: memory, a tool-result cache that defaults to True, max_rpm, and a checkpoint setting that saves state so runs can resume (crews). Memory is now one unified Memory class that “replaces separate short-term, long-term, entity, and external memory types.” With memory=True the crew creates a default Memory(), and agents share it unless one has its own (memory).

You start a crew with kickoff(). For high-concurrency workloads the docs recommend native async with akickoff() over kickoff_async(), which wraps the synchronous run in asyncio.to_thread (crews).

When to use it

CrewAI fits work you can describe as roles and tasks, such as one agent researching and another writing, with a Flow around it for control. If you need explicit graph control with checkpointed state and interrupts, compare LangGraph. For a lighter SDK with handoffs, see the OpenAI Agents SDK. For background, see what AI agents are and agentic workflows.

Guardrails and limits that shape reliability

  • A task guardrail validates output before the next task. When it fails, “the error is sent back to the agent, and the task is retried up to guardrail_max_retries times,” which defaults to 3 (tasks). For the wider topic, see AI guardrails.
  • max_iter caps agent loops at 20 by default, and max_rpm on a crew overrides each agent’s own rate limit (agents, crews).
  • Replay re-runs from one task. Find task IDs with crewai log-tasks-outputs, then run crewai replay -t <task_id> (crews).

How to test and evaluate a CrewAI app

The built-in crewai test runs your crew for a set number of iterations (default 2) and reports metrics. Task scores run from 1 to 10 with higher better, alongside a crew score per run and execution time. The --model option defaults to gpt-4o-mini, and as of October 2026 the docs say “for now, the only provider available is OpenAI” (testing).

That tells you whether a crew’s output looks good on your inputs. It does not replace a dataset of test cases with your own scorers. The general method is in LLM evaluation: curate cases, score outputs, compare runs after each change. For multi-agent crews, also check which agent handled each task and which tools it called, which needs traces (what agent observability is).

Tracing a CrewAI app in HoneyHive

HoneyHive traces CrewAI through OpenInference instrumentors, with no decorators. It captures crew runs, each agent’s role, prompts and outputs, model requests, tool runs with arguments and results, Flow.kickoff() and each flow step, and standalone Agent.kickoff() calls. Tool spans need openinference-instrumentation-crewai 1.1.11 or later. Pass tracer_provider to both instrumentors, because omitting it is a common reason traces do not appear (integration docs):

from honeyhive import HoneyHiveTracer
from openinference.instrumentation.crewai import CrewAIInstrumentor
from openinference.instrumentation.openai import OpenAIInstrumentor

tracer = HoneyHiveTracer.init()  # reads HH_INGESTION_API_KEY from env

CrewAIInstrumentor().instrument(tracer_provider=tracer.provider)
OpenAIInstrumentor().instrument(tracer_provider=tracer.provider)

The OpenAI instrumentor covers OpenAI-backed calls; for other providers use that provider’s instrumentor. Setup details are on the HoneyHive CrewAI integration page.

Related: Google ADK, LangGraph vs CrewAI vs AutoGen.

FAQ

Is CrewAI free?

Yes. The framework is released under the MIT License and installs with uv tool install crewai (GitHub).

Is CrewAI built on LangChain?

No (what LangChain is). The repository describes it as "a standalone Python framework with its own primitives for agents, tasks, crews, flows, tools, and orchestration" (GitHub). Agents can still use tools from LangChain (agents).

What is the difference between a Crew and a Flow?

A Crew is a team of agents working on tasks together. A Flow is the process around it, handling state, events and control flow, and it can call a Crew in one of its steps (introduction).

How do I trace a CrewAI app?

Install the OpenInference CrewAI instrumentor and point it at your tracer provider. Existing crew code runs unchanged (HoneyHive CrewAI integration).

Share