Agent planning: how LLM agents decompose and choose steps

Agent planning is how an LLM agent decides what steps to take to reach a goal. The model breaks the task into sub-tasks, orders them, picks tools, and revises the steps when something goes wrong. A 2024 survey by Huang and colleagues formalizes it as generating a sequence of actions from the environment, the task goal, the model’s parameters and the prompts.

Anthropic describes the behavior this way: once the task is clear, “agents plan and operate independently, potentially returning to the human for further information or judgement.” That is what separates an agent from a workflow, where code fixes the steps.

Five ways to plan

The survey, which calls itself the first systematic view of LLM-based agent planning, sorts existing work into five groups. The methods in the table are the ones it names.

Approach	How it plans	Methods the survey names
Task decomposition	Split the task into sub-tasks and plan each, either all up front or a step or two at a time	Plan-and-Solve, HuggingGPT, ProgPrompt (up front); Chain-of-Thought, ReAct, PAL (interleaved)
Multi-plan selection	Generate several candidate plans and pick one by voting or search	Self-Consistency, Tree of Thoughts, Graph of Thoughts
External planner	The LLM formalizes the task and a separate planner solves it	LLM+P, LLM+PDDL, LLM+ASP
Reflection and refinement	Look at a failed attempt and revise the plan	Self-Refine, Reflexion, CRITIC
Memory-augmented	Retrieve stored experience or knowledge to guide the plan	Generative Agents, MemoryBank, MemGPT

Decomposition

Plan-and-Solve prompting, an ACL 2023 paper, targets the errors of zero-shot chain-of-thought, which the authors say makes calculation errors, skips steps and misreads semantics. It asks the model to devise a plan that splits the task into subtasks and then carry them out. Across ten datasets with GPT-3, it beat zero-shot chain-of-thought “by a large margin.”

ReAct takes the other route. It has the model produce reasoning traces and actions in an interleaved way, and the authors say the reasoning helps it “induce, track, and update action plans.” The plan lives in the running trace and changes with each observation. ReAct agents covers the loop, and AI agent architecture compares it with plan-and-execute designs.

Plan selection

Tree of Thoughts lets the model explore several reasoning paths, evaluate them and backtrack. On the Game of 24, the paper reports that GPT-4 with chain-of-thought prompting solved 4% of tasks while Tree of Thoughts solved 74%. Search costs extra model calls, so it suits problems where early choices matter.

External planners

In LLM+P, as Lilian Weng’s overview of June 2023 describes it, the model translates the problem into PDDL, a classical planner produces the plan, and the model translates it back to natural language. It needs a PDDL description of the domain and a suitable planner, which she notes is common in some robotics setups but not in many others.

Reflection and memory

In Reflexion, Weng writes, reflections on earlier attempts are added to working memory, up to three, to guide later ones. Memory-augmented planning retrieves past experience instead. See agent memory.

Delegated planning

In the orchestrator-workers pattern Anthropic describes, “a central LLM dynamically breaks down tasks, delegates them to worker LLMs, and synthesizes their results.” The subtasks are set by the orchestrator from the specific input rather than fixed in advance. See multi-agent systems and agent orchestration.

Where plans fail

The survey lists these open problems:

• Hallucinated plans. It says LLMs often hallucinate, “leading to irrational plans.”
• Infeasible plans. Plans can ignore preconditions or uncommon constraints.
• Inefficient plans. Planning is largely greedy, and generated plans are not checked for efficiency.
• Weak evaluation. Benchmarks often score only final success and lack “fine-grained step-wise evaluations.”

Earlier, the PlanBench benchmark, built from domains used in the automated planning community, found that LLM performance on plan generation “falls quite short, even with the SOTA models.” That paper was first posted in June 2022 and last revised in November 2023, so it predates current reasoning models. In a September 2024 follow-up, the same group called OpenAI o1’s score “a quantum improvement on the benchmark” but “still far from saturating it.” Weng’s overview names two weak spots: planning over a long history, and adjusting a plan after an unexpected error.

Anthropic says agents should gain “ground truth” from the environment at each step, which lets a plan be checked against real tool results.

Inspecting and testing plans

The survey’s evaluation point has a practical side: score the steps, not only the outcome. A trace of each model call and tool call shows the plan as it ran. HoneyHive’s trajectory view plots a session’s spans by type and step order, and its docs say it helps spot patterns such as repeated tool calls or rising latency. Clicking a span opens it in the tree view with its inputs and outputs. Agent tracing explains what to capture.

For scoring, see agent evaluation metrics, which groups metrics into outcome, tool call, trajectory, reliability and efficiency, and testing AI agents. An LLM judge can grade a plan against criteria you write, as long as you check it against human labels first.

FAQ

What is agent planning?

The part of an agent that turns a goal into a sequence of actions. A 2024 survey frames it as generating a sequence of actions from the environment, the task goal, the model and the prompts.

What are the main approaches to LLM agent planning?

The same survey sorts them into five groups: task decomposition, multi-plan selection, external planner-aided planning, reflection and refinement, and memory-augmented planning.

What is the difference between ReAct and plan-and-solve?

ReAct interleaves reasoning and actions, so the plan updates after each observation. Plan-and-Solve writes the plan first, then carries out the subtasks. See ReAct agents.

Can LLMs plan reliably?

Not yet in general. The PlanBench authors reported in 2022 and 2023 that LLM performance on plan generation fell quite short even with the state-of-the-art models of the time. Their 2024 follow-up found OpenAI o1 much better but "still far from saturating" the benchmark, and the planning survey lists hallucinated and infeasible plans as open problems.

How do you debug an agent's plan?

Trace each step and read the sequence. HoneyHive's trajectory view plots a session's model and tool spans in order, which makes repeated tool calls easy to spot.