What is chain-of-thought prompting?
Chain-of-thought prompting gets a model to write out intermediate reasoning steps before its answer.
Chain-of-thought (CoT) prompting means getting a language model to write out intermediate reasoning steps before it gives a final answer. The paper that named it (Wei et al., January 2022) did so by putting a few exemplars with worked reasoning in the prompt. Google Research describes the idea as letting models “decompose multi-step problems into intermediate steps”. Nothing is retrained. It is a technique of prompt engineering, and the few-shot form builds on few-shot prompting.
Its role has changed since 2022. Today’s reasoning models think before they answer, so providers now advise asking for steps only in some cases.
The original research
- Few-shot CoT. Wei et al. report gains on arithmetic, commonsense and symbolic reasoning across three models. A 540B-parameter model with eight exemplars reached state of the art accuracy on GSM8K, beating finetuned GPT-3 with a verifier. Google’s write-up gives 58% against the prior 55%, and notes that the authors wrote the chains by hand and that the GSM8K runs used an external calculator.
- Scale matters. Per the same post, the benefits “only materialize with a sufficient number of model parameters (around 100B)”. On CommonsenseQA, StrategyQA and date understanding, CoT gave “additional small improvements”, while sports understanding gained the most.
- Zero-shot CoT. Kojima et al. (May 2022) found that adding “Let’s think step by step” before each answer lifted InstructGPT (text-davinci-002) from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on GSM8K.
- Self-consistency. Wang et al. (2022) sample a diverse set of reasoning paths instead of taking the greedy one, then pick the most consistent answer. They report gains over plain CoT of 17.9% on GSM8K, 11.0% on SVAMP and 3.9% on ARC-challenge.
These results come from 2022-era models. They show why the technique took off, not what it adds on a current model. For how it fits among other techniques, see prompt engineering techniques.
What providers say now
| Anthropic | OpenAI | Google (Gemini) | |
|---|---|---|---|
| Stance | Manual CoT is a fallback for when thinking is off | For reasoning models, “think step by step” or “explain your reasoning” is unnecessary | Generally not necessary to detail reasoning steps in the response |
| What to do instead | On Claude Opus 5.5, Sonnet 5.5 and Fable 5.1, rely on thinking, at a lower effort level if cost or latency matter; a prompt asking them to write out their reasoning may be declined | Reasoning models “perform best with straightforward prompts” | For harder problems, ask it to “Think very hard before answering”, which uses more thinking tokens |
| Multi-step tasks | Prompt chaining when you need to inspect intermediate outputs | Make each step a prompt and chain them |
Anthropic’s fallback advice is to ask Claude to think through the problem before it answers and put the final answer in <answer> tags so you can extract it. Anthropic notes that Claude 4.6 and later use adaptive thinking, where Claude “dynamically determines when and how much to think”.
A written chain is not proof of the reasoning
A visible chain is easy to read as an explanation. Lanham et al. (2023) intervened on chains, for example by adding mistakes, and found that reliance on the chain varied by task. They also report that “as models become larger and more capable, they produce less faithful reasoning on most tasks we study”, while CoT “can be faithful if the circumstances such as the model size and task are carefully chosen”. Treat the text as a useful trace, not a guarantee.
When to use it
- On a reasoning model or with thinking on, start without a CoT instruction, as the providers above advise.
- On a model with no built-in thinking, or with thinking off, try a step-by-step instruction or few-shot examples with worked reasoning.
- If you need to read or check intermediate results, split the task into chained prompts.
- Compare each variant on the same cases. A step-by-step instruction adds output tokens, so check cost and latency as well as accuracy. LLM evaluation covers how to grade outputs.
Comparing CoT prompts in HoneyHive
A HoneyHive experiment runs a function on a dataset with evaluate() and scores each output with evaluators. To test CoT, run one experiment per prompt variant (with and without the step-by-step instruction) on the same dataset and compare the aggregated metrics.
FAQ
What is chain-of-thought prompting in simple terms?
It is a prompt that leads the model to write out intermediate reasoning steps before the final answer. The original paper did this with a few exemplars that show "a series of intermediate reasoning steps".
What does "Let's think step by step" do?
It is the zero-shot version. Kojima et al. showed that adding the phrase before each answer made models "decent zero-shot reasoners", with no examples needed.
Do reasoning models still need chain-of-thought prompts?
Usually not. OpenAI's guide calls prompting reasoning models to "think step by step" unnecessary, and Google's says it is generally not necessary to have the model detail reasoning steps in the response.
Is a model's written chain of thought its real reasoning?
Not always. Anthropic researchers found it "unclear if the stated reasoning is a faithful explanation of the model's actual reasoning", and the answer varied by task and model size.