ReAct: connecting reasoning and action through environmental feedback

Original source

Shunyu Yao et al., submitted in 2022 with the ICLR version published in 2023: ReAct: Synergizing Reasoning and Acting in Language Models. This guide uses arXiv v3.

This is an original learning guide, not a reproduction. Copyright and the complete paper remain with the original source.

Learning objectives

By the end of this guide, you should be able to:

1. Describe ReAct in terms of input, state, action, observation, and termination rather than merely recalling three labels.
2. Walk through a complete Thought → Action → Observation → … → Finish trajectory and state exactly what changes at each step.
3. Explain why knowledge-retrieval tasks use relatively dense reasoning–action alternation, while long interactive tasks often need only sparse plan updates.
4. Diagnose a failed trajectory as a reasoning, tool-selection, argument, observation, or stopping-condition failure.

The idea in one sentence

ReAct interleaves language reasoning traces with real actions in an environment: reasoning selects the next action, and the resulting environmental observation revises the reasoning that follows.

First clarify what “Thought” means

A Thought in the paper is visible language generated as part of a task trajectory. It can record a plan, evidence summary, or progress marker, but it is not proof of the model’s complete or faithful hidden mental process. A modern system also need not expose private chain-of-thought. Short plans, state summaries, or structured decision rationales can play the same control-loop role.

Why neither thinking alone nor acting alone is enough

Imagine a question that requires two searches. A chain-of-thought-only model can write a coherent derivation, but it is confined to knowledge already represented in its parameters. If its first factual premise is wrong, later steps may build fluently on that mistake. An action-only model can search, but it must infer the next command directly from a growing sequence of the question, results, and previous actions. It can easily forget a subgoal, repeat a query, or submit too early.

ReAct addresses both gaps:

• Reason to act: use language to decompose the goal, choose a tool, prepare arguments, track progress, and decide when to stop.
• Act to reason: use search, page interaction, or an environmental action to obtain information the model did not already have, then update the next decision with that observation.

The paper starts from a conventional agent policy that chooses an action after observing its context. ReAct extends the task action space $A$ to $A\cup L$, where $L$ is the language space. A Thought from $L$ does not change the external world; it appends useful information to the trajectory context. An Action from $A$ affects or queries the environment and leads to a new Observation.

The five objects you need to distinguish

Object	Role in the paper	Changes the external environment?	What an implementation should retain
Task	Defines the goal and constraints	No	Original request, success criteria, and resource limits
Thought	Decomposes the goal, extracts observations, tracks progress, or revises a plan	No	A short plan or state summary, never mislabeled as verified fact
Action	Searches, queries, clicks, moves, or submits an answer	Yes	Tool name, arguments, call ID, and time
Observation	The environment’s response to an Action	The environment has been queried or changed	Raw return, source, error status, and trust level
Finish	Terminates the trajectory	It may submit the final output	Final answer and evidence, or an explicit failure state

There is also an implicit core state called the context: the Task and every prior Thought, Action, and Observation in order. The next decision must be conditioned on this updated trajectory rather than restarting from the original question each round.

The complete mechanism

flowchart TD
    Q["Task: goal and acceptance criteria"] --> C["Context: task + current trajectory"]
    C --> D{"What is needed next?"}
    D -->|"Organize the goal, evidence, or exception"| T["Thought: update language state"]
    T --> C
    D -->|"Need information or an environmental change"| A["Action: select tool and arguments"]
    A --> E["Environment: execute the action"]
    E --> O["Observation: fact, state, or error"]
    O --> C
    D -->|"Evidence and success conditions are satisfied"| F["Finish: answer or completion signal"]
    C -->|"Step limit, loop, or unrecoverable error"| X["Fail with a recorded reason"]

One turn of the loop can be decomposed into six operations:

1. Read the current context. The model receives the task and the trajectory so far.
2. Decide whether a language-state update is useful. If the goal still needs decomposition, an observation needs interpretation, or the plan must change, it emits a Thought. The Thought updates context but does not create environmental evidence.
3. Choose an executable action. The action must belong to the domain’s allowed action set, such as search[entity], lookup[string], a web click, or a text-game command.
4. Let the environment execute it. The model cannot simply declare that a search succeeded or an object was picked up. A runtime or simulator performs the operation.
5. Append the raw observation. A search snippet, page state, success message, or error is returned to the context and must inform the next decision.
6. Check termination. The agent calls finish[...] only when the task’s success conditions are met. A step budget, repeated lack of progress, or unrecoverable error should produce an explicit failure instead of an infinite loop.

For the knowledge tasks, the paper uses a deliberately small Wikipedia action space. search[entity] returns the beginning of a matching page or similar entities, lookup[string] finds the next sentence containing a string on the current page, and finish[answer] submits the answer. This detail matters: ReAct does not let the model merely narrate an action. The language policy selects from an executable action protocol.

Walk through one trajectory from start to finish

The following is an original teaching example, not an item from the paper. Assume a local knowledge base contains two searchable records. The task is:

“Who founded the laboratory of the first author of Vector Gardens?”

Step	Trajectory entry	State change
Task	The question above	Establishes two implicit subgoals: find the first author, then find the founder of that author’s lab
Thought 1	Search the paper record first to identify the author and affiliation	Creates a plan; no new fact has been established
Action 1	search[Vector Gardens]	Requests a knowledge-base query
Observation 1	“First author: Lin Wei; affiliation: River Lab”	Adds traceable external evidence and turns the remaining subgoal into a query about River Lab
Thought 2	The lab is known; retrieve its history next	Extracts an entity from the observation and updates progress
Action 2	search[River Lab]	Issues the second query
Observation 2	“River Lab was founded in 2012 by Amina Rao”	Supplies the fact needed for the answer
Thought 3	The question asks for the founder, and the evidence names Amina Rao	Checks that the evidence matches the requested field
Finish	finish[Amina Rao]	Submits the answer and terminates

The two Observations are the evidence. The Thoughts are decisions about that evidence. If Thought 1 had guessed “perhaps Amina Rao,” that would remain a hypothesis until the environment returned support.

If the first search returned no result, a proper closed loop would not repeat the identical call forever. It would revise the plan—search for an author, try a title alias, or use another retrieval tool. Environmental feedback matters only when it can make the next policy choice differ from the old one.

Minimal implementable pseudocode

def react(task, tools, max_steps=8):
    trajectory = [{"type": "task", "content": task}]
 
    for step in range(max_steps):
        decision = model_decide(trajectory, allowed_actions=tools.schemas)
 
        if decision.type == "note":
            # A visible short plan or state summary, not external evidence.
            trajectory.append({"type": "thought", "content": decision.text})
            continue
 
        if decision.type == "finish":
            if success_criteria_met(decision.answer, trajectory):
                return {"status": "success", "answer": decision.answer,
                        "trajectory": trajectory}
            trajectory.append({"type": "observation",
                               "content": "finish rejected: evidence incomplete"})
            continue
 
        result = tools.execute(decision.tool, decision.arguments)
        trajectory.append({"type": "action", "tool": decision.tool,
                           "arguments": decision.arguments})
        trajectory.append({"type": "observation", "content": result})
 
        if repeated_without_progress(trajectory):
            return {"status": "failed", "reason": "loop detected",
                    "trajectory": trajectory}
 
    return {"status": "failed", "reason": "step budget exhausted",
            "trajectory": trajectory}

This skeleton adds three production necessities beyond the labels: an allowed action protocol, an acceptance check before finishing, and explicit exits for loops and exhausted budgets. A real runtime would also need authorization, timeouts, untrusted-output handling, retries, and audit records.

Dense reasoning versus sparse reasoning

ReAct does not require a long Thought before every Action. The paper uses two different rhythms:

• Knowledge-intensive reasoning: HotpotQA and FEVER require multi-hop retrieval. Their demonstrations alternate Thought, Action, and Observation densely. Thoughts decompose questions, extract facts from retrieved passages, reformulate searches, and synthesize answers.
• Long-horizon interactive decisions: ALFWorld and WebShop can contain many consecutive actions. Repeating the plan before every movement would consume context and create more opportunities for error. The paper lets the model produce Thoughts sparsely—when decomposing a goal, completing a subgoal, handling an exception, or applying commonsense.

The conclusion is therefore not “more reasoning is always better.” A more accurate rule is: update language state at decision points where the update can change the selected action. Opening a drawer and immediately taking a visible object rarely requires replanning. An empty search, an unexpected object location, or conflicting product attributes usually does.

Diagnosing a failed trajectory

Failure class	Trace symptom	What to inspect	Possible remedy
Reasoning or planning	Observations are correct, but the next subgoal is irrelevant	Task decomposition and the mapping from evidence to conclusion	Better demonstrations or explicit subgoal state
Tool selection	The agent uses a calculator for retrieval, or vice versa	Tool descriptions and routing constraints	Narrower action space and tool-choice tests
Arguments	The tool is right, but the query or object ID is wrong	Whether arguments came from the latest Observation	Schema validation and constrained entity copying
Uninformative observation	Search is empty or unrelated	Query reformulation and tool coverage	Rewrite the query or use a fallback retriever
Repetition	The same Action and Observation recur	Whether tried actions and progress are tracked	Loop detection, forced replanning, or termination
Premature Finish	The agent submits while key evidence is missing	Whether success is machine-checkable	Evidence requirements and an external evaluator
Untrusted observation	A tool result is wrong, stale, or contains prompt injection	Source, timestamp, and corroboration	Result isolation, provenance, and secondary checks

The paper’s manual HotpotQA analysis is useful because ReAct failures were not limited to hallucination. They also included reasoning errors, repetitive steps, and uninformative searches. Connecting a real knowledge base does not guarantee a correct answer: an agent that cannot recover from a bad observation still fails in a closed loop.

How to read the original evidence

The paper evaluates a frozen PaLM-540B with a few human-written trajectories on four task families, comparing Standard, CoT, Act, and ReAct:

• On HotpotQA, standalone ReAct reached 27.4 EM, slightly below CoT at 29.4. On FEVER, ReAct reached 60.9 accuracy, above CoT at 56.3. Switching strategies that combined ReAct with CoT self-consistency performed better still. External retrieval and internal reasoning were complementary; ReAct did not dominate CoT on every knowledge task.
• In controlled ALFWorld trials, the best ReAct prompt achieved 71% overall success, compared with 45% for the best Act prompt and 37% for the cited BUTLER result. The paper associates the gain with goal decomposition, progress tracking, and commonsense object-location reasoning.
• On WebShop, ReAct achieved 40.0% success versus 30.1% for Act, while expert humans reached 59.6%. Sparse reasoning helped, but adequate exploration and query reformulation remained unsolved.
• The HotpotQA failure percentages came from a limited manually inspected sample under that setup. They are not universal failure rates for today’s tool-using agents.

Evidence boundary

These findings depend on models, few-shot prompts, constrained action spaces, and benchmarks from around 2022. They show that interleaving reasoning traces with environmental feedback had systematic value in those settings. They do not show that adding three labels improves every modern model, nor that systems should reveal a model’s complete hidden reasoning to end users.

The original contribution and today’s engineering interpretation

What the 2022–2023 work directly demonstrated	A sound modern engineering interpretation
Few-shot prompts made a frozen model generate interleaved reasoning traces and task actions	Structured state summaries, tool calls, and environmental events can implement the same loop without copying the textual format
Wikipedia, text-game, and web environments returned Observations	A runtime must execute tools; the model cannot invent execution results
Humans could inspect the trace and distinguish model knowledge from external information	Production logs should separate model proposals, tool evidence, validation outcomes, and final decisions
Plans and progress were maintained within one task	Cross-task learning, long-term memory, and skill accumulation are separate mechanisms, not automatic consequences of ReAct

ReAct’s durable contribution is a controllable loop, not a prompt template. A current framework may rename Thought to plan_update, encode Action as JSON, and store Observation as a provenance-bearing event. If the causal order remains decision → execution → real feedback → revised decision, the core mechanism remains intact.

Check your understanding

1. Both Thought and Observation enter the context. What is the essential difference between them?
2. Why does a long-horizon task such as ALFWorld not need a long Thought before every action?
3. An agent runs search[Front Row] three times and receives the same empty result. Classify the failure, then define its next step and stopping condition.
4. Write the shortest complete trajectory for “check today’s weather in Beijing and decide whether to take an umbrella.” Include an acceptance criterion.

Suggested answers

1. A Thought is a model-generated plan or judgment that does not change the environment. An Observation is the fact, state, or error returned after the environment executes an Action. The former is decision material; the latter is external feedback.
2. Many consecutive movements or manipulations can execute directly. Replanning is useful at goal decomposition, subgoal transitions, exceptions, and uncertain decisions. Excessively dense Thoughts consume budget and create more opportunities for repetition.
3. This is an uninformative-observation plus repetition failure. Reformulate the query, for example as Front Row software, or switch tools. If a bounded number of reformulations still produces no evidence, terminate with insufficient evidence.
4. Task: decide whether to take an umbrella. Thought: current precipitation or a short-term forecast is needed. Action: call the weather tool with Beijing and today’s date. Observation: receive precipitation probability, time range, and data timestamp. Thought: apply a predefined rule. Finish: issue a recommendation tied to the observation. Success requires a non-stale Beijing forecast and execution of the rule; a failed tool call cannot be reported as success.

Next: Toolformer: learning when to call tools · Back to the landmark reading index