Reflexion: learning from failure through verbal feedback

Original source

Noah Shinn et al., 2023: Reflexion: Language Agents with Verbal Reinforcement Learning. This guide follows arXiv v4.

This is an original learning guide, not a reproduction of the paper. Copyright and the complete work remain with the original source.

What you should be able to do after this guide

By the end, you should be able to:

1. Specify separate inputs and outputs for the Actor, Evaluator, and Self-Reflection modules instead of calling all three “self-reflection.”
2. Draw the data flow between a trial’s short-term trajectory, reflection memory across trials, and the next policy attempt.
3. Explain why Reflexion does not update model weights and how a changed context can still change behavior.
4. Distinguish the reliability boundaries of environmental rewards, task heuristics, model judgments, and self-generated tests.
5. Turn one failure into a three-part lesson: verified fact, inferred cause, and executable next change.
6. Define explicit rules for retry limits, reflection-memory capacity, success, and escalation to a human.

Start with the real problem: retrying is not learning

Suppose a coding agent writes a function that fails its tests. The simplest retry sends it the same task again. If the second attempt sees essentially the same context, it may repeat the same assumption. Another approach is to paste the entire execution log back into the prompt, but the useful failure signal can disappear inside a long, noisy trajectory.

Conventional reinforcement learning can change a policy through many interactions and parameter updates. For a large language model, that process may be expensive and unavailable at inference time. Reflexion takes a lighter path: translate a reward or failure signal into a concrete verbal experience and place that experience in episodic memory visible to the next attempt.

The unit of learning is therefore not a new set of weights. It is a text record of what happened, why the attempt may have failed, and what should change next.

The mechanism in one sentence

The Actor produces a trajectory, the Evaluator determines a checkable outcome, and Self-Reflection compresses the trajectory and outcome into an actionable verbal lesson; that lesson enters cross-trial memory, so the same model parameters can condition the next attempt on a different context.

flowchart TD
    T["Task + current reflection memory"] --> A["Actor: generate answer or trajectory τ"]
    A --> E["Environment / Evaluator: reward, pass/fail, or feedback"]
    E --> S{"Accepted?"}
    S -->|"Yes"| F["Stop and hand result to external acceptance"]
    S -->|"No"| R["Self-Reflection: trajectory + feedback + earlier lessons"]
    R --> M["Write lesson into episodic memory mem"]
    M --> B{"Retry budget or escalation limit reached?"}
    B -->|"No"| A
    B -->|"Yes"| H["Stop, report failure, or request human judgment"]

The contract: three modules and two kinds of memory

Actor: execute the task, but do not certify success

The Actor receives the task, observations from the environment, the current trial’s history, and any cross-trial reflections. It outputs text, code, or actions. The paper experiments with Actor styles such as Chain-of-Thought and ReAct; Reflexion itself does not require one fixed prompting pattern.

An interactive Actor produces a trajectory such as:

$${\tau }_t=[a_0,o_0,a_1,o_1,\dots ]$$

The actions and observations belong to the short-term memory of the current trial. When the environment resets, the full trajectory does not have to be copied into the next trial.

Evaluator: turn an outcome into a signal that can trigger learning

The Evaluator reads the Actor’s final output or full trajectory and returns a reward, pass/fail decision, score, or verbal feedback. The paper uses different evaluators for different tasks:

• deterministic rules such as exact match for some reasoning tasks;
• environment completion signals and predefined heuristics for interactive tasks;
• another language model acting as a classifier or judge in some settings;
• compilers, interpreters, executed tests, and self-generated tests for programming.

Its result determines whether the loop stops and what evidence the reflection can use. The Evaluator is not an optional “score this answer” prompt. Without a trustworthy outcome signal, the system has no trustworthy direction for improvement.

Self-Reflection: expand a sparse reward into a semantic direction

Self-Reflection receives the task, failed trajectory, Evaluator signal, and previous lessons. It outputs a short piece of text aimed at a future attempt. An environment might return only 0, while the reflector can infer from the trajectory: “I assumed the object was in the first drawer; next time search the room first and confirm the pickup before proceeding.”

The paper describes such text as analogous to a semantic gradient. It is not a mathematical gradient and does not guarantee an optimum. Its value is that it points to a step, check, or strategy that the next attempt can actually change.

Do not confuse a short-term trajectory with long-term experience

Memory	What it stores	Lifetime	Purpose
Short-term memory	Fine-grained actions and observations from the current attempt	Within one trial	Locate the concrete failure in this attempt
Long-term episodic memory	Compressed lessons produced by Self-Reflection	Across trials	Expose previous lessons to the next strategy

The paper keeps the long-term window deliberately small, usually one to three reflections; its ALFWorld configuration retains the three latest reflections. This matched the context-window constraints of the experiments and also prevented advice from growing without bound. A production system still needs explicit policies for deduplication, expiry, conflicts, versioning, and task isolation.

Why this is not a conventional weight update

Let the Actor’s language-model parameters be $M_a$ and the cross-trial memory be mem. Reflexion changes mem; it does not backpropagate into $M_a$. The next output can differ because its conditioning context has changed:

same task + same model parameters + empty memory
    → first strategy
 
same task + same model parameters + “previous failure and next change”
    → second strategy conditioned on the lesson

This has three practical advantages: it requires no fine-tuning, language can carry highly specific advice, and a human can inspect the stored experience. The trade-offs are equally important. Remove the context and the improvement is not consolidated in the weights; a bad reflection can directly poison the next decision; and the loop has no convergence guarantee.

How to interpret “verbal reinforcement”

Reflexion borrows the language of trials, rewards, and policy improvement from reinforcement learning, but implements strategy revision through context and episodic memory. Do not report its result as “the model updated a new policy through reinforcement-learning weight training.”

Complete control flow: from one trial to the next

1. Initialize the Actor, Evaluator, Self-Reflection module, and empty or existing mem.
2. The Actor uses the task and mem to produce trajectory ${\tau }_t$.
3. The Evaluator checks the trajectory or result and returns $r_t$ plus an acceptance decision.
4. If accepted, stop retrying and pass the result to the task’s real acceptance process.
5. If rejected, send ${\tau }_t$, $r_t$, and mem to Self-Reflection.
6. Generate reflection $s{r}_t$ and store it in mem; when the capacity is exceeded, compress or evict according to policy.
7. If retry budget remains, reset the task environment and begin a new trial.
8. Stop on the maximum trial count, repeated failure, or a risk condition that requires human review.

def reflexion_loop(task, actor, evaluator, reflector, max_trials=3):
    memory = []
 
    for trial in range(max_trials):
        trajectory = actor.run(task, lessons=memory)
        evaluation = evaluator.check(task, trajectory)
 
        if evaluation.passed:
            return {
                "status": "passed",
                "trajectory": trajectory,
                "evidence": evaluation.evidence,
            }
 
        lesson = reflector.summarize(
            task=task,
            trajectory=trajectory,
            feedback=evaluation,
            previous_lessons=memory,
        )
        memory = keep_recent_nonduplicate(memory + [lesson], limit=3)
 
        if evaluation.requires_human_judgment:
            break
 
    return {"status": "not_passed", "lessons": memory}

Notice the stopping contract: either success or budget exhaustion must end the loop. The paper’s pseudocode communicates iteration around a failed attempt; an implementation must spell out and test its Boolean boundaries instead of accidentally copying an infinite retry loop.

The feedback source sets the ceiling for reflection

External environments and tests: closest to task facts

Examples include whether an object actually reached the requested location, whether a unit test passed, or whether an API returned the required state. Such signals may be sparse, but they are usually more reliable than a model’s intuition about its own answer. The reflector should interpret the evidence, not overwrite it.

Predefined heuristics: good at known bad patterns

In ALFWorld, the paper uses rules to detect repeated action-observation pairs and overly long, inefficient trajectories before triggering reflection. Heuristics are cheap and reproducible, but they cover only failures their designer anticipated.

LLM evaluation: broad coverage, correlated mistakes

Another model can classify an answer or write a critique when no simple rule captures the task. It can also share the Actor’s false premise, prefer persuasive wording over a correct result, or drift as prompts and models change.

Self-generated tests: useful feedback that may encode the wrong specification

A coding agent can generate tests and execute its candidate without seeing hidden tests. That creates active feedback, but the tests and implementation may make the same misunderstanding together. Independent hidden tests or maintainer acceptance must still decide completion.

Feedback source	Strength	Main risk	Appropriate role
Real environment completion	Close to the actual outcome	May expose only a binary signal	Primary success criterion
Unit or integration tests	Repeatable and localizable	Incomplete coverage	Main programming evidence
Heuristics	Fast and stable	Miss unknown failure modes	Diagnostic trigger and early stop
LLM Evaluator	Handles semantic tasks	Confident misjudgment and preference drift	Calibrated auxiliary judgment
Self-generated tests	Actively searches for counterexamples	Tests and code can share one mistaken spec	Exploration evidence, not final truth

Work through one example: repairing a bracket checker

This simplified teaching example is not one of the paper’s HumanEval samples.

The task asks balanced(s) to determine whether parentheses are correctly nested. On the first attempt, the Actor only compares counts:

def balanced(s):
    return s.count("(") == s.count(")")

Trial 1: trajectory and evaluation

• Actor assumption: equal totals are sufficient.
• Self-generated tests: "()" and "(())", both pass.
• Independent Evaluator test: ")( " after ignoring the space fails because a closing parenthesis appears first.
• Reward: 0, together with the failing input and assertion.

Self-Reflection: separate fact, inference, and action

Verified fact: on input ")(", the two counts are equal, but the expected
result is False and my function returns True.
 
Inferred cause: I checked only the final totals and ignored prefix balance;
a closing parenthesis cannot appear before an unmatched opening one.
 
Next change: scan left to right with a balance counter; increment for '(',
decrement for ')', fail whenever balance becomes negative, and require zero
at the end.

That compact lesson enters long-term episodic memory. The system does not indiscriminately copy the whole runtime log.

Trial 2: condition the Actor on the lesson

def balanced(s):
    balance = 0
    for char in s:
        if char == "(":
            balance += 1
        elif char == ")":
            balance -= 1
            if balance < 0:
                return False
    return balance == 0

The Evaluator reruns both target and regression tests; only an all-passing result ends the loop. The essential change is not that “the model thought longer.” The second context contains an executable constraint derived from the first failure.

How trajectory feedback works in decision tasks

In a text environment such as ALFWorld, an Actor can use ReAct to produce actions. The environment often exposes an unambiguous success signal only at task completion. Intermediate failure may instead appear as repeated actions with unchanged observations, or as a mismatch between what the Agent assumes it holds and the actual environment state.

In the paper’s setup, heuristics trigger reflection when the same action-observation loop repeats beyond a threshold or the trajectory grows too long. Reflexion then:

1. searches the full trajectory for the earliest faulty assumption;
2. records an environment state to verify or an alternative action for the next trial;
3. retains only a small number of recent lessons; and
4. resets the environment before beginning a new trial.

This distinction matters: backtracking within a trial and learning between trials are not the same operation. The paper’s core contribution is the second one—compressing an ended episode into experience that changes a fresh attempt.

How to read the evidence

The paper evaluates the framework on sequential decision-making in ALFWorld, language reasoning in HotPotQA, and programming tasks including HumanEval, MBPP, and LeetCodeHardGym. It also compares feedback signals, ways of incorporating feedback, and Actor types.

In its historical arXiv v4 setup, the paper reports 91% pass@1 on HumanEval, above the cited GPT-4 result of 80%, and reports improvements in ALFWorld and HotPotQA. These results support a bounded claim: under the reported models, prompts, evaluators, and iteration budgets, verbal reflection memory improved later attempts.

Evidence boundaries

• These are results from a 2023 experimental setup, not a current model leaderboard.
• Tasks use different evaluators, heuristics, and tools, so one number is not a universal Reflexion gain.
• Multiple trials consume additional inference and execution budget. A fair comparison should report attempts and total cost alongside pass rate.
• Self-generated tests and LLM judgments are not independent ground truth; hidden tests do not necessarily cover the whole specification either.
• A readable reflection is not proof that the system recovered an internal causal explanation. It is a strategy prompt available to the next attempt.

Why reflection can fail

Failure mode	Example	Consequence	Mitigation
Evaluator error	A model judge rejects a correct answer	The correct strategy is discarded	Prefer deterministic external evidence; calibrate judges
Wrong credit assignment	It blames the final action, but the first state transition was wrong	The next trial fixes a symptom	Replay transitions and find the earliest divergence
Non-actionable advice	“Be more careful next time”	The Actor has no behavior to change	Name a tool, check, or concrete decision
Facts mixed with inference	One boundary failure is called proof that the entire algorithm is wrong	Unsupported belief enters memory	Label facts, causes, and actions separately
Conflicting lessons	One says “use cache,” another says “never cache”	Conditioning becomes unstable	Scope, version, deduplicate, and resolve conflicts
Oversized memory	Every trajectory and many reflections are retained	Useful advice is diluted	Small capacity, summarization, and task retrieval
Infinite retry	Each attempt changes only surface wording	Cost grows without progress	Trial limit, repeat detection, and human escalation
Gameable success signal	The Agent deletes a test or bypasses a check	False success	Isolated evaluation, regression tests, and audit logs

Historical contribution and today’s engineering interpretation

The original contribution was to compose an Actor, Evaluator, Self-Reflection module, and cross-trial verbal episodic memory into a modular loop, demonstrating that an agent could exploit trial feedback without fine-tuning its language model.

Engineering lessons that remain useful today include:

• convert each failure into a small, executable lesson instead of blindly retrying or replaying every log;
• let external acceptance decide whether the task succeeded, while reflection proposes what to change;
• record environmental facts, causal inferences, and action suggestions as different fields;
• put hard limits on memory, trials, repeated failure, and escalation;
• retain provenance for every lesson—task version, tests, and trajectory—so it cannot silently contaminate another task.

The paper does not establish that model self-evaluation is inherently reliable, that persuasive reflection reveals the true root cause, that the lesson has been permanently learned in the weights, or that unlimited retries will eventually solve every task.

How it differs from neighboring readings

Reflection in Generative Agents synthesizes daily experiences into beliefs about a character and its relationships. Reflexion’s reflection directly serves the next task attempt. Voyager additionally stores environment-verified successful behavior as retrievable, executable skills.

Check that you can apply the mechanism

1. What should the Actor, Evaluator, and Self-Reflection module each not be responsible for?
2. Why can reward=0 not tell the Actor what to change, and what operation does Reflexion add?
3. In “the compiler reports a type error on line 18, so the algorithm is wrong and must be rewritten,” which parts are fact, inference, and proposed action?
4. If three trials produce the same failure, should the system retry, rewrite the reflection, or escalate? What evidence should decide?
5. Why is passing self-generated tests insufficient for final acceptance?

Reference answers and judgment criteria

1. The Actor should not certify its own success; the Evaluator should not invent a root cause without evidence; Self-Reflection should not rewrite environmental facts. Their responsibilities are execution, outcome judgment, and future-oriented lesson generation.
2. A scalar describes a bad outcome but performs no credit assignment. Self-Reflection combines the trajectory and feedback into a language direction naming a concrete action or check, then stores it for the next trial.
3. “The compiler reports a type error on line 18” is verifiable fact. “The algorithm is wrong” is an under-supported causal inference. “Rewrite it” is a proposed action.
4. First confirm the evaluator is reliable, the failures truly share one cause, and the lesson reached the Actor’s context. Repeated trajectories normally trigger deduplication and escalation rather than unlimited spending; retry only when there is a genuinely new, evidence-based action.
5. The implementation and its generated tests may share the same misunderstanding, and generated coverage is limited. Final acceptance needs independent tests, a real environment, or a maintainer-defined contract.

Previous: Generative Agents · Next: Voyager · Back to the landmark reading index