Voyager: automatic curriculum, skill library, and verifiable open-ended learning
Original source
Guanzhi Wang et al., 2023: Voyager: An Open-Ended Embodied Agent with Large Language Models. This guide uses arXiv v2.
This is an original learning guide, not a reproduction. Copyright and the complete paper remain with the original source.
What you should be able to do after this guide
By the end, you should be able to do more than list Voyager’s three modules—automatic curriculum, skill library, and iterative prompting. You should be able to:
- Explain why the curriculum proposes a particular next task from the agent’s current state, successes, and failures.
- Trace the full state transition from task proposal through skill retrieval, code generation, execution, feedback, verification, and storage.
- Distinguish the jobs of environment feedback, program-execution errors, and self-verification.
- Explain why a skill library is neither just a vector database nor a complete chat-history archive.
- State exactly which high-level Minecraft-control setting the evidence supports and what is still missing before transferring the idea to a real system.
Start with the problem: how do you learn in a world with no finish line?
An ordinary task usually comes with a clear target, such as “fix this failing test” or “book this flight.” Minecraft is different. There is no single final assignment: an agent may gather, craft, explore, or fight. If all it receives is “discover as many things as possible,” three problems appear quickly:
- The goal is too broad. The agent does not know whether to collect wood or search for diamonds, and attempting a hard objective too early causes repeated failure.
- Experience does not accumulate. Even after crafting a wooden pickaxe once, the model may generate the same behavior from scratch next time.
- Failure is hard to diagnose. Code without a syntax error may still fail the task. The agent may gather the wrong material, fail to reach a target, or mistakenly declare success.
Voyager’s central contribution is not merely “GPT-4 plays a game.” It closes three engineering loops required for open-ended learning: who decides what to learn next, how a success becomes an executable asset, and how the environment proves that the new behavior was actually learned.
First, establish the vocabulary:
- An automatic curriculum proposes the next task from the current capability frontier; it is not a fixed lesson plan.
- A skill in the paper is primarily a verified executable JavaScript program that expresses behavior over multiple time steps.
- The skill library stores skill descriptions, a vector index, and the program bodies themselves.
- Self-verification uses another GPT-4 role to judge task completion from the task and current state, and to critique a failed attempt.
- Open-ended learning has no predetermined endpoint: the system keeps discovering new objectives and extending its capability boundary.
“Learning” here does not mean updating model weights
Voyager calls GPT-4 as a black-box API and accumulates capability in prompts and an external skill library. It performs no gradient training or parameter fine-tuning on GPT-4. “In-context lifelong learning with executable external memory” is the more precise description.
The mechanism contract: inputs, outputs, and state
| Part | What it contains in Voyager | Why it matters |
|---|---|---|
| Inputs | Inventory, equipment, nearby blocks and entities, biome, time, health, hunger, and position | Curriculum and code generation must know what exists now and where the agent is |
| History | Completed tasks, failed tasks, and exploration progress | The next objective must reflect what the agent can already do and where it just failed |
| Persistent state | Verified skill programs and embeddings of their descriptions | Successful experience survives across tasks and can be retrieved and composed |
| Working state | Current task, candidate code, environment feedback, interpreter errors, verifier critique, and attempt count | This state changes inside one task-solving loop |
| Outputs | A completed task, a verified program, and a new description/index entry | A round produces reusable capability, not merely another chat transcript |
| Stop condition | Verification succeeds, or four code-generation rounds are exhausted | Success stores the skill and advances; failure defers the task for a later curriculum decision |
The complete control flow is:
flowchart TD S["Read world state plus completed and failed tasks"] --> C["Automatic Curriculum: propose the next objective"] C --> Q["Retrieve Top-5 skills from task plan and environment feedback"] Q --> G["GPT-4 generates or revises an executable program"] G --> X["Execute code through high-level Mineflayer APIs"] X --> F["Collect environment feedback and interpreter errors"] F --> V["Self-Verification: judge completion and produce critique"] V -->|"Success"| K["Describe the skill and store its program"] K --> S V -->|"Failure, fewer than four rounds"| G V -->|"Failure after four rounds"| B["Record failure and defer the task"] B --> S
Two gates in this diagram are easy to miss: candidate code enters the library only after successful verification, and repeated failure does not cause an infinite retry loop—the curriculum receives control again.
Core mechanism 1: the automatic curriculum searches the capability frontier
The automatic curriculum is not a random task generator. The paper places several components in the curriculum prompt sent to GPT-4:
- The overarching goal and constraints—for example, discover diverse things without choosing a task that is clearly too difficult right now.
- The current world state, including inventory, equipment, local environment, and survival status.
- Completed and failed tasks, which estimate exploration progress and the present frontier.
- Additional context produced by having GPT-3.5 ask and answer questions about the current state; the authors use the cheaper model for these standard language operations.
You can think of the curriculum objective as choosing something novel, slightly beyond current ability, yet reachable with available resources and learned skills. A task that is too easy only repeats experience. One that is too hard burns the prompting budget. A random curriculum may ask a newly spawned agent to obtain a diamond immediately.
A failure is not a permanent ban. A task may be impossible now only because a prerequisite resource or skill is missing. Voyager allows at most four code-generation rounds on one task. If those attempts do not succeed, it switches objectives, and the curriculum may propose the task again after the agent has grown. That creates an “attempt, detour, improve, retry” path through an open world.
Core mechanism 2: the skill library stores executable behavior
After a program passes verification, GPT-3.5 produces a description. The embedding of that description becomes the retrieval key, while the program itself is the value. For a new task, the system first produces a general solution suggestion, combines the task plan and environment feedback into a query, and places the five most relevant skills in the code-generation prompt.
The library therefore has three layers:
| Layer | Job | What it does not replace |
|---|---|---|
| Description | Says what a skill does and supports query construction | It does not prove that the program works |
| Vector index | Finds semantically relevant candidates among many skills | It neither executes code nor guarantees applicability |
| Program body | Performs the behavior and can be called by new programs | It still depends on the current environment, control APIs, and verification |
The paper chooses code as the action space rather than making the model emit one primitive action at a time. Programs can contain loops, branches, and function calls, so they naturally represent behavior over time. A complex skill can call simpler skills. For example, “craft an iron pickaxe” may compose previously stored programs for collecting wood, making a crafting table, crafting a stone pickaxe, mining, and smelting.
“Composable” is not magic, however. The composed program still has to run in the current world and pass verification again. An old skill may fail because resources, terrain, APIs, or preconditions differ. Vector retrieval supplies candidates, not a correctness certificate.
Core mechanism 3: three feedback channels make generation repairable
Voyager does not require the model to write correct code in one shot. After execution, it collects three kinds of feedback with different meanings:
- Environment feedback describes intermediate progress or a missing world condition—for example, that seven more iron ingots are required. It answers, “Why did the world not reach the target state?”
- Execution errors expose syntax errors, invalid operations, and nonexistent API calls. They answer, “Why did the program not run correctly?”
- Self-verification and critique ask another GPT-4 instance to judge success from the task and current state. On failure, it suggests how the next round should change. It answers, “Even if the program finished, did the task actually finish?”
Looking only at exceptions misses “the program exited normally but achieved nothing.” Looking only at the final inventory may hide the exact line where execution failed. The three signals together connect symptom, cause, and acceptance.
Self-verification is not objective truth
The paper reports verifier failures, including a case where spider string was not recognized as evidence that a spider had been defeated. Production systems should prefer executable assertions, tests, structured state queries, and human approval when appropriate. Model judgment should cover only what cannot be programmed reliably.
A complete run: from owning a wooden pickaxe to learning “craft a stone pickaxe”
The following is a simplified teaching trace, not a verbatim paper log. Every step nevertheless maps to the published mechanism.
| Round | State observed by the system | Action | State transition |
|---|---|---|---|
| 0 | Inventory has planks and a wooden pickaxe; “collect wood” succeeded; “find a diamond immediately” previously failed | Curriculum proposes “craft a stone pickaxe” | A reachable current objective replaces another leap to diamonds |
| 1 | Library contains mineWoodLog and craftCraftingTable | Retrieve skills and generate a mining-and-crafting program | Candidate code exists but is not stored |
| Execution 1 | No directly mineable stone is nearby | Program runs; feedback reports insufficient cobblestone | There is no syntax error, but the goal state is absent |
| Verification 1 | Inventory contains no stone pickaxe | Verifier rejects the attempt and recommends finding exposed stone first | Critique becomes input to the next round |
| 2 | Previous code, feedback, and critique are all in context | Add exploration and path handling, then reuse collection skills | The agent gathers cobblestone and crafts the item |
| Verification 2 | A stone pickaxe appears in inventory | Verifier accepts the task | Create a reusable description and store the program |
| Next task | Completed history and the library both grew | Curriculum may now propose “collect iron ore” | The capability frontier moves outward |
The important event is not that the second program looks elegant. It is that failure information survives, acceptance occurs after execution, the successful program enters persistent memory, and the curriculum upgrades the next objective from that new state.
Pin down the control flow with pseudocode
completed, failed, skill_library = [], [], []
while exploration_budget_remaining():
state = observe_world()
task = curriculum.propose(state, completed, failed)
feedback = errors = critique = None
for attempt in range(4):
plan = make_general_plan(task, state, feedback)
skills = skill_library.retrieve(plan, feedback, top_k=5)
program = generate_program(
task=task,
state=state,
retrieved_skills=skills,
previous_feedback=feedback,
previous_errors=errors,
previous_critique=critique,
)
feedback, errors = execute_in_minecraft(program)
state = observe_world()
success, critique = verifier.check(task, state)
if success:
skill_library.add(describe(program), program)
completed.append(task)
break
else:
failed.append(task)The pseudocode deliberately shows two state layers. feedback/errors/critique serve the short loop for one objective. completed/failed/skill_library persist across objectives and drive long-term exploration.
Reading the evidence: support for a mechanism, not proof of general autonomy
The paper evaluates Voyager in a Minecraft environment built on MineDojo, using Mineflayer JavaScript APIs for high-level control. ReAct, Reflexion, and AutoGPT are adapted to that setting. The authors report that, within up to 160 prompting iterations, Voyager discovers 63 unique items—3.3 times the baselines—travels 2.3 times farther, and reaches some technology-tree milestones up to 15.3 times faster when measured in prompting iterations. Tests in a new world with an empty inventory also show reuse of the learned skill library on unseen tasks.
The ablations speak more directly to the mechanism. Replacing the automatic curriculum with a random one reduces discovered items by 93 percent. Removing self-verification reduces them by 73 percent. The version without a skill library tends to plateau later. The defensible conclusion is that all three modules contribute to sustained progress in the authors’ setting—not that the same percentages transfer to another domain.
The evidence boundary matters just as much:
- The text-agent baselines are reinterpreted and adapted for Minecraft rather than used unchanged as universal systems.
- The agent acts through a high-level API; the experiment does not compare pixel perception, 3D vision, or low-level motor control.
- Key technology-tree results use only a small number of repeated trials, so one game cannot establish performance on robotics or enterprise work.
- API cost, model snapshots, and the Minecraft environment are historical experimental conditions, not permanent performance guarantees.
Failure modes and a useful debugging order
| Symptom | Likely cause | First check |
|---|---|---|
| Curriculum proposes an item that does not exist | Model hallucination or insufficient world-rule constraints | Validate candidate tasks against a capability catalog or rules engine |
| Code repeatedly calls nonexistent functions | Weak control-API description or code-generation hallucination | Expose the real schema and return interpreter errors verbatim |
| Program exits normally but the objective is unmet | The system checks process completion instead of world state | Define an explicit success assertion for the task |
| A bad skill enters the library and keeps contaminating later work | Verifier error or a weak admission gate | Re-verify, version skills, add regression scenarios, and support invalidation |
| The agent loops over trivial tasks | Curriculum lacks novelty or frontier information | Inspect completed history, failed history, and difficulty policy |
| A complex program fails when it calls an old skill | Preconditions, environment, or API version changed | Record applicability, dependencies, and failure semantics with each skill |
Historical contribution versus today’s engineering interpretation
What the 2023 paper directly proposed and evaluated was the combination of an automatic curriculum, an executable skill library, and iterative prompting with three feedback channels in an open-ended Minecraft lifelong-learning agent—without updating the black-box LLM’s weights.
A useful engineering interpretation today is to design long-running agents with two loops. The inner loop uses real execution feedback to complete one objective correctly. The outer loop selects the next reachable objective from the current capability state. Only accepted artifacts become reusable assets. That pattern can be mapped to data analysis, repository maintenance, or operations, but the mapping does not mean the paper evaluated those domains.
Real systems must also add what the game setting minimizes: authorization, approval for irreversible actions, cost limits, concurrent state, skill versions, dependency pinning, privacy, and human takeover. In particular, “another model said it succeeded” cannot be the only gate before payment, deletion, or production deployment.
Check your understanding
- Why must the automatic curriculum see current state, completed tasks, and failed tasks? What goes wrong when each one is missing?
- Why is a highly relevant vector-search result still not safe to execute and accept automatically?
- Give one failure that each of environment feedback, execution errors, and self-verification can detect.
- What happens after Voyager fails one objective for four code-generation rounds, and why is that better than retrying forever in an open world?
- If you transfer Voyager’s pattern to a coding agent, which mechanisms map directly and which safety conditions must be added?
Answer key and reasoning cues
- Current state defines available resources, completed history prevents repetition and marks capability, and failed history reveals the present frontier. Omitting them respectively yields infeasible goals, repeated learning, or repeated collisions with the same failure.
- Similarity only makes a skill a candidate. Environment, preconditions, dependencies, and versions may differ, so a composed behavior must run and pass verification again.
- Environment feedback catches missing materials or progress; execution errors catch syntax, API, and invalid-operation failures; self-verification catches a program that completed without completing the task.
- Voyager records the failure and returns control to the curriculum. It may retry after gaining capability, which bounds cost and avoids a local infinite loop.
- A curriculum can become selection of the next reachable issue, the library can hold tested scripts or repair patterns, and feedback can come from tests and logs. Repository versions, sandboxing, permissions, approval, rollback, and supply-chain constraints must be added.
Previous: Reflexion · Next: AutoGen · Back to the landmark reading index