Toolformer: learning when and how to call tools

Original source

Timo Schick et al., 2023: Toolformer: Language Models Can Teach Themselves to Use Tools. This guide uses arXiv v1.

This is an original learning guide, not a reproduction. Copyright and the complete paper remain with the original source.

Learning objectives

By the end of this guide, you should be able to:

1. Reconstruct the complete pipeline from plain text through candidate position and call sampling, API execution, loss filtering, augmented-corpus construction, fine-tuning, and inference-time execution.
2. Explain why Toolformer compares three contexts: call plus result, no call, and a call without its result.
3. Calculate an illustrative loss improvement for a candidate call and decide whether it should enter the training set.
4. Separate learned model behavior, the runtime’s responsibility for actually executing an API, and the additional guarantees supplied by modern tool-calling platforms.

The idea in one sentence

Toolformer uses a handful of API demonstrations to make a base language model propose many calls in ordinary text, then keeps only calls that reduce future-token prediction loss enough to become useful self-generated training data.

The problem it actually addresses

Language models are strong text generators, yet simple tools routinely outperform them at exact arithmetic, recent facts, dates, translation, and long-tail lookup. Connecting an API does not by itself answer four decisions:

• When should the model call it? An API call has cost and is not useful after every phrase.
• Which tool is appropriate? Arithmetic calls for a calculator; a recent entity may require search.
• What arguments should be passed? The correct tool with the wrong expression or query still produces an unhelpful result.
• How should the result be used? The model must incorporate the return value into later text rather than ignore it.

Before Toolformer, one common route was to label these decisions extensively by hand. Another was to write a task-specific rule that invoked a known tool at a known point. The paper tests a more general route: give each API a few demonstrations, let the model propose annotations over a large corpus, and use the model’s existing language-modeling ability to decide which proposals are worth learning.

“Self-teaching” does not mean no humans or infrastructure

People still define the APIs, provide a few demonstrations, supply the plain-text corpus, execute every candidate call, and choose sampling and filtering thresholds. What becomes automatic is discovering and selecting useful call examples at scale—not API design, execution, or safety governance.

Objects and notation

Symbol or object	Meaning	Role in the pipeline
$M$	The language model being adapted	Proposes candidates, scores them with language-model loss, and is finally fine-tuned on the augmented corpus
$C$	A plain-text corpus	Supplies original token sequences with no tool annotations
$P(x)$	A few-shot prompt for one API	Demonstrates the tool’s call format and intended use
$c_i=(a_c,i_c)$	A candidate call inserted at position $i$	Contains API name $a_c$ and input $i_c$
$r_i$	The textual result of actually executing $c_i$	Reveals whether the call supplies useful information for the continuation
$e(c_i,r_i)$	A linearized call-and-result segment	For example, <API> Calculator(18/24) → 0.75 </API>
$C^{\ast }$	The corpus augmented with retained API segments	Fine-tunes $M$ with an ordinary language-modeling objective
${\tau }_s,{\tau }_f$	Sampling and filtering thresholds	Control the number of candidate positions and the minimum useful loss reduction

The paper requires every tool’s input and output to be representable as text and assumes that a few intended-use demonstrations are available. Its five tools are a question-answering system, Wikipedia search, a calculator, machine translation, and a calendar.

The six-stage mechanism

flowchart TD
    C["Plain corpus C"] --> P["1. Write a few demonstrations P(x) for each API"]
    P --> S["2. Sample candidate insertion positions and calls"]
    S --> E["3. Execute the APIs and obtain textual results r"]
    E --> L["4. Compare future-token losses"]
    L -->|"Improvement reaches threshold"| K["Keep the call and result"]
    L -->|"Insufficient improvement"| D["Discard the candidate"]
    K --> A["5. Merge calls into augmented corpus C*"]
    A --> F["Fine-tune with a standard LM objective"]
    F --> I["6. At inference, the model emits a call; runtime inserts the result and decoding resumes"]

Stage 1: define a tool’s language interface with a few demonstrations

The authors write one prompt per API showing how calls can be inserted into otherwise ordinary text. The demonstrations do not label answers for one downstream benchmark. They show textual situations in which that API might supply information needed to continue the passage. Candidates for different APIs are generated independently and merged later.

Conceptually, a call is linearized as:

<API> APIName(input) → returned text </API>

The implementation reuses existing token sequences for the delimiters and arrow rather than changing the model vocabulary. The important property is that the call, arguments, return value, and closing boundary all appear in a learnable text sequence.

Stage 2: sample positions first, then concrete calls

For every location $i$ in a text $x=x_1,\dots ,x_n$, the model estimates the probability of beginning an API segment after the current prefix:

$$p_i=p_M(\text{<API>}\mid P(x),x_{1:i-1})$$

Only positions with $p_i>{\tau }_s$ remain candidates. If there are too many, the system keeps the top $k$. At each retained position, it samples up to $m$ concrete calls using the preceding text and <API> as a prefix and the closing marker as the endpoint. A generation that never closes the call is discarded.

These are two separate decisions. Position probability asks whether a tool might be useful now. Call sampling proposes which tool and arguments to use. The output remains a noisy proposal set containing useful calls, irrelevant calls, bad arguments, and malformed generations. It is not yet supervision.

Stage 3: execute each proposal

The pipeline actually executes every candidate. Depending on the API, execution may invoke another neural model, run a Python calculation, or retrieve from a corpus. The result is converted into one text sequence $r_i$ that can be inserted into the training example.

This operation cannot be replaced by inspecting the call string. Until execution occurs, the pipeline does not know whether a query hits, whether a calculation succeeds, or whether the return value helps with the continuation.

Stage 4: filter by future-token loss

The paper calculates weighted cross-entropy over the original tokens following the candidate position. The loss when both call and result are available is:

$$L_i^{+}=L_i(e(c_i,r_i))$$

The negative baseline is the stronger—lower-loss—of two alternatives:

$$L_i^{-}=\min \left(L_i(ϵ),L_i(e(c_i,ϵ))\right)$$

• $L_i(ϵ)$ means there is no API call at all.
• $L_i(e(c_i,ϵ))$ exposes the call but withholds the returned value.

The candidate is retained only if:

$$L_i^{-}-L_i^{+}\ge {\tau }_f$$

In plain terms, the candidate must beat not only “no call” but also the strongest baseline in which the model sees the call without its result. This makes the returned information earn its place. If the call arguments already reveal the next words, or the result is irrelevant, the improvement should not clear the threshold.

What this signal really optimizes

The filter asks whether the result helps this base model predict original future tokens. It does not directly optimize factual correctness, end-user task success, safety, latency, or API cost. It is a clever but bounded proxy, not a universal definition of a good production tool decision.

Stage 5: construct $C^{\ast }$ and continue language modeling

Calls retained for all tools are merged and interleaved at their positions to create $C^{\ast }$. Apart from inserted API segments, $C^{\ast }$ retains the original text from $C$. The model is then fine-tuned with an ordinary language-modeling objective rather than a separate tool-classification head.

Within the same next-token objective, the model can learn when ordinary text should continue, when an API call should begin, how the name and arguments are written, and how to continue after a result. The many passages with no retained call implicitly teach the model not to invoke a tool everywhere.

Stage 6: the model initiates a call, but the runtime executes it

After fine-tuning, decoding proceeds normally. When the model emits the arrow indicating that it now expects an API result, the external runtime pauses generation, parses the tool and arguments, performs the call, inserts the result and closing marker, and lets the model resume its continuation.

Toolformer is therefore never a model acting without infrastructure. The model learns to emit a recognizable call; an executor remains a hard boundary. In the experiments, decoding was also modified so the API-start token could trigger when it was among the top $k$ choices, and each input was limited to at most one API call to prevent a call-only loop. Those are experimental settings, not a universal protocol every tool model must copy.

Work through one training example

Consider this original teaching passage:

A case contains 24 bottles and costs $18. Each bottle costs $0.75.

1. The model proposes candidates

Before $0.75, the model proposes:

Candidate A: <API> Calculator(18 / 24) → ? </API>
Candidate B: <API> Calendar() → ? </API>

Candidate A’s tool and arguments concern the number to be predicted. Candidate B may be perfectly well formed but unrelated to unit price.

2. The environment executes them

Result A: 0.75
Result B: Today is Sunday, August 9, 2026.

3. The filter calculates an illustrative improvement

Assume ${\tau }_f=1.0$. The following loss values are invented for teaching; they are not reported measurements from the paper:

Candidate	Better no-result baseline $L^{-}$	Call-and-result loss $L^{+}$	Gain $L^{-}-L^{+}$	Decision
Calculator	4.8	1.1	3.7	Keep
Calendar	4.8	4.6	0.2	Discard

The augmented text may become:

A case contains 24 bottles and costs $18. Each bottle costs
<API> Calculator(18 / 24) → 0.75 </API> $0.75.

The model can later learn to call a calculator in a similar exact-arithmetic context. This example does not teach it to call a calendar whenever numbers appear.

4. Why the retained set still contains noise

Future-token loss is a proxy. An irrelevant return can coincidentally correlate with later words and pass. A call useful to a real task might not improve the immediate continuation and can be rejected. The paper’s qualitative analysis likewise shows that filter scores and human judgments are not perfectly aligned. Automated selection reduces annotation labor; it does not produce noiseless gold labels.

Teaching pseudocode

def build_augmented_corpus(model, corpus, api_specs,
                           sample_threshold, filter_threshold):
    kept = []
 
    for text in corpus:
        annotations = []
        for api in api_specs:
            positions = sample_positions(
                model, text, api.demonstrations, sample_threshold
            )
            calls = sample_calls(model, text, positions, api)
 
            for position, call in calls:
                result = api.execute(call.arguments)
                loss_with_result = future_loss(
                    model, text, position, call=call, result=result
                )
                loss_without_call = future_loss(
                    model, text, position, call=None, result=None
                )
                loss_call_only = future_loss(
                    model, text, position, call=call, result=None
                )
                baseline = min(loss_without_call, loss_call_only)
 
                if baseline - loss_with_result >= filter_threshold:
                    annotations.append((position, call, result))
 
        kept.append(interleave(text, merge_compatible(annotations)))
 
    return kept
 
 
def generate_with_tools(model, prompt, runtime):
    tokens = []
    while not model.is_finished(tokens):
        token = model.next_token(prompt, tokens)
        tokens.append(token)
        if token == "ARROW_EXPECTING_RESULT":
            call = parse_open_api_call(tokens)
            result = runtime.execute(call.name, call.arguments)
            tokens.extend(encode_result_and_close(result))
    return decode(tokens)

build_augmented_corpus is the offline data pipeline. generate_with_tools is the online inference pipeline. Keeping them separate prevents the most common misunderstanding: the central contribution is learning call behavior from self-generated data, but inference still requires a tool runtime.

How to read the original evidence

The paper uses a 6.7B-parameter GPT-J as its primary base model and evaluates five tools on question answering, mathematics, fact completion, and time-aware tasks. The most informative evidence is mechanistic rather than one headline score:

• With tools enabled, the Toolformer-trained model improved over corresponding no-tool settings on multiple zero-shot tasks and beat a larger GPT-3 baseline on several of the tasks examined in the paper.
• When APIs were disabled at inference, training on the API-augmented corpus did not add a language-model perplexity penalty relative to continued training on the same plain corpus. This supports the limited claim that the inserted tool examples did not obviously destroy general text modeling.
• Across the GPT-2 scales tested in the paper, the smaller models did not derive clear benefit from tools, and useful tool behavior emerged at roughly the 775M scale. That is an observation under this architecture and setup, not a permanent parameter threshold for all models.
• Expanding the decoding trigger for the API-start token greatly increased call frequency but could degrade calibration. “Can call” and “calls only when useful” are distinct evaluation targets.

Evidence boundary

The experiments use fixed 2023-era models, textual APIs, sampling thresholds, and constraints such as one call per input. They show that this self-supervised pipeline can teach selective use of a tool set. They do not establish multi-tool planning, production reliability, or effortless transfer to arbitrary new APIs.

Limitations stated by the paper

1. No tool chains. Candidates for each API are generated independently, so the training set does not feed one tool’s output into another tool.
2. No interactive browsing. Search results are reduced to one text response. The model cannot page through many results or reformulate a query after inspecting the first response.
3. Sensitivity to wording. Small changes in input phrasing can change whether the model calls an API.
4. Potentially poor sample efficiency. The paper notes that processing more than a million documents can yield only a few thousand useful calculator calls.
5. No cost in the filter. Usefulness is determined by future-token loss, without latency, price, or failure risk.
6. A one-call evaluation limit. The experiments allow at most one API call per input to prevent endless calling. Naturally, this excludes many complex tasks.

These limitations explain why Toolformer does not replace a runtime loop such as ReAct. It can learn “a query is useful here” without demonstrating how to inspect that observation and plan a different second tool call.

Boundary with modern tool calling

What Toolformer directly addresses	What a production runtime must still provide
Learning timing, tool choice, argument text, and result continuation from data	JSON Schema, type validation, authorization, and user confirmation
Executing real APIs while generating training examples	Timeouts, retries, idempotency, rate limits, and concurrency
Filtering calls with language-model loss	Task-success evaluation, cost budgets, safety policy, and auditing
Triggering execution with textual markers	Structured protocols, error classes, and isolation of untrusted returns

Modern function calling often asks a model to emit a structured object and lets application code decide whether to execute it. Toolformer uses linearized text and fine-tuning to learn the call. Both preserve the boundary “model proposes, external system executes, result returns to context,” but their training method, protocol, and reliability guarantees differ.

Check your understanding

1. Why can a sampled API call not become a positive example immediately? Why must it be executed and filtered first?
2. Why does the filter compare against a condition that shows the call but withholds the result?
3. If a search call lowers future-token loss but returns stale information, will Toolformer’s original objective necessarily reject it? Why?
4. Design one useful and one well-formed but useless Calendar call. State the loss behavior you expect.
5. Which layer of a tool system is addressed by Toolformer, and which by ReAct?

Suggested answers

1. Sampling produces bad arguments, irrelevant tools, malformed calls, and calls with no useful result. Execution supplies the actual return value; filtering tests whether that return helps the continuation.
2. It rules out cases where the call text itself already gives away the continuation. A retained result must beat the stronger of no call and call-only contexts.
3. No. The objective measures future-token prediction, not factual freshness. If stale information matches the corpus continuation, it may pass. Production data therefore needs separate truth and recency checks.
4. Useful: a sentence is about to state tomorrow’s weekday, so a Calendar return of today’s date reduces the continuation loss. Useless: a product description is about to state its color, and a Calendar result does not help predict that color.
5. Toolformer primarily learns call timing, choice, and arguments through data and training. ReAct primarily organizes inference-time planning around successive environmental observations. They can be combined, but neither substitutes for the other.

Previous: ReAct · Next: Generative Agents · Back to the landmark reading index