Errors, retries, and data consistency

Corresponding source

Subject	Source
Provider retries and error metadata	nanobot/providers/base.py, nanobot/providers/fallback_provider.py
Runner requests and Tool errors	nanobot/agent/runner.py::_request_model, _run_tool
Turn-level containment	nanobot/agent/loop.py::_dispatch
Atomic Session save	nanobot/session/manager.py::SessionManager.save
Atomic Cron write	nanobot/cron/service.py::_atomic_write
Memory and configuration writes	nanobot/agent/memory.py, nanobot/config/loader.py::save_config

Learning objectives

After this lesson, you should be able to:

• distinguish Provider retry, model-loop retry, Tool error, and entry-layer error;
• explain why visible streaming constrains safe retry;
• assess the durability of Session, Cron, Memory, and configuration writes;
• diagnose a failure by asking which layer produced it.

Prerequisites

• Understand AgentRunner’s model-tool loop.
• Understand Session files and replay boundaries.
• Know idempotency: repeating the same operation should not add unintended side effects.

1. “Retry” is not one mechanism

Different retries in nanobot have different goals:

Layer	Trigger	Behavior
Provider request	429, temporary network or service failure	LLMProvider._run_with_retry() waits and retries
Provider fallback	current Provider/Preset unavailable	fallback wrapper switches backend
Runner empty response	model returns neither content nor Tool Call	request again a bounded number of times
Runner finalization	tool/budget boundary reached without a usable final answer	request again with a no-tools finalization prompt
Tool loop	Tool returns recoverable error text	give result to model so it can change approach
Gateway/Channel	connection or send failure	use the Channel/service’s own strategy

“The system retries automatically” is not precise enough. Ask: who retries what, can it repeat a side effect, and what stops it?

2. Provider retry decisions

LLMProvider.chat_with_retry() and chat_stream_with_retry() eventually enter _run_with_retry(). It classifies transience from structured error fields and content:

• LLMResponse.error_should_retry
• error_status_code
• error_code
• error_retry_after_s
• retry_after
• recognized statuses such as HTTP 429

Explicit quota, balance, or billing failures do not recover merely by waiting and should not retry indefinitely. An unknown 429 is currently treated as retryable.

Standard mode uses a bounded delay sequence. persistent mode can continue waiting, but it still:

• caps each wait;
• stops after too many consecutive identical transient failures;
• publishes wait state through on_retry_wait.

Verify constants and decisions in:

• nanobot/providers/base.py::LLMProvider._run_with_retry
• LLMProvider._is_transient_response
• LLMProvider._is_retryable_429_response
• LLMProvider._extract_retry_after_from_response

3. Why streaming responses are harder to retry

If a Provider fails before emitting any delta, the request can retry like a non-streaming call. Replaying after content is already visible can cause:

• duplicate text;
• a second completion inconsistent with the first partial output;
• repeated Tool Calls or billing.

chat_stream_with_retry() uses has_streamed_content as a guard. Depending on segment support, the code can open a new segment or suppress duplicate deltas; when it cannot do so safely, it skips retry.

Once a side effect is externally visible, revisit idempotency

The same principle applies to sending messages, writing files, charging payments, and creating tickets. Retrying an Agent’s “thought” is not the same as retrying a real-world operation.

4. How Tool errors enter model context

ToolRegistry.execute() and AgentRunner._run_tool() convert many failures into Tool Results rather than crashing the process. Runner also adds guidance such as “analyze the error and try another approach.”

Security-boundary errors instead include explicit non-retryable guidance. A workspace or SSRF guard tells the model not to bypass policy with Shell tricks, encoded IPs, or another Tool.

flowchart LR
    A["Execute Tool"] --> B{"Result type"}
    B -->|"Success"| C["Append tool message"]
    B -->|"Recoverable error"| D["Append error and change-approach hint"]
    B -->|"Policy denial"| E["Append non-bypassable security guidance"]
    B -->|"Cancellation / fatal"| F["Raise upward and stop"]
    C --> G["Next model request"]
    D --> G
    E --> G

Turning an ordinary error into an observation lets the Agent self-correct. Making a policy denial a hard boundary discourages repeated probing of the same restriction.

5. Outer exception fallback

When AgentLoop._dispatch() catches an ordinary exception, it:

• records the exception log;
• publishes a general error response;
• publishes Turn completion;
• tells the Cron coordinator that the run failed.

This is a final user-experience fallback, not a root-cause fix. When you see “Sorry, I encountered an error,” inspect the preceding exception and state trace instead of debugging only the Channel.

6. Evaluate write consistency separately

Session

SessionManager.save():

1. writes the complete .jsonl.tmp;
2. optionally calls flush() and os.fsync();
3. atomically replaces the target with os.replace();
4. attempts parent-directory sync when fsync=True;
5. removes the temporary file on failure.

Ordinary saves do not fsync by default. During graceful Gateway shutdown, flush_all() rewrites cached Sessions with fsync=True.

Cron

CronService._atomic_write() always uses a temporary file, file fsync, os.replace, and parent-directory sync. Its comment says this prevents a container shutdown from leaving a truncated jobs.json.

Memory history

MemoryStore uses different paths for append and rewrite:

• a file lock coordinates append;
• rewrite uses a temporary file, fsync, and replace;
• cursor and history formats have dedicated recovery tests.

Implementations:

• nanobot/agent/memory.py::append_history
• nanobot/agent/memory.py::_write_entries

Configuration

nanobot/config/loader.py::save_config() currently opens the target in write mode and emits JSON directly. It does not use a temporary file and os.replace. The atomic-write guarantee of Session cannot be generalized to config.json.

Confirmed current behavior

This is not a prediction that configuration will be corrupted. It states only that the current save_config() source does not prove the same crash durability as Session.

7. Consistency does not mean “every file updates transactionally”

One Turn may touch:

• Session JSONL;
• Memory history;
• Goal metadata;
• Cron store;
• WebUI transcript/usage index;
• real external Tool side effects.

The current code has no cross-file, cross-service transaction around all of them. Its design instead aims to:

• make each core store recoverable where practical;
• record runtime checkpoints in Session metadata;
• continue from available facts after restart;
• avoid assuming that external side effects can roll back.

8. Practical diagnostic order

When a request fails:

1. identify whether failure occurred in the entry point, Provider, Runner, Tool, persistence, or response delivery;
2. inspect finish_reason, status code, and retry metadata;
3. determine whether any stream delta or Tool side effect is already visible;
4. inspect the Session checkpoint and final JSONL;
5. check whether fallback or finalization retry ran;
6. only then consider replaying the request manually.

Before replay, decide whether the operation is idempotent.

9. Source and tests

Source:

• nanobot/providers/base.py
• nanobot/providers/fallback_provider.py
• nanobot/agent/runner.py::_request_model
• nanobot/agent/runner.py::_run_tool
• nanobot/agent/loop.py::_dispatch
• nanobot/session/manager.py::save
• nanobot/cron/service.py::_atomic_write
• nanobot/agent/memory.py
• nanobot/config/loader.py::save_config

Tests:

• tests/providers/test_provider_retry.py
• tests/providers/test_provider_retry_after_hints.py
• tests/providers/test_provider_error_metadata.py
• tests/agent/test_runner_errors.py
• tests/agent/test_runner_fallback.py
• tests/agent/test_runner_core.py
• tests/agent/test_runner_persistence.py
• tests/session/test_session_fsync.py
• tests/agent/test_memory_store.py

Self-check

1. Why does retry become riskier after one streaming delta is visible?
2. How does a Tool error string differ from a Tool raising CancelledError?
3. What problems do os.replace and fsync solve separately?
4. Why does atomic Session persistence not prove atomic configuration persistence?

Next: Logging, runtime events, and testing