An LLM can select the correct tool, yet the function it requests can time out instead of returning a result. Calling the function again immediately may add pressure to a struggling service, while repeating an unsafe write may create the same booking or payment twice.

Handle failed LLM tool calls by classifying the execution error before retrying. Retry only temporary failures for operations known to be safe to repeat, wait with capped exponential backoff plus jitter or a server-provided Retry-After value, and stop at both an attempt limit and a total-wait limit. Return the final failure status to the agent instead of inventing a result.

This tutorial builds that boundary around two fictional bicycle-share tools. get_dock_status only reads data. hold_bicycle changes state, so it needs stronger protection. A deterministic fake service lets us execute every success and stop path without an API key, a network request, or real waiting.

Prerequisites and Setup

You need Python 3.10 or later and basic knowledge of functions, loops, and dictionaries. We use pandas to inspect the executed records and Matplotlib to create the chart.

Create a virtual environment, activate it, and install both packages. On macOS or Linux, run:

python -m venv .venv
source .venv/bin/activate
python -m pip install pandas matplotlib

On Windows PowerShell, replace the activation line with .venv\Scripts\Activate.ps1.

The tutorial build ran with Python 3.13.2, pandas 3.0.3, and Matplotlib 3.11.0. Download these three files if you want to inspect or reproduce the same cases:

• synthetic retry cases
• executed attempt log
• per-case results

The data is original, fictional, and released under CC0 1.0. It does not contain model output, service measurements, user data, or real bicycle-share records. The delays are scheduled values recorded by the test; they are not timing benchmarks.

If tool calling itself is new to you, read Function Calling in Python first. That lesson covers the model request, Python execution, and tool-result round trip. Here, we start after Python validates the requested tool name and arguments and calls the matching function, but the operation fails.

How a Retry Decision Works

A transient failure is a problem that may disappear without changing the request. A network timeout or temporary service overload can be transient. A permanent failure needs a different request or an external fix. Missing credentials and an invalid dock ID do not improve because a loop waits for half a second.

The retry boundary answers four questions in order:

1. Is this error in the application’s explicit retryable set?
2. Is repeating this operation safe?
3. Is another attempt still available?
4. Does the next delay fit inside the total wait budget?

Only four “yes” answers schedule another attempt. Otherwise, the function returns a structured stop status that the agent can explain or escalate.

HTTP status codes help with classification, but they do not replace application knowledge. The current policy retries connection timeouts and HTTP 408, 429, 500, 502, 503, and 504 responses. The HTTP semantics standard says a client may repeat a pending request after 408. It describes 503 as a condition in which the service is temporarily unable to handle the request and allows the server to suggest a delay. The 429 specification says the service may include Retry-After when it rate-limits a client.

That list is a teaching policy, not a rule for every API. In particular, 500 is a general server error. Add it only when the service contract and your observations show that a later attempt is useful. Errors such as 400, 401, and 404 stop immediately in this example.

Repeating the operation must also be safe. An idempotent operation has the same intended effect whether it runs once or several times. Reading one dock is naturally safe to repeat. A write such as holding a bicycle is not automatically safe because the first request may have succeeded before its response was lost. The HTTP standard also warns clients not to retry non-idempotent requests automatically unless they know repetition is safe. A client and service can provide that safety with a unique idempotency key and server-side deduplication, but the client must not assume the feature exists.

Represent Failures Without Losing Their Meaning

The execution layer needs a small error type. It carries a machine-readable code and an optional server delay. Keeping those values structured avoids parsing human error messages later.

from dataclasses import dataclass

@dataclass
class ToolFailure(Exception):
    code: str
    retry_after_s: float | None = None

RETRYABLE_CODES = {
    "timeout",
    "http_408",
    "http_429",
    "http_500",
    "http_502",
    "http_503",
    "http_504",
}

A real HTTP adapter would translate its client’s timeout exception and status response into ToolFailure. Keep that provider-specific translation outside the retry function. This lets the policy work with several tools without importing one networking library into every layer.

For this reproducible lesson, a fake tool returns a scripted outcome on each call. It returns a normal dictionary for ok and raises ToolFailure for everything else:

class ScriptedTool:
    def __init__(self, outcomes):
        self.outcomes = outcomes
        self.index = 0

    def __call__(self, dock_id):
        outcome = self.outcomes[min(self.index, len(self.outcomes) - 1)]
        self.index += 1

        if outcome == "ok":
            return {"dock_id": dock_id, "bicycles": 7, "spaces": 5}

        code, separator, retry_after = outcome.partition("@")
        raise ToolFailure(
            code=code,
            retry_after_s=float(retry_after) if separator else None,
        )

For example, ScriptedTool(["[email protected]", "ok"]) raises a rate-limit error first and succeeds on the second call. This is a controlled test input, not a claim about how often a real service recovers.

Calculate Backoff, Jitter, and Server Delays

Exponential backoff increases the pause after each failure. With a base delay of 0.5 seconds, the uncapped sequence is 0.5, 1.0, 2.0, and 4.0 seconds. The larger gaps reduce pressure on a service that is already struggling.

Jitter is a small random addition to that delay. Without it, many workers that fail together can retry at the same times. Our function adds up to 25% jitter and caps the exponential part at four seconds:

import random

def choose_delay(
    failure_number,
    retry_after_s,
    rng,
    base_delay_s=0.5,
    delay_cap_s=4.0,
):
    backoff = min(
        delay_cap_s,
        base_delay_s * 2 ** (failure_number - 1),
    )
    jitter = rng.uniform(0, backoff * 0.25)
    calculated = backoff + jitter
    return round(max(calculated, retry_after_s or 0.0), 2)

The rng argument receives a random.Random instance. The build gives each case a fixed seed, so the generated dataset and chart stay reproducible. In a live process, create a normal unseeded generator instead; many clients should not share the same retry pattern.

The final line uses the larger of the calculated delay and retry_after_s. The current Retry-After definition permits either a number of seconds or an HTTP date. This lesson’s HTTP adapter has already converted the value into seconds. A production adapter should parse both allowed forms, reject invalid values, and still apply a maximum wait budget.

When the retry really should pause, pass time.sleep as the sleeper. Python’s time.sleep documentation describes the standard blocking call. An asynchronous application should use its framework’s non-blocking sleep instead so one retry does not pause unrelated work.

Put a Bounded Policy Around the Tool

The retry function below accepts the operation rather than knowing anything about bicycle docks. It also accepts the random generator and sleeper, which makes the behavior easy to test.

Read the checks inside the except block from top to bottom. A permanent error stops first. An unsafe operation stops before waiting. The attempt limit and total-wait limit then prevent a temporary outage from becoming an endless loop.

def call_with_retry(
    operation,
    *,
    dock_id,
    retry_safe,
    rng,
    sleeper,
    max_attempts=4,
    max_total_wait_s=5.0,
):
    attempts = []
    total_wait_s = 0.0

    for attempt in range(1, max_attempts + 1):
        try:
            value = operation(dock_id=dock_id)
        except ToolFailure as error:
            event = {
                "attempt": attempt,
                "outcome": error.code,
                "delay_s": 0.0,
                "elapsed_s": total_wait_s,
            }
            attempts.append(event)

            if error.code not in RETRYABLE_CODES:
                return {"ok": False, "status": "permanent_error"}, attempts
            if not retry_safe:
                return {"ok": False, "status": "unsafe_retry"}, attempts
            if attempt == max_attempts:
                return {"ok": False, "status": "attempt_limit"}, attempts

            delay_s = choose_delay(attempt, error.retry_after_s, rng)
            if total_wait_s + delay_s > max_total_wait_s:
                return {"ok": False, "status": "wait_budget"}, attempts

            event["delay_s"] = delay_s
            sleeper(delay_s)
            total_wait_s = round(total_wait_s + delay_s, 2)
        else:
            attempts.append({
                "attempt": attempt,
                "outcome": "ok",
                "delay_s": 0.0,
                "elapsed_s": total_wait_s,
            })
            return {
                "ok": True,
                "status": "succeeded",
                "value": value,
            }, attempts

    raise AssertionError("the retry loop must return inside max_attempts")

A retry budget bounds the repeated work or delay allowed for one tool request. max_attempts limits executions, while max_total_wait_s limits scheduled waiting. Both matter: a server can request a delay longer than the user-facing request should remain open, and several short delays can still add up. The wait budget is not an end-to-end timeout because it does not include time spent inside the operation; add a separate request deadline when your application needs one.

The build passes observed_sleeps.append as sleeper. Calling it records the delay without pausing the test. Assuming real_get_dock_status is your provider adapter, a live synchronous program would pass time.sleep:

import time

result, attempts = call_with_retry(
    real_get_dock_status,
    dock_id="DOCK-04",
    retry_safe=True,
    rng=random.Random(),
    sleeper=time.sleep,
)

Do not send the agent a made-up dock status when result["ok"] is false. Return the stable status, say that current data is unavailable, and let the surrounding application decide whether to ask the user to try later or use a documented fallback.

Read the Executed Retry Cases

The build ran 13 designed cases and asserted that every final status matched its predefined expected status. It produced 23 attempt events. Seven cases succeeded and six stopped at a guard:

Python 3.13.2; pandas 3.0.3; Matplotlib 3.11.0
cases=13 attempt_events=23 succeeded=7 stopped=6
case_id  attempts  total_wait_s    final_status
    R02         2          0.56       succeeded
    R03         2          2.00       succeeded
    R07         1          0.00 permanent_error
    R10         4          4.13   attempt_limit
    R11         1          0.00     wait_budget
    R12         1          0.00    unsafe_retry
    R13         2          0.53       succeeded
all expected statuses matched

R02 waits 0.56 scheduled seconds after a connection timeout, then succeeds. R03 honors the service’s two-second rate-limit delay instead of using its shorter calculated delay. R07 stops after one invalid request because repetition cannot repair it.

R10 demonstrates the attempt limit. Four 503 responses use all four allowed executions, so there is no fifth attempt. R11 receives a 12-second server delay, but the teaching policy allows only five seconds of total waiting. It returns wait_budget before sleeping or calling the fake function a second time.

The last pair shows why retry safety is separate from error classification. Both cases call a function that attempts a write, and both attempts time out. R12 has no idempotency protection, so Python cannot know whether the first hold was created and stops with unsafe_retry. R13 represents the same write protected by an application-verified idempotency key and server deduplication; the policy can call the function again, and the case succeeds on attempt two. The fake service does not implement a real reservation system, so this is a policy test rather than proof of a provider feature.

The generated chart shows every executed attempt. A short bar can mean immediate success, but it can also mean a useful early stop. Read the label at the end of each bar together with its length.

This distinction is important for monitoring. A lower attempt count is not automatically better, and a successful retry is not automatically harmless. Track the error class, delay, final status, and operation safety without storing secrets or unnecessary tool arguments.

Common Mistakes and Troubleshooting

Every exception is retried. Catch only expected transport or service failures, then map them to explicit codes. Programming errors such as TypeError and KeyError should surface during testing instead of entering a retry loop.

Every 4xx response is treated as permanent. Most client errors need a changed request, but 408 and 429 are important exceptions in this policy. Some APIs also define special behavior for 409 or other codes. Follow the contract of the service you call.

Every 5xx response is retried. A broad class rule can repeat a request that the provider cannot handle safely. Use an allowlist, decide whether 500 fits your service, and keep a hard limit even for known temporary codes.

Retry-After becomes an unlimited sleep. Parse it, compare it with the remaining budget, and stop when it is too large. For background jobs, you may store a next-attempt time in a queue instead of keeping one worker asleep.

All workers retry together. Add jitter and avoid one shared fixed seed in production. A fixed seed belongs in a reproducible test, not across a deployed fleet.

A write is retried because it timed out. A timeout means the client did not receive a complete answer. It does not prove that the server made no change. Require a documented idempotency mechanism or check the resulting state before repeating the action.

The model decides whether repetition is safe. Retry safety belongs to application-owned tool metadata and code. A model may propose another call, but the executor should enforce the same limits and idempotency rules again.

Retries happen inside another unbounded agent loop. A four-attempt tool policy can still run many times if the agent keeps proposing the same call. Combine this boundary with a run-level step or cost limit. The safe Python agent loop tutorial shows that outer control, while detecting loops from JSONL traces helps diagnose repeated work after a run.

What You Learned

A retry loop should make fewer decisions than an LLM, but each decision should be explicit and testable:

• classify failures instead of treating every error alike;
• repeat only operations the application knows are safe;
• increase delays with capped exponential backoff and jitter;
• honor a valid server delay only when it fits the retry budget;
• stop at both an attempt limit and a total-wait limit; and
• return a structured failure instead of hiding missing data.

The durable sequence is classify, check safety, wait, retry, stop. Keep the retry boundary in ordinary Python around the tool executor. Then a different model, API client, or agent framework can propose calls without changing the rules that protect the service and its users.