Route LLM Requests Across Providers with Fallback in Python

Once you’ve wired one large language model into a Python app, you’ve probably wanted a second one. Maybe it’s a cheaper model for easy questions, maybe it’s a different vendor as backup when the first one starts timing out, or maybe you’ve been burned once already by a demo that died live because a single provider was having a bad afternoon. The obvious fix — an if/elif chain of vendor SDK calls scattered through your code — holds together for exactly as long as you have one call site. Add a second, and the retry logic, the timeout handling, and the “try the other one” fallback all have to be copied and kept in sync by hand.

The fix is a small piece of infrastructure called a gateway: one class your application calls, with everything provider-specific hidden behind it. You define a Provider protocol with a single complete(request) -> response method, wrap each vendor’s client in an adapter that implements it, and hand the gateway an ordered list of those adapters. On every call the gateway tries the first provider, retries it a fixed number of times, and — if it still fails — moves to the next provider in the list, all before your application code sees anything but a normal response or one gateway-level error. Add a circuit breaker that stops sending requests to a provider that’s clearly down, and the rest of your codebase never has to know which vendor actually answered.

(If you haven’t already put a narrow interface between your code and a model provider, testing an LLM pipeline without API calls covers the same Protocol-based boundary for a single provider — this post extends the idea to route across several, with real retries and a real fallback path.)

One Class, Three Jobs

Strip away the code and a gateway only does three things, and the difference between a fragile one and a solid one is usually which of these three got skipped.

1. Enforce a shared shape. Every provider has to accept the same request and return the same response, so your application code never branches on which vendor answered.
2. Enforce an order. Providers are tried in priority order, with a small, fixed number of retries per provider before the gateway gives up on it and moves to the next.
3. Remember trouble. A provider that’s clearly down shouldn’t keep eating retries on every request — the gateway should stop sending it traffic for a while and come back later.

Everything below builds those three jobs in order, on a small, fully reproducible example.

Two Providers That Don’t Need an API Key

A tutorial that only works if you’re paying for two or three different vendor accounts is a bad tutorial — you’d be debugging your credit card, not the gateway. So instead of calling real APIs, this post simulates two providers with Python’s random.Random, seeded so every run — yours included — produces the exact same sequence of successes, failures, and latencies:

• aurora: cheap and fast, but flaky — a 35% simulated failure rate.
• borealis: reliable but pricier and slower — a 5% failure rate.

Swapping either one for a real vendor SDK later is a one-class change; the gateway itself never needs to know the difference, which is the entire point of hiding providers behind one interface.

LLMRequest and LLMResponse: One Shape Every Provider Returns

Before writing a single provider, fix the shape that all of them speak:

from dataclasses import dataclass


@dataclass(frozen=True)
class LLMRequest:
    prompt: str
    max_tokens: int = 200


@dataclass(frozen=True)
class LLMResponse:
    text: str
    provider: str
    latency_ms: float
    cost_usd: float


class ProviderError(Exception):
    pass


req = LLMRequest(prompt="Summarize the Q3 shipping report in two sentences.")
print(req)

LLMRequest(prompt='Summarize the Q3 shipping report in two sentences.', max_tokens=200)

LLMRequest and LLMResponse are frozen dataclasses — plain, immutable data with no behavior. The provider field on the response matters more than it looks: it’s what lets you find out, after the fact, which vendor actually answered a given call.

A Provider Is Anything With One complete() Method

The gateway shouldn’t care whether it’s talking to a real SDK or a stand-in, so define the minimal shape with typing.Protocol:

from typing import Protocol


class Provider(Protocol):
    name: str

    def complete(self, request: LLMRequest) -> LLMResponse: ...

A class doesn’t need to inherit from Provider to satisfy it — it only needs a matching complete() method and a name attribute. The Python typing documentation calls this structural typing, and it’s what makes an adapter swap-in-place instead of a rewrite. Here’s the simulated version:

import random


class SimulatedProvider:
    def __init__(self, name, failure_rate, latency_range_ms, cost_per_1k_tokens, seed):
        self.name = name
        self.failure_rate = failure_rate
        self.latency_range_ms = latency_range_ms
        self.cost_per_1k_tokens = cost_per_1k_tokens
        self._rng = random.Random(seed)

    def complete(self, request: LLMRequest) -> LLMResponse:
        if self._rng.random() < self.failure_rate:
            raise ProviderError(f"{self.name} timed out")
        latency = round(self._rng.uniform(*self.latency_range_ms), 1)
        cost = round((request.max_tokens / 1000) * self.cost_per_1k_tokens, 5)
        text = f"[{self.name}] {request.prompt[:30]}..."
        return LLMResponse(text=text, provider=self.name, latency_ms=latency, cost_usd=cost)

Every call draws from the provider’s own seeded generator, so a fresh SimulatedProvider("aurora", failure_rate=0.35, latency_range_ms=(150, 300), cost_per_1k_tokens=0.002, seed=42) behaves identically every time you run it. Three calls in a row show exactly what “35% failure rate” looks like in practice:

preview = SimulatedProvider(
    "aurora", failure_rate=0.35, latency_range_ms=(150, 300),
    cost_per_1k_tokens=0.002, seed=42,
)
for _ in range(3):
    try:
        print(preview.complete(req))
    except ProviderError as exc:
        print("error:", exc)

LLMResponse(text='[aurora] Summarize the Q3 shipping repo...', provider='aurora', latency_ms=153.8, cost_usd=0.0004)
error: aurora timed out
error: aurora timed out

One clean response, then two failures back to back. That’s not a rigged example — it’s what a 35% failure rate does over a short run, and it’s exactly the pattern a single retry won’t reliably survive. This is the gap a fallback provider exists to close.

Remembering Trouble: A Circuit Breaker

Retrying a provider that’s genuinely down doesn’t just waste time on one request — it wastes time on every request until the provider recovers. A circuit breaker tracks consecutive failures and, past a threshold, stops sending traffic to that provider for a cooldown period instead of trying it again on every call:

class CircuitBreaker:
    def __init__(self, failure_threshold=3, cooldown_requests=5):
        self.failure_threshold = failure_threshold
        self.cooldown_requests = cooldown_requests
        self.consecutive_failures = 0
        self.open_until_request = 0

    def is_open(self, request_index):
        return request_index < self.open_until_request

    def record_success(self):
        self.consecutive_failures = 0

    def record_failure(self, request_index):
        self.consecutive_failures += 1
        if self.consecutive_failures >= self.failure_threshold:
            self.open_until_request = request_index + 1 + self.cooldown_requests

This one is deliberately simple: it counts requests handled by the gateway, not wall-clock time, so the cooldown is exactly reproducible in a blog post. (More on why that’s a simplification, not a production recommendation, in the gotchas below.) A breaker starts closed — traffic flows normally. Three failures in a row opens it, and it stays open, skipping the provider outright, until cooldown_requests more gateway calls have gone by.

The Gateway: Retry, Then Fall Back, Then Remember

With a shared shape, a Provider interface, and a breaker, the gateway itself is mostly bookkeeping around three nested loops — providers, then retries within each provider:

class GatewayError(Exception):
    pass


class Gateway:
    def __init__(self, providers, retries_per_provider=1, breaker_threshold=3, breaker_cooldown=5):
        self.providers = providers
        self.retries_per_provider = retries_per_provider
        self.breakers = {
            p.name: CircuitBreaker(breaker_threshold, breaker_cooldown) for p in providers
        }
        self._request_index = 0
        self.log = []

    def complete(self, request: LLMRequest) -> LLMResponse:
        idx = self._request_index
        self._request_index += 1
        last_error = None

        for provider in self.providers:
            breaker = self.breakers[provider.name]
            if breaker.is_open(idx):
                self.log.append({"request": idx, "provider": provider.name, "outcome": "skipped_open_circuit"})
                continue

            for attempt in range(1, self.retries_per_provider + 1):
                try:
                    response = provider.complete(request)
                except ProviderError as exc:
                    last_error = exc
                    self.log.append({"request": idx, "provider": provider.name, "outcome": f"failed_attempt_{attempt}"})
                    continue
                else:
                    breaker.record_success()
                    self.log.append({"request": idx, "provider": provider.name, "outcome": "success"})
                    return response

            breaker.record_failure(idx)

        raise GatewayError(f"all providers exhausted for request {idx}") from last_error

Note what Gateway.complete() never does: it never raises a ProviderError. Callers either get a normal LLMResponse or a single GatewayError if every provider in the list is exhausted — the retries and the vendor-specific failure are entirely internal. Every attempt, success, skip, and failure also gets appended to self.log, which is what turns this from a black box into something you can actually audit.

Wire up two fresh providers — aurora first, borealis as the fallback — and send 40 requests through:

aurora = SimulatedProvider(
    "aurora", failure_rate=0.35, latency_range_ms=(150, 300),
    cost_per_1k_tokens=0.002, seed=42,
)
borealis = SimulatedProvider(
    "borealis", failure_rate=0.05, latency_range_ms=(400, 700),
    cost_per_1k_tokens=0.012, seed=7,
)
gateway = Gateway([aurora, borealis], retries_per_provider=2, breaker_threshold=3, breaker_cooldown=5)

results, errors = [], 0
for i in range(40):
    try:
        results.append(gateway.complete(LLMRequest(prompt=f"request-{i}", max_tokens=200)))
    except GatewayError:
        errors += 1

print(f"completed={len(results)} errors={errors}")
for r in results[:5]:
    print(r)

completed=40 errors=0
LLMResponse(text='[aurora] request-0...', provider='aurora', latency_ms=153.8, cost_usd=0.0004)
LLMResponse(text='[borealis] request-1...', provider='borealis', latency_ms=445.3, cost_usd=0.0024)
LLMResponse(text='[aurora] request-2...', provider='aurora', latency_ms=251.5, cost_usd=0.0004)
LLMResponse(text='[aurora] request-3...', provider='aurora', latency_ms=163.0, cost_usd=0.0004)
LLMResponse(text='[aurora] request-4...', provider='aurora', latency_ms=154.5, cost_usd=0.0004)

Every one of the 40 requests eventually got a real response — that’s what a fallback buys you. Request 1 already shows why: aurora’s second and third draws (from the earlier three-call preview) were both failures, so here the gateway retries aurora twice, gives up on it for this request, and borealis answers instead. Your application code never sees any of that; it just gets results[1].

What 40 Requests Actually Looked Like

gateway.log records every attempt, so you can see the breaker do its job instead of taking it on faith. Requests 19 through 27 are the interesting stretch:

for row in gateway.log:
    if 18 <= row["request"] <= 27:
        print(row)

{'request': 19, 'provider': 'aurora', 'outcome': 'failed_attempt_1'}
{'request': 19, 'provider': 'aurora', 'outcome': 'failed_attempt_2'}
{'request': 19, 'provider': 'borealis', 'outcome': 'success'}
{'request': 20, 'provider': 'aurora', 'outcome': 'failed_attempt_1'}
{'request': 20, 'provider': 'aurora', 'outcome': 'failed_attempt_2'}
{'request': 20, 'provider': 'borealis', 'outcome': 'failed_attempt_1'}
{'request': 20, 'provider': 'borealis', 'outcome': 'success'}
{'request': 21, 'provider': 'aurora', 'outcome': 'failed_attempt_1'}
{'request': 21, 'provider': 'aurora', 'outcome': 'failed_attempt_2'}
{'request': 21, 'provider': 'borealis', 'outcome': 'success'}
{'request': 22, 'provider': 'aurora', 'outcome': 'skipped_open_circuit'}
{'request': 22, 'provider': 'borealis', 'outcome': 'success'}
{'request': 23, 'provider': 'aurora', 'outcome': 'skipped_open_circuit'}
{'request': 23, 'provider': 'borealis', 'outcome': 'success'}
{'request': 24, 'provider': 'aurora', 'outcome': 'skipped_open_circuit'}
{'request': 24, 'provider': 'borealis', 'outcome': 'success'}
{'request': 25, 'provider': 'aurora', 'outcome': 'skipped_open_circuit'}
{'request': 25, 'provider': 'borealis', 'outcome': 'success'}
{'request': 26, 'provider': 'aurora', 'outcome': 'skipped_open_circuit'}
{'request': 26, 'provider': 'borealis', 'outcome': 'failed_attempt_1'}
{'request': 26, 'provider': 'borealis', 'outcome': 'success'}
{'request': 27, 'provider': 'aurora', 'outcome': 'success'}

aurora fails both of its attempts on requests 19, 20, and 21 — three full failures in a row, which is this breaker’s threshold. From request 22 on, the gateway doesn’t even try aurora; it goes straight to borealis (which itself needs a retry on request 26, and gets one). Request 27 is the first one to give aurora another chance, five requests after the breaker tripped — exactly the cooldown_requests=5 we configured:

for name, breaker in gateway.breakers.items():
    print(name, "opened_until_request=", breaker.open_until_request)

aurora opened_until_request= 27
borealis opened_until_request= 0

Adding Up Latency and Cost

None of this matters if you can’t answer “how much did this actually cost, and was it fast enough?” With every response carrying its own latency_ms and cost_usd, a pandas groupby answers both in two lines:

import pandas as pd

success_df = pd.DataFrame(
    [{"provider": r.provider, "latency_ms": r.latency_ms, "cost_usd": r.cost_usd} for r in results]
)
summary = success_df.groupby("provider").agg(
    requests_served=("provider", "count"),
    avg_latency_ms=("latency_ms", "mean"),
    total_cost_usd=("cost_usd", "sum"),
).round(3)
print(summary)

          requests_served  avg_latency_ms  total_cost_usd
provider                                                 
aurora                 27         221.011           0.011
borealis               13         518.123           0.031

(This run used Python 3.13.2 and pandas 3.0.3; none of the code here depends on anything version-specific.) aurora answered two-thirds of the traffic at less than half the average latency and roughly a third of the cost — which is exactly the trade you’d want from a “cheap primary, reliable fallback” pair. If those numbers ever flipped — if the fallback started serving most of the traffic — that would be a signal to look at why the primary is failing so often, not a reason to just accept the higher bill.

Three Ways This Still Bites You

Retries aren’t free. On requests 19 through 21 above, aurora gets called twice before the gateway moves on — in this simulation that’s instant, but a real network timeout of even two seconds per attempt means four seconds of dead air before borealis is ever tried. Keep retries_per_provider low and keep per-attempt timeouts short; more retries feels safer and is usually just slower.

A fallback answer isn’t an equivalent answer. aurora and borealis are stand-ins here, but in a real gateway they’re different vendors running different models. Silently accepting whichever one answered first is a bet that the cheaper or faster model’s output is an acceptable substitute for the other — sometimes true, sometimes not. LLMResponse.provider exists so you can log and monitor which vendor actually served each request, not just assume they’re interchangeable.

This breaker’s cooldown is measured in requests, not time. That’s what makes its behavior exactly reproducible in a tutorial, but it’s a poor fit for uneven real traffic: five requests might arrive in fifty milliseconds (barely any cooldown at all) or over five minutes (a needlessly long one). A production breaker should key its cooldown off elapsed time or a rolling window, not a call counter.

Wiring In a Real Provider

The whole reason to hide providers behind Provider is that connecting a real one is a one-class job, not a gateway rewrite. Keep vendor-specific code — the actual SDK call, its request shape, its response shape — inside a small adapter and nothing else:

import time


class RealProviderAdapter:
    def __init__(self, name, client, model_name, cost_per_1k_tokens):
        self.name = name
        self.client = client
        self.model_name = model_name
        self.cost_per_1k_tokens = cost_per_1k_tokens

    def complete(self, request: LLMRequest) -> LLMResponse:
        started = time.perf_counter()
        reply = self.client.generate(
            model=self.model_name,
            prompt=request.prompt,
            max_tokens=request.max_tokens,
        )
        elapsed_ms = (time.perf_counter() - started) * 1000
        return LLMResponse(
            text=reply.text,
            provider=self.name,
            latency_ms=round(elapsed_ms, 1),
            cost_usd=round((request.max_tokens / 1000) * self.cost_per_1k_tokens, 5),
        )

The method and field names on client above are placeholders — follow whichever SDK you’re actually wiring in, since every vendor’s client shape is different and they change over time. What doesn’t change is where that code lives: inside the adapter, translating one vendor’s format into your shared LLMRequest/LLMResponse shape, with no retry logic, no fallback order, and no breaker state anywhere near it. The Gateway you built above doesn’t need to change at all to accept RealProviderAdapter instances alongside — or instead of — the simulated ones.

If you want to take this further — real vendor SDKs behind the same adapters, wall-clock-based circuit breakers, and production error handling instead of a printed log — the retries and error-handling lessons in our Shipping AI Applications module in the free Generative AI & LLM Engineering course pick up exactly where this post leaves off. And once you’re running more than one provider for real, monitoring LLM requests with JSONL and pandas is the natural next step for turning a log like gateway.log into an ongoing release check instead of a one-off printout.