How to Limit Concurrent Hugging Face Calls with FastAPI

An API can accept requests faster than a remote model can answer them. If every FastAPI request immediately starts another model call, a short traffic burst can fill connection pools, increase memory use, and make all callers wait. Asynchronous code can keep the server responsive during network waits, but async alone does not limit how much work reaches the provider.

To limit concurrent Hugging Face calls with FastAPI, reuse one httpx.AsyncClient, place an asyncio.Semaphore around the outgoing request, and set a short deadline for acquiring a slot. Return HTTP 503 when no slot becomes available, use explicit network timeouts, and release the slot in finally so failures cannot reduce future capacity.

We will apply that pattern to a small sentiment endpoint for fictional repair-service feedback. An endpoint is an HTTP method and URL that perform one API operation. Eight requests will arrive at once, but only three will cross the mocked network boundary. Separate tests will verify a timeout, a provider rate limit, a provider server error, and an invalid provider response without using a real token.

What You Need and What We Will Test

You need Python 3.10 or later. You should know how to run a Python file and how a dictionary stores key-value pairs. No FastAPI, HTTPX, or asynchronous-programming experience is required.

Create a virtual environment so these packages do not conflict with another project. On macOS or Linux, run:

python -m venv .venv
source .venv/bin/activate
python -m pip install fastapi uvicorn httpx matplotlib

On Windows PowerShell, activate with .venv\Scripts\Activate.ps1. Matplotlib is only needed to reproduce the results chart. The verified build used Python 3.13.2, FastAPI 0.136.1, HTTPX 0.28.1, Pydantic 2.13.4, and Matplotlib 3.11.0.

The lesson uses Hugging Face’s serverless HF Inference service. Hugging Face distinguishes this from its dedicated, managed Inference Endpoints product. The current HF Inference documentation lists the router URL used below. Its text-classification API specification defines inputs, optional top_k, and the returned label-score objects.

For a live call, create a fine-grained Hugging Face token with permission to call Inference Providers and expose it only through the environment:

export HF_TOKEN="your_token_here"

In Windows PowerShell, set the same variable with $env:HF_TOKEN = "your_token_here". Do not put the token in a Python file or CSV. The executed lesson does not need one because its tests replace the network transport.

Download the synthetic feedback requests if you want the same inputs. The 13 requests, delays, and failure behaviors are original fictional teaching data released under CC0 1.0. They are not real user comments, traffic measurements, or Hugging Face performance data.

Understanding Backpressure: Let Only Safe Work Enter

Concurrency is the number of tasks that are in progress during the same period. An asynchronous endpoint can pause one network wait and work on another request. That is useful for input/output (I/O) work, but it can also create hundreds of outgoing calls if hundreds of requests arrive together.

Backpressure is a rule that slows or rejects new work when a downstream system is already full. Our rule has three parts:

incoming request
      |
      v
wait up to 35 ms for one of 3 slots
      | available                 | still full
      v                           v
call Hugging Face             return HTTP 503
      |
      v
release the slot on success or failure

An asyncio.Semaphore(3) represents the three slots. Acquiring a slot decreases the available count. Releasing it increases the count. The semaphore controls calls that are already inside the remote-work section; it does not create threads or make the model run faster.

The 35-millisecond wait is an admission deadline. It limits how long excess tasks remain in the waiting line; it does not cap how many requests may arrive during those 35 milliseconds. The value is intentionally short so the behavior is easy to execute. A production value must come from your latency target, normal request duration, traffic pattern, and provider quota.

The limit and deadline solve different problems from an HTTP timeout. The admission deadline limits time spent waiting before a call. HTTPX timeouts limit connection, write, read, and connection-pool waits during a network operation. HTTPX documents those four controls in its timeout guide.

Reuse One Asynchronous HTTP Client

Start by defining the provider URL and the public request and response shapes. A Pydantic model is a Python class that checks incoming fields and creates API documentation for them.

from pydantic import BaseModel, ConfigDict, Field

MODEL_ID = "distilbert/distilbert-base-uncased-finetuned-sst-2-english"
INFERENCE_URL = (
    "https://router.huggingface.co/hf-inference/models/" + MODEL_ID
)

class FeedbackRequest(BaseModel):
    model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
    request_id: str = Field(pattern=r"^[A-Z][0-9]{2}$")
    text: str = Field(min_length=3, max_length=500)

class SentimentResponse(BaseModel):
    request_id: str
    label: str
    score: float
    model: str

This schema accepts IDs such as B01 and text between 3 and 500 characters. Validation is not the focus here; How to Validate ML API Inputs with FastAPI and Pydantic develops that boundary in detail.

Next, create one client for the application lifetime. A connection pool keeps network connections ready for reuse. Creating a new client inside every request discards that benefit and adds repeated connection setup.

FastAPI’s current lifespan guide recommends a lifespan context for resources created before requests and closed during shutdown. HTTPX likewise recommends reusing an AsyncClient instead of constructing clients repeatedly inside a busy loop.

import os
import httpx
from contextlib import asynccontextmanager
from fastapi import FastAPI

MAX_CONCURRENT = 3

@asynccontextmanager
async def lifespan(app: FastAPI):
    timeout = httpx.Timeout(12.0, connect=3.0)
    limits = httpx.Limits(
        max_connections=MAX_CONCURRENT,
        max_keepalive_connections=MAX_CONCURRENT,
    )
    async with httpx.AsyncClient(
        headers={"Authorization": f"Bearer {os.environ['HF_TOKEN']}"},
        timeout=timeout,
        limits=limits,
    ) as client:
        app.state.http_client = client
        yield

app = FastAPI(lifespan=lifespan)

The 3-second connect timeout covers establishing a connection. The 12-second default applies to other timeout types. These are example values, not a general recommendation. A classifier may need less time, while a large generation request may need more.

max_connections=3 keeps the HTTPX pool aligned with this one-process example. A pool limit is still not a replacement for admission control. Without a semaphore and a short queue deadline, excess tasks can wait inside the pool instead of receiving the deliberate response that our API defines.

Guard the Remote Call with a Semaphore

Now build the small service object that owns the semaphore. asyncio.wait_for() gives slot acquisition its own deadline. The finally block is essential because it runs after success, timeout, parsing failure, or another exception.

import asyncio
from dataclasses import dataclass

@dataclass(frozen=True)
class ServiceProblem(Exception):
    code: str
    status_code: int
    message: str

class SentimentService:
    def __init__(
        self, client, max_concurrent=3, queue_wait_seconds=0.035
    ):
        self.client = client
        self.slots = asyncio.Semaphore(max_concurrent)
        self.queue_wait_seconds = queue_wait_seconds

    async def classify(self, request_id: str, text: str):
        try:
            await asyncio.wait_for(
                self.slots.acquire(), timeout=self.queue_wait_seconds
            )
        except TimeoutError as exc:
            raise ServiceProblem(
                "local_capacity",
                503,
                "The service is busy; retry after a short delay.",
            ) from exc

        try:
            response = await self.client.post(
                INFERENCE_URL,
                json={"inputs": text, "parameters": {"top_k": 1}},
            )
            response.raise_for_status()
            prediction = response.json()[0]
            return {
                "label": str(prediction["label"]),
                "score": float(prediction["score"]),
            }
        finally:
            self.slots.release()

Do not release the slot before the response has been read and checked. The protected region should cover the expensive downstream work. Also do not release only on success. One forgotten release after an exception reduces capacity permanently until the process restarts.

This first version isolates the capacity rule. The request_id is used by the complete build to label test events; it is not sent to Hugging Face. The next section replaces the minimal response handling with stable error mappings. The code calls an asynchronous HTTP method with await inside an async def function. FastAPI’s async explanation recommends this pairing when the client library provides awaitable network operations. Calling a blocking client from this path would stop the event loop while the request waits.

Translate Provider Failures into Stable API Errors

Callers should not receive a raw HTTPX exception or the provider’s full response. The ServiceProblem class above carries a public error code, HTTP status, and safe message while keeping the original exception internal.

Inside classify(), replace the simple request-and-parse block with the code below. It catches a network timeout, inspects error statuses before parsing JSON, and checks the response shape instead of assuming every HTTP 200 contains the expected fields. Keep this code inside the existing outer try block so its finally clause always releases the slot.

try:
    response = await self.client.post(
        INFERENCE_URL,
        json={"inputs": text, "parameters": {"top_k": 1}},
    )
except httpx.TimeoutException as exc:
    raise ServiceProblem(
        "upstream_timeout", 504, "The model service timed out."
    ) from exc

if response.status_code == 429:
    raise ServiceProblem(
        "upstream_rate_limited", 503, "Retry later."
    )
if response.status_code >= 500:
    raise ServiceProblem(
        "upstream_failed", 502, "The model service failed."
    )
if response.status_code >= 400:
    raise ServiceProblem(
        "upstream_request_rejected", 502,
        "The model service rejected the proxy request.",
    )

try:
    prediction = response.json()[0]
    result = {
        "label": str(prediction["label"]),
        "score": float(prediction["score"]),
    }
except (ValueError, TypeError, KeyError, IndexError) as exc:
    raise ServiceProblem(
        "upstream_payload", 502, "Unexpected model response."
    ) from exc

HTTP 504 means the dependency did not answer within the configured timeout. HTTP 502 means this proxy received an unusable response from its upstream dependency. This design maps the provider’s 429 to 503 because the proxy’s dependency is temporarily unable to serve the request; another API may choose to preserve 429. Document the choice so clients know when and how to retry.

Finally, connect the service to the endpoint. The complete artifact factory stores the service in application state during lifespan. The shortened endpoint below shows the public behavior:

from fastapi import HTTPException, Request

@app.post("/feedback-sentiment", response_model=SentimentResponse)
async def feedback_sentiment(
    payload: FeedbackRequest, request: Request
) -> SentimentResponse:
    service = request.app.state.sentiment_service
    try:
        result = await service.classify(payload.request_id, payload.text)
    except ServiceProblem as problem:
        headers = {"Retry-After": "1"} if problem.status_code == 503 else None
        raise HTTPException(
            status_code=problem.status_code,
            detail={"code": problem.code, "message": problem.message},
            headers=headers,
        ) from problem

    return SentimentResponse(
        request_id=payload.request_id,
        label=result["label"],
        score=round(result["score"], 4),
        model=MODEL_ID,
    )

The Retry-After header gives a simple hint for temporary 503 responses. It does not mean every client should retry at exactly one second. Production clients should use a bounded retry policy with a small random delay, often called jitter, and stop retrying permanent request errors.

Prove the Limit Without Calling a Live Model

A concurrency test must send requests at the same time. A loop that awaits one response before sending the next never fills more than one slot.

The build uses httpx.MockTransport, which HTTPX documents as a way to return controlled responses without network access. The handler waits 90 milliseconds for each burst case. The service records how many admitted provider calls are active. FastAPI itself runs through httpx.ASGITransport, so request parsing, the endpoint, admission control, and response mapping remain real.

This is the central test pattern:

burst_results = await asyncio.gather(
    *(send_request(row) for row in burst_rows)
)

assert service.max_active_calls == 3
assert sum(row["outcome"] == "success" for row in burst_results) == 3
assert sum(
    row["outcome"] == "local_capacity" for row in burst_results
) == 5

asyncio.gather() schedules all eight request coroutines together. The first three acquire slots. The other five wait for at most 35 milliseconds. Since the mocked provider holds each admitted call for 90 milliseconds, those five stop locally before a slot opens.

Run the complete reproducible program from the repository:

.venv-mlm-tutorials/bin/python tutorial-pipeline/work/57bad2f29aa1/build_tutorial.py

It printed this verified summary:

burst_requests=8 forwarded=3 local_503=5 max_active=3
E01: status=200 outcome=success
E02: status=504 outcome=upstream_timeout
E03: status=503 outcome=upstream_rate_limited
E04: status=502 outcome=upstream_failed
E05: status=502 outcome=upstream_payload

Read the first line as a capacity check. Eight requests reached FastAPI together, three crossed the transport boundary, and five received the planned local 503 response. The observed remote concurrency never exceeded three.

The remaining lines prove that different dependency failures remain distinguishable. A timeout is not reported as a malformed response, and a provider rate limit is not reported as a successful empty result. You can inspect every measured row in the verified request results.

The green bars are the three forwarded requests. The orange bars are not failed Hugging Face calls; they never reached the mocked provider. The exact milliseconds can move slightly between computers, but the asserted counts and maximum concurrency are the durable result.

This test proves the local control flow under a deterministic transport. It does not measure Hugging Face speed, sentiment accuracy, production throughput, or an appropriate limit for another service. Keep a smaller live integration test to verify the current token, model availability, router behavior, and real response shape.

Common Mistakes and Troubleshooting

Using a synchronous client inside async def. A blocking call can stop other work on the event loop. Use httpx.AsyncClient and await, or place synchronous work behind a design that does not block the event loop.

Creating one client per request. Repeated clients lose connection pooling and need repeated cleanup. Create one during FastAPI lifespan and close it at shutdown.

Treating a connection-pool limit as complete backpressure. A pool limit caps active connections, but waiting tasks can still collect inside the process. Add an explicit admission deadline and a response that callers understand.

Forgetting finally. If a timeout or parsing error skips release(), the service eventually reports no capacity even when no calls are running. Put release in the smallest correct finally block.

Retrying without a total deadline. Retries consume a slot and extend request time. If you add retries, keep them few, add a delay, honor one overall deadline, and retry only failures that may recover. The separate LLM gateway tutorial covers retry and provider-fallback decisions.

Assuming one semaphore limits every worker. The semaphore lives in one Python process. Four Uvicorn workers with a limit of three may create up to 12 outgoing calls. Divide a global provider budget across workers or use an external shared limiter when the limit must cover several processes or machines.

Choosing three and 35 milliseconds without measurement. They are teaching values. Start from the provider quota and your response-time target. Run load tests with representative request sizes, monitor admission rejection and timeout rates, and change one control at a time.

Logging tokens or complete user text. Keep secrets in environment or secret storage. Decide whether request text may contain personal or confidential data before logging it. Prefer request IDs, timing, outcome codes, and carefully selected metadata.

Testing only successful JSON. A provider can time out, return 429 or 5xx, or return HTTP 200 with a changed body. Exercise each branch with a controlled transport and keep a live contract test for changes outside your code.

Recap and a Safe Next Step

The lasting pattern is share, admit, call, release, translate:

• share one asynchronous HTTP client for connection reuse;
• admit only a known number of outgoing calls;
• stop waiting after a deliberate local deadline;
• release capacity in finally; and
• translate timeouts, provider failures, and invalid responses into stable API errors.

The semaphore protects one process from sending more remote work than planned. The queue deadline prevents unlimited waiting, while HTTPX timeouts bound network stages after admission. Keep these controls separate because each answers a different failure mode.

Next, replace the mock transport in one small integration test and send a harmless sample to the current HF Inference model. Record the model ID and test date, but keep deterministic transport tests as the main suite. That combination checks both your own concurrency rules and the changing external contract without making every code change depend on a live service.