Retry model API calls under rate limits

Goal

Make a model-API client that reacts correctly when a call is rejected for a rate limit, an upstream failure, or an exhausted balance.

Prerequisites

• A working model-API client (see Call the model API from Python).

What to branch on

Branch on the HTTP status, not the wording of the error message. A 429 can come from a per-key requests-per-minute cap, an account limit, or an upstream provider. The per-key limiter states the retry interval in the error message but does not set a Retry-After header; an upstream 429 may carry the provider's Retry-After, which the API does not forward. So a client honors Retry-After when present and otherwise applies its own bounded backoff.

Retry 429, 502, 503, and 529. Do not retry 400 or 401 — correct the request or the credential instead. A 402 means the account is out of credits; add credits rather than retrying (see Check your model-API credit balance).

A minimal backoff loop

import time, random, httpx

def call_with_retry(send, max_attempts=5):
    for attempt in range(max_attempts):
        resp = send()
        if resp.status_code not in (429, 502, 503, 529):
            return resp  # success, or a 4xx to correct rather than retry
        retry_after = resp.headers.get("Retry-After")
        if retry_after:
            time.sleep(float(retry_after))
        else:
            time.sleep(min(2 ** attempt, 30) + random.random())
    return resp  # surface the final provider-shaped error to the caller

Expected behavior

A transient 429/5xx is retried with backoff and eventually succeeds; a 400/401/402 returns immediately for you to fix; after the attempt budget is spent, the final provider-shaped error reaches the caller instead of being swallowed.

When it breaks

• Retries never stop — the loop is retrying a status it should not. Confirm only 429, 502, 503, 529 are retried.
• A 429 repeats immediately — you are not honoring Retry-After, or the per-key cap is genuinely exceeded; read the interval in the error message.

Source

Limits and retries and Errors.