How to Fail Over Between Free AI API Endpoints Without 429s
How to Fail Over Between Free AI API Endpoints Without 429s
The standard pattern for building on free tiers is two providers: a primary and an overflow. The standard mistake is choosing the overflow by size — which free tier gives the most tokens — when what actually determines whether failover works is shape. Two free tiers can carry identical daily volume and still be architecturally useless to each other, because their limits bite in different places, on different clocks, at different times.
This is the operational companion to our provider directory: not which free tier is biggest, but how to wire two of them together so that one provider's bad day does not become your outage.
Why rate-limit shape matters more than rate-limit size
A free tier is not a tank of tokens. It is a set of meters, and each meter fails in a specific way. When you route a request around a failing provider, the question is not "does the overflow have capacity this month?" It is "is the overflow's limiting meter idle right now?"
If both providers are limited by the same kind of meter, they are two readings on the same clock. When your primary starts rejecting bursts, the fallback is usually rejecting the same burst, because the burst is what triggered the cap — not the volume. That is how teams end up with a fallback layer that has never once absorbed a failure.
The three shapes of a free-tier limit
Almost every free tier in this space reduces to one of three geometries.
1. Per-minute request caps (RPM). Groq's published developer-plan limits put its 120-billion-parameter open-weight model at 30 requests per minute (Groq rate limits). NVIDIA's NIM API is documented around a comparable per-minute request ceiling (NVIDIA NIM docs). RPM caps bite bursts: your daily volume can be tiny and you will still receive a rate-limit error if the traffic arrives inside one 60-second window. Fan-out and parallelism are what kill you here.
2. Daily pools with hard resets. Groq's docs also cap the same model at 200,000 tokens per day and 1,000 requests per day; Cloudflare Workers AI gives every account a free allocation of 10,000 Neurons per day (Workers AI pricing); the Gemini API free tier publishes per-model requests-per-day quotas and states that RPD quotas reset at midnight Pacific time (Gemini API rate limits). Daily pools bite sustained volume, and they fail as a cliff: requests after the pool empties do not queue on most implementations — they fail until the reset.
3. Depleting credits. Modal's Starter plan includes $30 per month of free compute (Modal pricing); Google Cloud's free trial preloads $300 of Welcome credit valid for 90 days (Google Cloud free trial). These bite spend, and the pool shrinks continuously. There is no busy window and no reset-at-midnight — just a balance that trends down.
| Geometry | Representative free tier | What it absorbs | Failure signature |
|---|---|---|---|
| Per-minute RPM | Groq (30 RPM on gpt-oss-120b), NVIDIA NIM | Short bursts, spiky fan-out | 429 with retry hint inside a minute window |
| Daily pool + reset | Cloudflare Workers AI (10,000 Neurons/day), Groq (200K tokens/day), Gemini API free tier RPD | Sustained daily volume | 429 that persists until the daily reset |
| Depleting credit | Modal ($30/month Starter), Google Cloud trial ($300/90 days) | Exhaustion of the other two | Success, then hard stop until top-up |
The architectural rule falls out immediately: a failover partner only helps if its geometry is idle while yours is maxed. An RPM-capped tier overflows best into a daily pool, because bursts are cheap inside a pool. A daily pool overflows best into credits, because spend is the one meter a pool cannot exhaust. Two identical shapes fail together.
How we verified this
We checked the limit values in this article on 2026-09-17 against each provider's own documentation — Groq's rate-limits page for RPM, RPD, TPM and TPD values, Cloudflare's Workers AI pricing page for the daily free allocation, Modal's pricing page for the Starter plan's included monthly compute, Google's free-trial page for the credit amount and validity window, and Google's Gemini API rate-limit docs for the reset behaviour. Where a provider publishes per-account or per-model values that change frequently, we describe the shape and link the source rather than freezing a number into the article.
Detecting what actually failed before you route around it
Failover logic that treats every error as "try the other one" will thrash. The status code tells you what happened, and each cause has a different correct reaction.
| Signal | What it means | Correct reaction |
|---|---|---|
| 429 with a retry hint | Window-level congestion on this provider | Wait in place for the hint's duration, then retry the same provider |
| 429 without a retry hint, repeated | Structural cap reached (daily pool or sustained RPM) | Route to the overflow tier and open the circuit for this provider |
| Payment/credit error | Account state, not rate state | Stop routing and surface a billing alert — never retry in a loop |
| Upstream error inside an aggregator response | The provider behind the model, not the gateway | Read the error metadata before declaring the route dead |
Groq's documentation states that a retry-after response header is set when the rate limit is hit and the 429 is returned, alongside always-present headers such as x-ratelimit-limit-requests, x-ratelimit-remaining-requests, x-ratelimit-limit-tokens and their reset variants. Those headers are the cheapest observability you will ever get: they tell you how much quota is left before you hit the wall, so your router can decide rather than react.
On an aggregator the picture is subtler. OpenRouter's rate-limit documentation explains that a rate-limit error can originate at the platform or at the upstream provider serving your request, that the upstream provider's original code appears in the error metadata when available, and that fallback routing already retries other providers for the same model before the error reaches you. The same documentation separates credit limits from rate limits explicitly: a negative account balance produces a payment-required error including for free models, and the fix is adding credits, not backing off.
A minimal router that respects those distinctions:
[object Object]Three properties matter here. A payment or auth error trips the circuit for an hour instead of burning attempts. A structured 429 with a retry hint waits in place rather than failing over, because failing over for a window-level limit is what causes thundering herds across providers. And no path retries more than a bounded number of times, so a total provider outage degrades to a clean error instead of a retry storm.
Worked example: sizing the overflow pool
Take Groq's published daily pool for a mid-size open-weight model: 200,000 tokens per day, with 30 requests per minute and 8,000 tokens per minute on the same model.
Now model your own traffic. Suppose your application sends 40 requests per hour at roughly 2,000 tokens each:
- Hourly consumption: 40 × 2,000 = 80,000 tokens per hour
- Daily pool exhaustion: 200,000 ÷ 80,000 = 2.5 hours
Your primary is finished before lunch. Everything after that belongs to the overflow. A burst-friendly fallback will not save you, because the overflow's daily meter is what is exhausted — if you pick another daily pool as your fallback, it empties on the same schedule. The correct partner is a credit-based tier, which has no daily clock at all.
Size it from the worst case: the primary is dead from 10:30 onward, so the overflow must absorb roughly 80,000 tokens per hour for the remaining 13.5 hours — about 1.1 million tokens in a single day. Price that against published paid rates for a small, fast model (Google's Flash-Lite-class rates are published at $0.30 per million input tokens and $2.50 per million output tokens on the Gemini API pricing page). Assuming a 90/10 input-to-output split:
- Input: 1.008M × $0.30 / 1M = $0.302
- Output: 0.112M × $2.50 / 1M = $0.280
- Worst-case day total: about $0.58
A single catastrophic day of primary failure costs roughly sixty cents on a credit-based overflow — and Modal's Starter plan includes $30 per month of free compute, which covers dozens of such days. That is the entire argument for shape-based pairing: the overflow you need is usually cheaper than the argument you had about buying it.
Keeping the overflow path warm
A failover path you have never executed is a hypothesis, not a plan. Free tiers have cold-account quirks that only appear on first contact: unverified trial keys that refuse certain models, regional availability differences, a billing profile that was never completed, and error formats that your parser has never seen.
Send a weekly canary: one tiny request to each configured provider, once per week, on a schedule. Log the status code, the rate-limit headers and the latency. It costs a handful of tokens and it converts three separate unknowns — reachability, authentication and response shape — into knowns. The first time the canary fails is a much better day to discover a broken fallback than the day your primary goes down.
The canary also gives you something else: a per-provider history. When you can see that provider B has been returning 429s every Tuesday between 09:00 and 11:00 for a month, you stop treating it as a reliable overflow and start treating it as a scheduling problem.
Circuit-breaker rules that stop the thrash
Four rules cover almost every real incident:
- Never retry a payment or auth error. Log it, alert a human, trip the circuit. Retrying a blocked account is the single most common wasted-hours bug in free-tier systems.
- Honour retry hints before failing over. A window-level 429 is not a reason to send your traffic to a second provider; it is a reason to wait a few seconds. Failing over at the first 429 is how a single congested provider becomes two congested providers.
- Trip on repetition, not on a single error. Open the circuit after a small number of consecutive structural failures, then half-open it after a cooldown with a single probe request.
- Track quota, not just errors. Where a provider returns remaining-quota headers, store the last known value and prefer the provider with headroom over the provider that answered last. This turns failover from reactive to proactive.
A reference architecture that follows from the shapes
| Role | Geometry to pick | Why |
|---|---|---|
| Primary for sustained volume | Daily pool with a scheduled reset | Absorbs the bulk of the day cheaply and refills on a clock you can plan around |
| Overflow for pool exhaustion | Depleting credit | Has no daily clock, so it cannot be exhausted by the same event |
| Burst relief | Per-minute RPM cap | Handles short fan-out the pool cannot shape, with retry semantics you can honour |
| Detection layer | Status codes plus quota headers | Distinguishes congestion from exhaustion from billing from upstream failure |
Notice that this design never routes from one partner to another of the same shape. That is deliberate. The value of a second provider is not redundancy of volume; it is redundancy of failure mode.
Limitations and failure modes to plan for
- No free tier carries a service-level agreement. The limits are the product. If a workflow has a paying customer waiting on it, a free tier must not be the only path.
- Shared upstreams defeat fallback. Two aggregators fronting the same model on the same infrastructure can fail together. Provider diversity means different upstream capacity, not just different API keys.
- Tokenizers differ. A 200,000-token daily pool on one provider is not the same amount of text as 200,000 tokens on another. Budget in characters or requests as a sanity check, and never compare free tiers on token counts alone.
- Retry budgets amplify. Ten parallel workers each retrying three times is thirty upstream attempts for one logical job. Cap concurrency and attempts together, or your retry policy becomes a self-inflicted denial of service.
- Limits move. Values published today can change next week, and some providers only publish exact per-account numbers inside their console. Verify before you size, and re-verify when you scale.
FAQ
Is it worth paying for a tiny credit balance just to have a stable overflow? Often yes. A minimal top-up on a credit-based provider removes the daily-clock coupling entirely, which is the property that makes it useful as an overflow. It is usually one of the cheapest reliability purchases available to a free-tier project.
Can I use two free tiers as primary and fallback if they are from different vendors? You can, but check the geometry first. Two daily-pool tiers from different vendors still fail together under sustained load, because the trigger is your traffic shape, not their infrastructure.
Should the router prefer the cheapest provider or the fastest? Prefer the one with remaining quota headroom, then the fastest. Optimising for price on a free tier is meaningless — the correct optimisation is staying inside every meter simultaneously.
What should I log to debug failover? Per attempt: provider name, status code, retry hint, remaining-quota headers, upstream provider code from the error metadata, and the attempt number. Those six fields explain almost every failover incident after the fact.
Does streaming change any of this? Yes, at the edges. If a limit is hit after a streaming response has started, the error can arrive inside the stream rather than as an HTTP status, with the finish reason indicating an error instead of a normal stop. Your client must handle mid-stream failure explicitly if you stream.
Bottom line
The tokens are the commodity; the shape is the architecture. Two free tiers of equal size with complementary geometries outperform one free tier of triple the size with no partner, because the first system survives its own bad day and the second one simply fails at a higher volume. Build for the shape, size for the worst case, keep the failover path warm, and treat every status code as a different instruction rather than the same one.
- Why does my fallback provider get rate limited at the same time as my primary?
- Because both limits bite in the same place. If your primary is a daily token pool and your fallback is also a daily token pool, the same traffic spike exhausts both meters on the same clock. Pair a daily pool with a credit-based tier so the two limits are never maxed simultaneously.
- What is the difference between a 429 and a 402 on a free tier?
- A 429 means you are being rate limited and the request may succeed later, so backoff or route around it. A 402 means the account's payment or credit state is blocking the request, and OpenRouter's documentation confirms this can block even free models when the balance is negative. Backing off will never fix a 402.
- Should I retry a rate-limited request on the same provider or switch providers?
- If the response carries a retry-after header, the limit is window-level congestion and you should wait it out in place. If 429s continue without a retry hint, you have hit the structural cap and should route to your overflow tier instead, with a circuit breaker to stop repeated attempts.
- How much overflow capacity do I actually need for a free tier?
- Size it for the worst case, not the average: assume your primary is exhausted for a full day and your overflow must absorb the entire day's traffic. On credit-based tiers that worst case is often measured in cents, so the correct answer is nearly always more than you think you need.
- Do free tiers offer any uptime guarantee?
- Generally no. Free tiers are capacity-shaped, not contract-shaped: the published limits are effectively the product. Treat every free endpoint as best-effort, keep at least two independent providers wired in, and never let a single free tier sit on the critical path of a paid customer flow.
- How do I know if a 429 came from the aggregator or the upstream model provider?
- On aggregators such as OpenRouter, the error metadata carries the upstream provider's original code, and fallback routing already retries other providers for the same model before the error reaches you. Log the metadata field before you conclude your route is dead.