Building a Free-First AI Router: Architecture That Spends $0 First

The best free tiers are individually wasteful. Each one has a geometry — a reset clock, a per-minute meter, a depleting balance — that matches only part of your traffic. A daily token pool is idle at night while an interactive burst queues; a per-minute cap sits unused during a long batch job; a monthly credit balance quietly drains because nobody routed around it.

A router that treats all of that as one pool fixes the mismatch. It ranks every request against every available free shape, spends the shape whose clock fits the request, and only touches paid capacity for the residue no clock can catch. This is the architecture we would ship, built from the published limits we track.

The core idea: capacity has geometry, and geometry does not aggregate

Two free tiers with the same nominal size are not interchangeable. A daily pool and a per-minute ceiling differ in three ways that matter to a router:

• Reset period — when capacity comes back (every minute, every day at 00:00 UTC, every billing cycle).
• Depletion curve — whether usage drains a balance permanently or resets a meter.
• Collision behaviour — whether concurrent requests fight for the same budget or pass through independently.

Because those properties differ, free shapes can be unioned but not summed. A router that adds "10,000 Neurons plus 200,000 tokens plus $30 of credit" into one number has already lost, because the request that arrives at 23:59 UTC is a different request from the one that arrives at 00:01 UTC.

Layer 1: Classify the request before you route it

Every inbound request carries three properties that decide which geometry can serve it. Classification is cheap and happens before any provider is chosen.

1. Latency class. now (a human is waiting) or later (a job, pipeline, or evaluation). This decides whether per-minute shapes are eligible at all. A per-minute cap can absorb a burst now, but a later job should sit in a daily pool instead, where it costs nothing extra and collides with nothing.
2. Size class. Small (roughly under 8K tokens in and out), medium, or large. A 200,000-token-per-day pool (the shape we track on SambaNova's free tier) serves perhaps one large request before the job is done, no matter how many small ones would fit.
3. Repeat class. First call, or a repeated call sharing context with an earlier one. Only the repeated class earns cache-read pricing, and it is the class where free capacity stretches furthest.

[object Object]

A router that routes on vendor preference instead of request shape is a load balancer with an opinion.

Layer 2: The geometry-aware ranking

Rank available free capacity by shape complement, not by token count.

The rule that makes the ranking work: never burn a non-renewing shape when a renewing shape is idle. A credit spent at 10:00 UTC that could have been a daily-pool call spent at 01:00 UTC is money — not because the token costs more, but because the credit never comes back on a window.

Ranking is therefore two passes. First, filter shapes whose clock can serve this request at all (a 200K-token pool cannot serve one 300K-token job; a rolling per-minute cap is a poor fit for a four-hour backfill). Second, order the survivors by renewability, cheapest-to-replace first, and prefer the shape whose clock is closest to being reset — idle capacity that is about to disappear is the capacity to spend.

Layer 3: Detect exhaustion the way the provider tells you

Each geometry has its own exhaustion signal, and the router must honour all three.

Three practical notes:

• Window caps. A 429 with a retry hint means wait, not fail. A client that treats 429 as a hard error converts a cheap free tier into an outage.
• Reset clocks. Schedule the reset as a first-class event. A pool that refills at 00:00 UTC should be re-enabled at 00:00 UTC, not rediscovered three hours later through a successful request.
• Balances. Where a documented balance endpoint exists — OpenRouter, for example, exposes remaining credit and a free-tier flag through its limits API↗ — poll it and stop routing at the threshold you choose. A balance exhaustion discovered after the fact is a failed request, and on some platforms a negative balance blocks even the free models.

Log every exhaustion event with a shape label. After two weeks, that log tells you which geometry your traffic is actually shaped like, which is the input to the next routing decision.

Layer 4: The paid overflow path, and how to keep it honest

When all free shapes are exhausted, the request goes to paid capacity. This is where free-first designs quietly stop being free-first. Three rules keep the paid path honest.

1. The paid path is a price trigger, not a vendor choice. Route to the cheapest eligible paid rate rather than defaulting to your primary provider's paid tier. On a DeepSeek-style card, off-peak Flash input is cheaper than most premium models' list price — and the same card halves again outside the published peak windows.
2. Batch the paid tail. Anything later that crosses into paid should check the batch discount first. A 50% async discount is structural on several major providers and applies precisely to the overflow workload shape.
3. Budget the paid path at a number you will notice. A daily paid-spend alert that fires at a small threshold is the difference between "the free tier ran out and we found out at the end of the month" and "the free tier ran out and the router told us".

[object Object]

Why the union beats the best single tier

The arithmetic is the argument. Consider three shapes as published: a daily compute pool measured in normalized units, a daily token pool of 200,000 tokens per model, and a monthly credit balance. One daily pool has one zero-moment each day. A monthly credit has one zero-moment each cycle — but at realistic consumption it is spent long before the cycle ends, so it behaves like a one-shot allowance.

Union them and the zero-moments separate:

• When the daily token pool hits zero at some point in the afternoon, the per-minute cap is idle and can carry interactive traffic.
• When the per-minute meter saturates during a burst, the daily pool still has hours of capacity left.
• When both pools are exhausted, the monthly credit is the floor.
• When everything is exhausted, the paid path is the residue — and the residue is small precisely because three clocks were spent in sequence rather than one being destroyed first.

That is what free-first actually means. Not "use free APIs". Not "find the biggest free tier". It means shaping spending so that zero-cost capacity is exhausted on its own clock, and paid capacity is touched only by the residue no clock can catch.

An honest caveat: a union of shapes is a union of eligible capacity, not interchangeable quality. Pool A may serve a 4B open model while your workload needs a frontier model. Ranking must therefore filter on a model-quality requirement before it ranks on cost, or the router will happily route your hardest request to the cheapest capacity that cannot do the job.

The observability that makes it improve

A free-first router without logs is a guess with a scheduler attached. Record at minimum:

• Shape served, per request, as a label — not just the provider name.
• Exhaustion events with their signal type (429, reset, balance threshold) and the timestamp.
• Reset confirmations, so you can verify that a documented clock matches observed behaviour.
• Paid escalation events, with the token volume that triggered them. This is the number that tells you whether to add free capacity or reduce workload size.
• Token estimates versus actuals, by size class, so classification improves.

Two weeks of that data converts the ranking table from assumption to evidence.

Failure modes specific to free-first designs

• Silent escalation. The paid path absorbs everything and nothing alerts. Fixed with a spend threshold, not with better routing.
• Clock drift. You assume a daily reset happens at the documented hour but never verify it. Fixed by logging reset confirmations.
• Shape starvation. A high-priority shape never gets traffic because classification rarely marks requests eligible for it. Fixed by reviewing shape utilisation weekly.
• Quality leakage. Requests land on models that satisfy the cost rank but not the task. Fixed by filtering on a quality requirement before ranking.
• Retry storms. A 429 without backoff multiplies load across every shape at once. Fixed with per-shape concurrency limits.

How we verified this

How we verified this (2026-09-17): the capacity shapes in this architecture come from the providers' own documentation — Cloudflare's Workers AI pricing page for the daily Neuron allocation and reset, SambaNova's rate-limit documentation for the daily token pool, NVIDIA's NIM documentation for the default per-minute request ceiling, Groq's rate-limit documentation for per-model limits, and OpenRouter's limits documentation for the balance and free-tier flags. We have not published latency, uptime, or throughput measurements for these routes, because we have not run instrumented load tests against them; any number used here is a published limit or arithmetic over one.

What to build first, in order

1. Classification. Latency, size and repeat class on every request. Nothing else works without it.
2. Two shapes, not six. Start with one daily pool and one per-minute cap. Two clocks already demonstrate the union effect.
3. Exhaustion logging. Shape labels and timestamps from day one; you cannot tune what you did not record.
4. The paid path behind a threshold. One alert, one cheap fallback model.
5. Batch for the tail. Add the async path once free capacity is demonstrably saturated.

Build in that order and you will have a router whose free tier is exhausted on schedule and whose bill is a rounding error — which is the entire point of free-first.