The gateway earns its keep

Every model call Klippy makes already goes through your AI Gateway, default, since course A. So far it only routes and falls back. In this lesson it starts to earn its keep: it labels each call, so you can see who spent what, and it caches the one answer that's the same for everyone.

1. Label every call

model() now takes a Call: which part of Klippy is asking, and whose Klippy it is. One gateway() helper puts that on every request:

The metadata shows on each request in the gateway's logs. It also scopes spend limits: a dollar budget per model, provider or metadata value that blocks requests once it's spent. So a user label is what a per-user daily limit keys on. Claude and the fallback get the same labels, because a fallback model inherits the options of the chain it's part of.

Labels only: never the question, a note or a token. Note that the gateway's own log keeps each prompt and reply by default. For a personal Klippy that's your own data in your own account, but know it's there.

A chat turn says whose Klippy is talking (gh-<id> from the web app):

For that to compile, model() takes the Call (with /ask's as the default), and the ingest step's contextModel() labels itself:

ask() takes a Call too and passes it on, so the check below can label (and cache) its probe:

And the note tidier says what it is:

2. Caching needs a key

The obvious way to cache is createAI({ cacheTtl: 300 }). On a new gateway, that caches nothing. AI Gateway's caching is off unless you turn it on, and cacheTtl only sets how long a cached answer lives for a request that already uses caching. What opts one request in is cacheKey. So Klippy sets both, and only for the check probe:

Why only the probe? It's the same question for every caller, so a fresh isolate or another datacenter can reuse the answer for five minutes instead of a new model call. A chat is personal and changes by the minute (the time, your notes). With a wrong key, a cached chat reply could answer one person with another's words. Cache only what's the same for everyone, and put the tag in the key, so a new version never serves an old answer.

ask() returns the gateway's cache status too:

Copy test/gateway.spec.ts. It swaps env.AI for a stand-in that records what it's asked, and checks that a chat carries its labels on both legs and no cache key, while the probe carries cacheKey and cacheTtl. No model or network is called. Ours: 22 test files, 98 tests passed.

3. Watch it hit

Three calls to the check, on wrangler dev (the Workers AI fallback answered, not Claude):

Calling it again on the same server says MISS too, but makes no model call: the module's own five-minute cache answered. Then we stopped the server and started a new one, with an empty module cache:

The same words, from the gateway's cache. (The tag still says e-09: it changes at the checkpoint.)

4. Set a spend limit

This one is in the dashboard, not the code: select the default gateway, then Settings → Spend limits. A rule is a dollar budget over a time window. Before each request, the gateway checks every rule, and once one is spent it answers 429 instead of calling the provider. Add one for the whole gateway, and one by the user metadata, split by value, so each Klippy gets its own budget. Know its limits, from Cloudflare's docs: a request's cost is recorded after it completes, so a burst of requests at once can go briefly over. Cost is a best-effort estimate from token counts. And only models with known pricing count. Budget alerts under Billing also email you, but they fire the next day, so they're no brake. We haven't set limits on the recording account. Do it before you deploy.

Your code matches tag f-02 (changes from f-01).