99.2% of our consumption wasn't user demand
One day Isobaria burned through its data providers' quota. Splitting the spend by who caused it showed that people were 0.8% of it. What we changed, what we ruled out, and why paying for a plan was the wrong answer.
On August 20, at 06:00 UTC, Isobaria was at 101% of its daily Open-Meteo budget and 100% of its AEMET budget. The call limiter, which already existed, did its job: it degraded gracefully and nothing went down. But a limiter that works doesn't answer the uncomfortable question: who spent all this?
Until then there was a single budget per provider. One number going up. And a number going up can't tell the difference between a person checking the forecast for their town and a process that's spent all night requesting pages nobody will ever read.
Splitting the spend by who caused it
What we did was change the question. Instead of "how much have we used," "who asked for it." Three classes: people, crawlers and scheduled jobs. With that day's real consumption, for Open-Meteo:
| Who | Calls per day | Share |
|---|---|---|
| Crawlers | ~4,161 | 59% |
| Scheduled jobs | 2,932 | 41% |
| People | ~60 | 0.8% |
Read that slowly: 99.2% of what we were spending hadn't been requested by any person. That wasn't a calibration detail. It was the whole premise.
And we should be honest about the 0.8%, because it's a measurement, and measurements come with fine print. The ~60 calls come from about 15 pages viewed per day, which is the average over the days when telemetry managed to write. Counting every day in the two weeks, it's 5.4. The difference is that several days in that window show zero because the quota on the key-value store where we keep analytics was also exhausted, so nothing could be written. The demand measurement was suffering from the very problem it was trying to measure. With 15 or with 5.4 the conclusion is the same, which is why both figures are written down in the document.
The extreme case
The most instructive example is the confidence map. Its scheduled job was spending 2,676 calls a day. Over fourteen days the screen got 5 visits.
37.464 llamadas / 5 visitas = 7.493 llamadas por visita
The map wasn't badly built. Its cadence had been set by the physics of the data: the models publish every six hours, so it was recalculated four times a day. Nobody had ever checked whether anyone was looking at it. It's a very natural way to get things wrong: the rhythm is set by what the source allows, not by what the product needs.
The obvious fix that arithmetic killed
There was already a ticket for this: precompute every populated cell on each model cycle, so that no visit has to wait or pay. It sounds responsible. And it's what we would have done without running the numbers:
748 celdas × 3 modelos × 4 ciclos = 8.976 puntos al día, haya visitas o no
Against a demand of about 60 calls, that locks the cost in at roughly 150 times what's actually being asked for. It swaps a variable waste for a fixed one and calls it a solution.
The ticket wasn't wrong. It was badly timed. The document that replaces it also works out the break-even point: precomputing pays off from 2,244 cold pages a day (8,976 divided by the 4 calls one page costs). Today we have 5.4. The ticket isn't closed: it's deferred, with that number written down as the trigger. When traffic crosses it, we implement it without having the argument again.
What we decided
The budget is split by class, and every consumer that isn't a person has to justify its share with measured usage:
- People get no quota of their own. Their ceiling is the provider's. If people ever exhaust the quota, that's a good problem, and you solve it by paying for a plan, not by degrading.
- Crawling is an investment we choose, not one we discover. It gets its own quota of 200 page requests a day (about 800 calls), which indexes the whole catalog in around six weeks. When it runs out, the answer is a 503, even if the global budget is at 20%.
- Every scheduled job justifies its cadence with its usage, not with when the model publishes. The map grid goes from four passes a day to one: from 2,676 calls to 669.
| Consumer | Before | After |
|---|---|---|
| Map grid | 2,676 | 669 |
| Pre-warming | 256 | 256 |
| Crawlers | ~4,161 | ≤ 800 |
| People | ~60 | ~60 |
| Total | ~7,093 | ~1,785 |
The total is 25.5% of the budget, leaving 74.5% free. And the share of useful spend goes from 0.8% to 3.4%, which is still small because traffic is small. The difference is that now the rest is bounded and chosen by someone.
What we ruled out, and why
Blocking long-tail crawlers. It contradicted a decision we'd already documented: the sitemap argues that the pages for small municipalities are crawlable on purpose. Giving up that tail to save quota is selling the asset to pay the rent.
Paying for an Open-Meteo plan. It fixes the symptom, and we ruled it out as a first step, not forever. Paying without fixing the split is buying more quota to keep spending 99% of it on something nobody asked for. When people are the main consumer, upgrading will be a growth decision rather than a patch.
Keeping the global budget. The argument is in the ADR itself: it can't tell who is spending, so whoever arrives last gets stuck with the bill. On August 20, the ones who arrived late were the people.
What it costs, and what's still unresolved
The long tail takes weeks to get indexed, not days. That's the explicit price of the crawl quota, and it's reversible by raising it. The confidence map can now be up to 24 hours old, and the screen has to say so, because an old map with no warning is exactly what the product forbids itself from doing.
The counter is more complex: three classes where there was one, and a classification that isn't infallible. That's why the global ceiling stays behind it, as a safety net: splitting by class organizes the spend, it isn't the emergency brake.
And there's one limitation the ADR spells out because we found it while implementing. To know whether a page was requested by a person or a crawler, we'd have to read the request headers inside the page itself, and in this version of Next.js that makes it dynamic: it would stop being cached, which is precisely what keeps its cost down. We'd be trading one cost for a worse one. So pages count as "person" knowingly, and attributing that traffic gets solved somewhere else. A number you know overstates people, and leave written down that way, beats an exact one that costs more than it measures.
And a warning about the figures in the table above: they're design numbers, not an after-the-fact measurement. The acceptance criterion calls for one measured week, with the share of people above 1% and total consumption below 7,000, and it sits in the document as an unchecked box. The signal that reopens the decision is written down too: if people become more than 50% of consumption, the split by class is no longer needed.
What we're taking away
A total counter tells you how much, not who. Until we split the spend by who caused it, the problem looked like a thresholds problem: tuning the 80%, the 100%. It wasn't. No threshold fixes the fact that 99% of consumption belongs to something nobody asked for.
The cadence of a scheduled process has to be justified by its usage, not by the physics of the data. That the source publishes every six hours tells you how often you can refresh, not how often you should. The map that cost 7,493 calls per visit is what happens when nobody supplies the second number.
A deferral with no trigger is just forgetting politely. Per-cell precomputing wasn't rejected: it was deferred with a number, 2,244 cold pages a day. That's a question you answer by looking at a dashboard, not by arguing about it again.
If your product consumes a third-party API with a quota and you don't know how much of that consumption comes from real users, write to me: it usually takes an afternoon of measuring, and what comes out is almost never what you expected.