TOKENS · WINDOWS · LIMIT CHECKS · STREAM · LIVE COUNTERS · LEDGER
System design: a usage meter that counts every LLM token in seconds
Every request you send to a coding assistant uses tokens, and your plan allows so many per five hours and per week. The meter behind /usage has to price each request, stop you at the limit before the next one starts, and show the new percent seconds later, for millions of people at once.
Tidewell is an AI coding assistant. Developers run its tide CLI and IDE plugins, which send model requests through Tidewell’s model gateway to its model servers. Each plan allows so much model use per five hours and per week, and /usage and a settings page on the web show how much is left. You design the meter that sits beside the gateway.
Here is where the design ends up. Most boxes won’t mean much yet; each chapter adds one or two.
MapThe dashed outline is the AWS Region and the solid one the VPC, the private network. Arrows show who calls or writes to whom.
The chapters follow the order a system design interview runs in: requirements, estimates, the API, a simple design, then deep dives on what it misses. The design runs on AWS in us-east-1, with list prices and quotas checked on 21 November 2026. The scale numbers are assumptions stated in chapter 01, and Tidewell is fictional. You need to know what an HTTP request, a database table and a queue are. Tap any figure to open it full screen and zoom. Dotted terms are tappable.
01
Requirements: what the meter must do, and how well
/usage in a coding assistant prints how much of your plan you have used: 41% of the current session, resetting at 14:12. Behind that line sits a meter, a service that counts the tokens of every model request against each user’s plan, for millions of developers at once. Before you draw any of it, write down what it must do, for how many people, and how well. What must a usage meter do, and how good is good enough?
STEP 01 / 03
THE METER SITS BESIDE THE GATEWAY
Fig 1.1The accent arrows are everything that enters the meter; faint parts arrive in later chapters.
The meter has two kinds of caller. The model gateway, the service every Tidewell model request passes through, asks it before each request may start and reports the request’s token counts after the reply ends. People call it too: the tide CLI’s /usage command and the web app’s usage page read how much of the plan is left.
So three things cross the boundary: a question (may this request run?), a report (this request used these tokens) and a read (how much of my plan is gone?). The model gateway and the model servers already exist. You design only what sits behind those three arrows.
STEP 02 / 03
WHAT IT MUST DO
Fig 1.2The accent row is the one that adds a check before every request.
The functional requirements say what the meter does for its users, and five answers settle them: a limit counts tokens priced as money, two windows apply, a request past a limit is refused or paid from usage credits, users read their usage in the CLI and on the web, and each team seat has limits of its own.
That gives three requirements. F1: every model request’s tokens count against the user’s plan, by window, model and token type. F2: a request past the 5-hour or weekly limit is refused with its reset time, or paid from credits when the user has turned them on. F3: users see percent used, reset time and tokens for each window.
Out of scope: payment and invoicing, turning text into tokens (the transformer tutorial covers the tokenizer), per-minute API rate limits, organisation analytics and abuse detection. Saying so out loud makes leaving them out a choice.
YOU ASK
THE ANSWERIT DECIDES
What does a limit count: requests, tokens or money?
the answer Tokens, priced at the model’s list price, by model and token type.
it decides One cost counter per window, plus token totals for display (chapter 04).
Which time windows apply?
the answer A 5-hour session that opens at the first request, and a week with a fixed reset per account.
it decides Two counters per user, and who opens a session (chapter 05).
What happens when a user reaches a limit?
the answer The request is refused with the reset time, or paid from usage credits if the user turned them on.
it decides A check before every request, and a credit ledger (chapter 05).
Where do users see their usage?
the answer The /usage command and the status line in the CLI, and a settings page on the web.
it decides One read API, and usage carried on model responses (chapters 06 and 10).
Do team seats share one pool?
the answer No. Each seat has its own limits; the organisation sets a monthly cap on credits.
it decides Per-user keys, and one organisation key only for credits (chapter 05).
STEP 03 / 03
HOW WELL IT MUST DO IT
Fig 1.3Each row names the chapter that meets it; the accent row is the tightest number.
The non-functional requirements say how well, each with a number. The meter takes ≈ 28,000 usage events a second at peak and loses none (N1). The check adds at most 5 ms at p99 (N2), and a meter outage refuses no model request (N3). Parallel requests end a window at most one request past its limit, and no reply is cut off (N4).
Credits are debited exactly once, and window totals stay within 0.1% live and are exact within an hour (N5). Per-request usage is kept 13 months (N6). Any read shows a finished request within 5 s, every CLI response carries the percent, and an open web page refreshes every 15 s (N7).
Consistency differs per operation. A credit debit must be exact, because it is money. A limit may be off by 0.1% until the hourly pass corrects it, and a display may be 5 s stale. All of these numbers are assumptions: an interviewer gives them, or you propose them.
YOU ASK
THE ANSWERIT DECIDES
How many users and requests?
the answer 2 million daily active users, 400 model requests each a day, and a peak of 3 times the average.
it decides A sharded stream and spread-out counters (chapters 02 and 07).
How much may the check add to a request?
the answer 5 ms at p99.
it decides The counters the check reads live in memory (chapter 08).
If the meter is down, do model requests stop?
the answer No. Requests keep running, and their usage is counted when the meter is back.
it decides The check fails open, and usage waits in a durable stream (chapters 07 and 08).
May a user go past a limit?
the answer By at most one request. A reply is never cut off mid-stream.
it decides Each request reserves an estimate before it starts (chapter 08).
How exact must the numbers be?
the answer Credits exact to the micro-dollar. Limits within 0.1%, made exact within an hour.
it decides Idempotent debits and an hourly reconciliation (chapter 09).
How fresh must usage be?
the answer Any read shows a finished request within 5 s. The CLI updates on every response.
it decides The stream’s lag budget, and usage in response headers (chapter 10).
How long is per-request usage kept?
the answer 13 months.
it decides An archive in object storage, not the live store (chapter 09).
COMMON QUESTIONS
What if the interviewer won’t give numbers?
Propose them out loud (2 million daily users, 400 requests each) and design for those. The method matters more than the values.
Why count money and not tokens?
A cache-read token costs 0.1× the price of a fresh input token, and an output token 5× it. A limit in plain tokens would treat all three alike.
Is the session window rolling?
No. In this design a session opens at the first request and lasts 5 hours. Chapter 05 shows why opening it needs one writer.
WHAT YOU NOW KNOW
The meter answers two callers: the model gateway before and after each request, and people reading /usage.
It must count every request (F1), stop the next one at a limit (F2), and show percent and reset time (F3).
The hard numbers: ≈ 28,000 events a second, a 5 ms check, exact credits, and usage visible within 5 s.
DEEP DIVE: qualities set aside, how real plans meter, a rate limit is not a usage limit, interviewer follow-ups
Qualities set aside
Multi-Region: model serving runs in one Region, so the meter does too. If serving ever runs in two, each Region meters its own traffic. Read-your-writes on the usage page: a page a few seconds stale is fine, because the check, not the page, enforces the limit.
Security sits in place rather than in a chapter: usage events carry counts and ids, never prompt text, and each service can reach only its own store. Cost gets no chapter either, because the finale prices the design as a monthly bill. Geography changes nothing: users anywhere reach the same gateway.
How real plans meter
Anthropic’s help center, read on 21 November 2026, describes Claude’s paid plans with two windows: a five-hour session that starts at the first message, and a weekly limit that resets at a fixed time per account, shared between claude.ai and Claude Code. Past a limit, usage credits pay at API rates.
OpenAI’s Codex usage-limits page, read the same day, describes the same two windows and says “you need allowance remaining in both”. Claude Code shows its numbers two ways: /usage prints each window’s percent and reset time, and a custom status line receives the five-hour and seven-day percentages in its rate_limits fields. Tidewell copies the shape: percents and reset times outside, budgets inside.
A rate limit is not a usage limit
An API rate limit caps requests or tokens per minute. Anthropic’s API docs describe theirs as a token bucket: capacity refills continuously up to a maximum, so a burst drains it and a pause refills it. A plan’s usage limit is a window with a reset time: what you used stays used until the window closes.
The two have different jobs. A rate limit protects the servers from a burst over seconds; a usage limit shares capacity over hours and days. Tidewell would run both as separate systems, and this tutorial designs only the second.
INTERVIEWER FOLLOW-UPS
THE ANSWER
“Why not just count requests?”
→ Requests vary a hundredfold in cost. An agent turn reading 46,000 cached tokens and a one-line question are not the same request.
“Who decides the limits?”
→ Product does. The meter needs them only as numbers per plan, so they can change without a deploy.
“Should limits be shown as dollars?”
→ Show percent. Keeping budgets internal lets the provider tune them without breaking a promise.
02
Estimates: the numbers that change the design
An estimate turns chapter 01’s assumptions into a rate or a size you can hold against one machine. 2 million daily active users sending 400 model requests each decide nothing until they become numbers like that. Three of them change this design: usage events a second, counter writes a second, and bytes kept. Round each as you would on a whiteboard, and end it with what it decides. Which of these numbers is too big for one machine?
STEP 01 / 03
EVERY MODEL REQUEST IS A USAGE EVENT
Fig 2.1Follow the lines down: the accent line is the peak the design is sized for.
Every model request a Tidewell user sends ends in one usage event, so the meter’s rate is the model fleet’s rate. 2 million daily active users times 400 requests each is 800 million requests a day. Divide by the 86,400 seconds in a day and you get ≈ 9,300 usage events a second on average.
Traffic is not flat. Developers work office hours on two continents, and the busiest hour runs at about 3 times the daily average, the peak factor. So the meter must take ≈ 28,000 usage events a second at peak, the number every later chapter sizes for.
STEP 02 / 03
TWO COUNTERS PER EVENT DOUBLE THE WRITES
Fig 2.2Every bar shares one scale; the accent bar is the demand that outgrows a single table’s default.
Each usage event adds to 2 counters, the user’s session and the user’s week, so the writes double to ≈ 56,000 counter writes a second at peak. The check reads the same two counters, but reads can be served from copies and writes can’t, so the writes set the size.
Hold the demand against what one unit takes. One stream shard accepts 1,000 records a second, and one database partition takes 1,000 writes a second. One table’s default quota is 40,000 writes a second, below the demand. One cache node handles about 100,000 simple operations a second, which fits.
So no single shard, partition or table holds the counters. The events must spread across many shards, and the counters across many partitions or into memory; chapters 07 and 08 do both.
STEP 03 / 03
STORAGE IS SMALL, THE WRITE RATE ISN’T
Fig 2.3Two totals: the archive at the top and, in accent, everything the check needs in memory.
The archive keeps every usage event for 13 months, the retention requirement. At 400 bytes an event as JSON, 800 million events a day is 320 GB a day. Stored as Parquet, a compressed column format, it shrinks about 10× to 32 GB a day, ≈ 12.7 TB over 13 months.
The live state is what the check reads on every request: each subscriber’s plan, two open windows and credit balance, about 300 bytes. For 6 million subscribers that is ≈ 1.8 GB. The archive fits in one bucket of object storage, and the live state fits in one cache node’s memory.
So storage decides nothing in this design. The write rate from step 02 does, and it shapes chapters 07 and 08.
COMMON QUESTIONS
Why a peak factor of 3?
Developers work office hours across two continents, so the busiest hour runs at about 3 times the daily average.
Why 400 requests a day?
An agent turn is many model requests: each tool call is a round trip to the model, so a working day adds up fast.
Why estimate storage if it decides nothing?
To show that it decides nothing. ≈ 12.7 TB in one bucket and ≈ 1.8 GB in memory move the conversation on to the write rate.
WHAT YOU NOW KNOW
≈ 28,000 usage events a second at peak, one for every model request the fleet serves.
≈ 56,000 counter writes a second: too many for one partition or one table’s default quota.
≈ 12.7 TB kept and ≈ 1.8 GB live: storage is not the problem; the write rate is.
DEEP DIVE: rounding on a whiteboard, estimates left out, bytes or records, interviewer follow-ups
Rounding on a whiteboard
The exact peak is 27,778 usage events a second, and no decision depends on its last three digits. Round to one or two significant figures, write ≈ to say you did, and the arithmetic stays in your head: ≈ 9,300 times 3 is ≈ 28,000.
Counts of machines are the exception. 27,778 records a second at 1,000 a shard needs 28 shards, rounded up, because one fewer would fall short. Round demand to the nearest; round capacity up.
Estimates left out
Two estimates decide nothing here. Reads: an open usage page polls every 15 s, so 10,000 open pages make ≈ 670 reads a second, small beside the writes. Bandwidth: ≈ 28,000 events of 400 bytes each is ≈ 11.1 MB a second at peak, which one network link carries easily.
Bytes or records
A stream shard limits records and bytes: 1,000 records or 1 MB a second, whichever comes first. At 400 bytes a record, the count binds first: ≈ 28,000 records a second needs 28 shards by records and only 12 by bytes. Small events are limited by count, large ones by size.
INTERVIEWER FOLLOW-UPS
THE ANSWER
“What if usage doubles?”
→ The stream adds shards and the cache adds a shard. Only the table’s default quota needs a request to raise it.
“What dominates the cost?”
→ Writes, never storage. Chapter 08 moves the counters into memory, priced by the node-hour rather than per write.
“How many stream shards would you provision?”
→ 28 by records at peak. With 1.4× headroom for bursts and uneven keys, 39.
03
Entities and API: five records, three calls
The requirements name three operations: ask before a model request, report after it, and read usage. Each becomes one endpoint, a method and URL a caller sends to, and together they touch a few core records: account, plan, window, usage event and credit debit. The choice that matters is timing, because the model gateway sits on the user’s path. Which call must the gateway wait for, and which must it never wait for?
STEP 01 / 04
THREE CALLS CROSS THE BOUNDARY
Fig 3.1Each chip on an accent arrow is one endpoint.
Each functional requirement becomes one endpoint. The model gateway calls two of them: POST /internal/v1/check before inference (F2) and POST /internal/v1/usage-events after the reply ends (F1). People call the third, GET /v1/usage, from the CLI and the web app (F3).
The two internal calls never leave the private network. The public one comes in through a front door that checks who is calling before the meter sees the request; step 03 picks it.
STEP 02 / 04
FIVE RECORDS HOLD THE WHOLE STATE
Fig 3.2The accent row is the record every request adds to twice.
Five records hold everything the meter knows. An account carries the user’s plan, weekly reset anchor and credit balance, and a plan carries the limits. A window is one session or one week of one user, with what it has used. A usage event is one finished model request, and a credit debit is one request paid from credits.
Money is a whole number of micro-dollars, in the fields ending in _u: the example request costs 57,250 µ$. A window is identified by user, kind and start time, so the next session is a new record rather than a counter reset to zero. Reservations arrive in chapter 08 and are not a core record.
STEP 03 / 04
THE API, ONE CALL PER REQUIREMENT
Fig 3.3Each row is one endpoint; the accent row is the one the gateway waits for.
Both internal calls carry request_id, which the gateway mints once per model request. It is the idempotency key: a retried check or record with the same id takes effect once. GET /v1/usage names no user at all. The caller comes from the token the request carries, never from the URL or the body.
Inside the network, the gateway reaches the meter through ECS Service Connect, which gives the meter’s containers a name and spreads calls across them from the client side, with no load balancer in between. The runner-up, an internal Application Load Balancer, adds a network hop and an hourly bill for nothing the check needs.
GET /v1/usage comes in through an API Gateway HTTP API, AWS’s managed front door, which checks the user’s JWT before the request reaches the meter. A REST API would add caching and usage plans the meter doesn’t need, at $3.50 per million requests against $1.00.
THE THREE ENDPOINTS, WITH THE EXAMPLE VALUES
# before inference: model gateway → meter, private, the gateway waitsPOST /internal/v1/check{"request_id": "req_01JD7Q", "user_id": "usr_8c1f", "model": "tide-large", "est_u": 59750}→ 200 {"allow": true, "source": "plan", "session_start": "2026-11-19T09:12:00Z", "week_start": "2026-11-16T08:00:00Z", "five_hour_pct": 41, "seven_day_pct": 33}# after the reply ends: fire and forget, retried from the gateway’s spoolPOST /internal/v1/usage-events[{"request_id": "req_01JD7Q", "user_id": "usr_8c1f", "model": "tide-large", "input": 600, "cache_write": 1400, "cache_read": 46000, "output": 900, "session_start": "2026-11-19T09:12:00Z", "week_start": "2026-11-16T08:00:00Z"}]→ 202# the user’s view: CLI or web → API Gateway (JWT) → meterGET /v1/usageAuthorization: Bearer eyJhbGciOiJSUzI1NiJ9…→ 200 {"five_hour": {"used_pct": 42, "resets_at": "2026-11-19T14:12:00Z"}, …}
STEP 04 / 04
THE CHECK WAITS, THE RECORD DOESN’T
Fig 3.4Follow 1 to 8: the bracket marks what the user waits for, and the accent record falls outside it.
The check is synchronous: the gateway sends it and waits for “allow” before inference starts, because the answer decides whether the request runs at all. It must answer within 5 ms at p99. If it can’t, the gateway lets the request through: the check fails open, and chapter 08 makes that the rule.
The record is asynchronous. By the time the gateway knows the token counts, the reply has already reached the user, so nothing is left to hold. The gateway posts the usage event, the meter answers 202 Accepted before applying it, and if the call fails, the gateway keeps the event in a local spool on disk and retries it.
So a slow or failing meter can delay a request by at most the check’s timeout, and can never take back a reply.
COMMON QUESTIONS
Why not let the gateway write the counters itself?
Pricing and plan rules would then live in every gateway. One meter service owns them and can change them without a gateway deploy.
Why 202 and not 200?
202 says “accepted, not yet applied”: the usage reaches the counters a second or two later.
WHAT YOU NOW KNOW
Three endpoints: check before inference, usage-events after it, and GET /v1/usage for people.
The gateway waits only for the check; the record is fire and forget, retried from a spool.
Every internal call carries the request id, the key that makes a retry take effect once.
DEEP DIVE: the front door on AWS, why not WebSockets, history and cursors, interviewer follow-ups
The front door on AWS
API Gateway offers two kinds of API for this job. A REST API adds response caching, usage plans and API keys, at $3.50 per million requests; an HTTP API has JWT authorizers and private integrations built in, at $1.00. Those are us-east-1 list prices on 21 November 2026. The meter needs only the token check and a route to a private service, so the HTTP API wins on price and loses nothing.
The quota that bites is shared: 10,000 requests a second per account and Region, across every API in the account, with a burst of 5,000. The first can be raised and the burst can’t. At ≈ 670 usage reads a second the meter uses a small slice, but other teams’ APIs draw from the same pool.
Why not WebSockets
A WebSocket would let the meter push each new percent to an open page instead of waiting to be asked. It costs a connection held open per page and a service to fan updates out to them. The requirements ask only that a page be 15 s fresh, so this chapter keeps plain HTTP; chapter 10 prices push against polling.
History and cursors
There is no list endpoint, because no requirement asks for past windows. A history endpoint would take a time range and return pages, with a cursor (an opaque token naming where the last page ended) rather than a page number, so new windows arriving between calls don’t shift the pages.
INTERVIEWER FOLLOW-UPS
THE ANSWER
“What if the check times out?”
→ Allow the request after 5 ms and count it afterwards. Chapter 08 makes failing open the rule, with a cap per user.
“Where does the idempotency key come from?”
→ The gateway mints it once per model request and reuses it on every retry of the check and the record.
“Can a client call the check endpoint itself?”
→ No. Only the gateway’s private network reaches it; otherwise a client could skip the check or ask about someone else’s allowance.
04
Counting a request: from tokens to micro-dollars
The gateway now reports each finished request to the meter as a usage event, and nothing turns that report into usage yet. Something has to take the token counts, price them at the model’s list price, and add the cost to the user’s session and week. Before you price anything, you need to know who counted the tokens and whether to trust the count. Where do token counts come from, and how does a request become usage?
STEP 01 / 04
THE MAP SO FAR
Fig 4.1The accent boxes are this chapter’s; the meter writes each report to the table.
The meter service is the program that owns pricing and windows. It takes each usage event from the gateway, prices it, and adds the cost to the user’s current windows. It runs as containers on AWS Fargate, which runs containers without servers for you to manage, in private subnets across three Availability Zones.
The usage table holds the totals: one item per user and window, with the cost used so far in micro-dollars and the tokens by type. It is a DynamoDB table, a key-value store that AWS runs and bills per request. Chapter 05 adds the check that reads it.
STEP 02 / 04
THE MODEL SERVER COUNTS, THE GATEWAY REPORTS
Fig 4.2Follow 1 to 6; the counts are born at hops 2 and 3 and leave at hop 4.
The model server is the only part that sees every token. During prefill it reads the whole prompt in one pass and knows the input counts: fresh tokens, tokens written to the prompt cache, and tokens read from it. During decode it writes one output token per step, thinking included, and counts each one.
The server streams the reply back through the gateway, and its last stream event carries the usage so far, which by then is the total. For req_01JD7Q that is 600 fresh, 1,400 cache-write, 46,000 cache-read and 900 output tokens. The gateway copies the counts into a usage event and sends it after the reply has ended.
The meter prices the event, stamps it with the price version it used, and adds the cost to two items in the usage table: the session and the week.
Fig 4.3Read across each line, tokens × rate; the accent row is what the request costs.
A model’s price depends on the token type. tide-large lists $5 per million input tokens and $25 per million output tokens. A write to the prompt cache costs 1.25× the input price and a read from it 0.1×, the usual shape of real list prices.
A cache read on tide-large costs 0.5 µ$, so the meter keeps rates in hundredths of a micro-dollar, where every rate is a whole number: 500, 625, 50 and 2,500. req_01JD7Q costs 5,725,000 hundredths, rounded up once at the end to 57,250 µ$.
The 46,000 cache-read tokens are 94% of the tokens and 40% of the cost; the 900 output tokens are 39%. A limit counted in tokens would weigh them alike.
INTEGER PRICING, ROUNDED UP ONCE
# List prices per token, in hundredths of a micro-dollar, so every rate is an integerRATES = { "tide-large": {"input": 500, "cache_write": 625, "cache_read": 50, "output": 2500}, "tide-small": {"input": 100, "cache_write": 125, "cache_read": 10, "output": 500},}def cost_u(event: dict) -> int: """The request cost in whole micro-dollars, rounded up once at the end.""" rates = RATES[event["model"]] hundredths = sum(event[kind] * rate for kind, rate in rates.items()) return -(-hundredths // 100) # ceiling division, integers onlycost_u({"model": "tide-large", "input": 600, "cache_write": 1400, "cache_read": 46000, "output": 900})# 57250 micro-dollars, about 5.7 cents
STEP 04 / 04
ONE ITEM PER USER AND WINDOW
Fig 4.4Each row is an access pattern and the key that serves it; the accent row runs per request.
Three access patterns use the usage table: add a request to its windows, read the current windows, and list past ones. Each is a lookup by user and window, so a key-value table fits. The partition key is the user, USER#usr_8c1f; the sort key is the window’s kind and start time, such as 5H#2026-11-19T09:12:00Z.
Adding a request is one UpdateItem per window with ADD used_u :57250, which DynamoDB applies to the stored value in place. The runner-up, Aurora PostgreSQL, brings SQL this design never uses and one writer for every update.
This design breaks at scale. Two writes per event make ≈ 56,000 writes a second at peak, above one table’s default quota of 40,000. Chapters 07 and 08 fix that.
THE USAGE TABLE, KEYED BY USER AND WINDOW
UsageTable: Type: AWS::DynamoDB::Table Properties: TableName: usage BillingMode: PAY_PER_REQUEST AttributeDefinitions: - AttributeName: pk # USER#usr_8c1f AttributeType: S - AttributeName: sk # 5H#2026-11-19T09:12:00Z or WK#2026-11-16T08:00:00Z AttributeType: S KeySchema: - AttributeName: pk KeyType: HASH - AttributeName: sk KeyType: RANGE PointInTimeRecoverySpecification: PointInTimeRecoveryEnabled: true SSESpecification: SSEEnabled: true SSEType: KMS
COMMON QUESTIONS
Why not count tokens in the gateway with a tokenizer?
The model server already counts them exactly, cache hits included. A second count from another tokenizer would disagree with it, and with the bill.
Why count cost and not a token total?
So a cache read and an output token weigh what they cost: 50 against 2,500 hundredths of a micro-dollar on tide-large. The token totals are still stored for display.
WHAT YOU NOW KNOW
The model server counts tokens at prefill and decode, and its last stream event carries them to the gateway.
The meter prices four token types: 57,250 µ$ for req_01JD7Q, 39% of it for output.
Each request adds its cost to two items in the usage table: the user’s session and week.
DEEP DIVE: what a usage object holds, thinking and server tools, integers for money, interrupted streams, interviewer follow-ups
What a usage object holds
Anthropic’s Messages API returns a usage object with input_tokens, output_tokens, cache_creation_input_tokens and cache_read_input_tokens, plus a split of cache writes by how long they live and a count of server tool uses. When the reply streams, each message_delta event carries the usage so far, so the last one holds the totals (docs read 21 November 2026).
OpenAI’s Chat Completions usage puts cached input in prompt_tokens_details.cached_tokens and thinking in completion_tokens_details.reasoning_tokens. A stream carries usage only when the request sets stream_options.include_usage, and the docs warn: “If the stream is interrupted, you may not receive the final usage chunk.”
Thinking and server tools
Thinking tokens are billed as output tokens at the output price, even when the reply shows only a summary of them, so the meter prices them with output. Server tools, such as a web search the model runs, are billed per use on top of tokens. A usage event carries them as one more counted type with its own rate.
Integers for money
A cache-read token on tide-large costs 0.5 µ$, and floating-point sums drift as millions of them add up. Hundredths of a micro-dollar make every rate a whole number. The cost is rounded up once per request, at the end, so the meter never undercounts and never errs by a whole micro-dollar on any request.
Interrupted streams
If the model server stops mid-stream, the gateway records the counts from the last event it saw, marked partial, so output already generated is paid for. If the client disconnects, the gateway either reads the stream to its end or cancels it, and records what was produced either way. A request refused before inference records nothing.
INTERVIEWER FOLLOW-UPS
THE ANSWER
“What if prices change in the middle of a window?”
→ Each event carries the price version it was priced with, so a window can mix versions safely. Old events are never re-priced.
“Do failed requests count?”
→ Count what the model computed. A request refused before inference costs nothing; one that fails mid-stream pays for the tokens it produced.
“Why round up and not to the nearest micro-dollar?”
→ Rounding up once per request means the meter never undercounts, and the error stays under one micro-dollar a request.
05
Checking the limits: windows, refusals and usage credits
The usage table now holds every user’s session and week. Nothing yet stops a user at the limit: the gateway sends the request to the model whether the session stands at 41% or far past its limit. To refuse in time, the meter has to answer before inference starts, and every parallel request has to agree on when the session began. Who opens a session, and how does the check say no in time?
STEP 01 / 04
THE MAP SO FAR
Fig 5.1The check edge and the accounts table are new; the rest is chapter 04’s.
The accounts table holds what the check needs to know about each user: the plan and its limits, the weekly reset time, whether usage credits are on, and the credit balance. It is a second DynamoDB table. The meter keeps each account in memory for 5 minutes, so most checks never read it.
The check is a new call to the same meter service. The gateway asks before inference, with the request’s estimated cost, and waits up to 5 ms for the answer. The meter compares the user’s current windows with the plan’s limits.
STEP 02 / 04
THE CHECK OPENS THE SESSION
Fig 5.2The accent slice is the open session; the tick is req_01JD7Q at 11:40.
A session window opens at the first request after the previous one ended and lasts five hours. The check opens it: when a request finds no open session, the meter writes the start time with a conditional write that succeeds only if no session is open. Two parallel first requests both try, one write wins, and the other reads the winner’s start.
For usr_8c1f, the session opened at 09:12 on Thursday and resets at 14:12. The week needs no write to open: it starts at the account’s fixed time, Mon 08:00, and resets at Mon 08:00 seven days later.
The check returns both start times, and the gateway stamps them on the usage event. The meter then adds the cost to the window the check saw, even when the event arrives after that window has closed.
ONE SESSION PER USER, OPENED BY A CONDITIONAL WRITE
from datetime import datetime, timedeltaimport boto3from botocore.exceptions import ClientErrortable = boto3.resource("dynamodb").Table("usage")def session_start(user_id: str, now: datetime) -> str: """The open session start; the first request after the last session ended opens one.""" key = {"pk": "USER#" + user_id, "sk": "5H#CURRENT"} try: return table.update_item( Key=key, UpdateExpression="SET started = :now", # Two parallel first requests: only one of them passes this condition ConditionExpression="attribute_not_exists(started) OR started <= :ended", ExpressionAttributeValues={ ":now": now.isoformat(), ":ended": (now - timedelta(hours=5)).isoformat(), }, ReturnValues="ALL_NEW", )["Attributes"]["started"] except ClientError as e: if e.response["Error"]["Code"] != "ConditionalCheckFailedException": raise return table.get_item(Key=key, ConsistentRead=True)["Item"]["started"]
STEP 03 / 04
ONE ROUND TRIP DECIDES
Fig 5.3Follow 1 to 5 for an allowed request; the red leg is what a refused one gets.
The check is one call. The gateway sends the user, the model and an estimate of the cost, here 59,750 µ$. The meter takes the plan’s limits from its cached account and reads the two current windows from the usage table, then compares what is used against each limit.
For req_01JD7Q the session stands at 41% and the week at 33%, so the meter answers allow, with both percents, and inference starts. When either window is full, the answer is a refusal, and the gateway returns 429 Too Many Requests with that window’s reset time.
This check compares only what is already used; the estimate rides along unused for now. Chapter 08 uses it to hold room for requests that are still running.
STEP 04 / 04
PAST THE LIMIT, CREDITS PAY
Fig 5.4The accent row is the ledger entry; the balance on the account item moves in the same write.
With usage credits on, a request past a limit still runs and is paid from the user’s prepaid balance at list price. The check answers allow with source: credits instead of a refusal, once it has made sure the balance covers the estimate.
The meter writes the debit when the usage event arrives, as two items in one transaction: the account’s balance falls by the request’s cost, and a ledger row keyed by the request id records the debit. Had req_01JD7Q run past the limit, $12.00 would become $11.94, and the row DEBIT#req_01JD7Q would hold 57,250 µ$.
A team organisation adds a monthly cap on credits: one more item, updated in the same transaction and shared by every seat.
COMMON QUESTIONS
Why does the session start at the first request?
It matches how people work: five hours from when you sit down, not from midnight. A user who starts at 09:12 has the whole allowance until 14:12.
Why a fixed weekly reset per account?
Resets spread across accounts instead of landing at one instant, and each user can plan around a known time.
What does the large-model cap add?
One more counter in the week item (tide-large at most 50% of the week) and one more comparison in the check.
WHAT YOU NOW KNOW
The check opens a session with a conditional write, and the gateway stamps its start on every event in it.
A request needs room in both windows; a refusal is a 429 with the reset time, 14:12 for the example session.
Past the limit, usage credits pay at list price: a debit and its ledger row in one transaction, $12.00 to $11.94.
DEEP DIVE: fixed, rolling and sliding windows, what real products show at a limit, organisation caps, interviewer follow-ups
Fixed, rolling and sliding windows
A fixed window resets at set times, like the week here. A session window starts at first use, so each user’s resets fall at different times. A sliding window counts the last five hours at every instant, which is smoother and needs per-request history or an estimate.
Cloudflare’s rate limiter estimates a sliding window from two fixed-window counters, weighting the previous one by how much of it still overlaps. Across 400 million requests it measured a 0.003% error (Cloudflare blog, 7 June 2017).
What real products show at a limit
Claude plans use the same two windows: a five-hour session that starts at your first message and a weekly limit that resets at a fixed time per account. Claude Code warns before the limit, “You’ve used 85% of your session limit · resets 3:45pm”, and at it, “You’ve hit your weekly limit · resets Mon 12:00am” (Claude Code docs, read 21 November 2026).
Anthropic’s API keeps spend caps apart from rate limits. A request past an organisation’s spend cap fails with enforced_spend_limit_reached and no retry-after header, because waiting does not help; a rate-limit refusal carries one.
Organisation caps
Team seats keep their own windows, so one busy seat never spends another’s allowance. The organisation’s monthly credit cap is the one key every seat shares: each paid request updates it in the debit’s transaction. One item takes at most 1,000 writes a second, so a large team’s debits are batched, up to 100 items per transaction.
INTERVIEWER FOLLOW-UPS
THE ANSWER
“What happens when a user upgrades mid-session?”
→ The cached account expires within 5 minutes and the new limit applies. What was used stays, so the percent drops.
“Can the check run before the prompt is tokenized?”
→ It needs only an estimate. The gateway already knows the prompt’s size and its cached prefix, which is close enough to admit or refuse.
“What does the check do when the accounts table is slow?”
→ It uses the cached account. Chapter 08 adds the rule for a slow meter: after 5 ms, the gateway lets the request through.
06
Showing the usage: one read, two percents
The meter now counts each request and refuses the next one at a limit, yet the only way a user learns their usage is a refused request. The /usage command in the CLI and the usage page on the web need the numbers the check reads, for the signed-in user only, without exposing the limits behind them. What does GET /v1/usage return, and why a percent instead of a budget?
STEP 01 / 03
THE MAP SO FAR
Fig 6.1The upper lane is the read path; only API Gateway is new.
API Gateway is the public front door for usage reads. It checks the caller’s JWT, the signed token from their Tidewell sign-in, and forwards GET /v1/usage over a private link to the meter service, which already holds the plans and reads the windows.
It is an API Gateway HTTP API, at $1.00 per million requests against $3.50 for a REST API, whose caching and usage plans this read doesn’t need.
STEP 02 / 03
ONE READ, TWO WINDOWS
Fig 6.2Follow 1 to 6; at hop 4 the meter divides each window’s use by its limit.
When you type /usage, the CLI sends GET /v1/usage with your JWT. API Gateway checks its signature and expiry and passes on the user id from the token’s sub claim; the request never names a user. The meter reads that user’s current session and week items.
It divides each window’s used cost by the plan’s limit and rounds down: 2,107,250 µ$ of 5,000,000 µ$ is 42%, and 13,357,250 µ$ of 40,000,000 µ$ is 33%. The reply gives each percent with its reset time, 14:12 for the session and Mon 08:00 for the week, plus token totals and an as_of time.
The CLI draws a bar per window, and the web page renders the same JSON.
Fig 6.3The accent row is what users see first; the muted rows stay inside the meter.
The response carries percent used, reset time and token totals by type, and never the limit in dollars. The session’s tokens, 19,000 fresh, 56,000 cache-write, 1,640,000 cache-read and 33,690 output, tell a user what used the allowance without telling them what it cost.
Keeping the budget inside lets the product change a plan’s limits without a client release, and the meter stays the only place that knows what a token costs. Dollars appear only for usage credits, where the user is spending money.
COMMON QUESTIONS
Why not let the web page read DynamoDB directly?
The percent logic would then live in two places, and the browser would need database credentials. API Gateway and the meter keep both inside.
What about usage from other devices?
The meter counts by user, so the CLI, the IDE plugins and the web share one total. A count kept inside one CLI would miss the others.
WHAT YOU NOW KNOW
GET /v1/usage goes through API Gateway, which takes the user from the JWT, never from the request.
The meter computes each percent from the windows the check reads: 42% of the session, 33% of the week.
Users get percent, reset time and token totals; the budget in micro-dollars stays inside the meter.
DEEP DIVE: what real clients show, caching the read, usage history, interviewer follow-ups
What real clients show
Claude Code’s /usage draws its bars from a server usage endpoint. When that endpoint throttles it, the command shows the last values it got, kept for up to 60 minutes, under “Showing last-known usage”. Its status line script receives rate_limits.five_hour and rate_limits.seven_day, each with used_percentage and resets_at (Claude Code docs, read 21 November 2026).
The same docs say the session cost breakdown Claude Code prints is counted per machine. A total kept by the provider is the one that covers every device.
Caching the read
A /usage read is one key-value read per window, far below the meter’s write load. The limit that binds first is API Gateway’s account quota of 10,000 requests a second, shared by every API in the account and Region. A few seconds of caching per user in the meter absorbs a burst, and as_of tells the client how old the answer is.
Usage history
A history view lists past windows: one Query on the user’s partition with begins_with(sk, "5H#2026-11") returns that month’s sessions in order, because the start time is in the sort key. It needs from, to and a cursor (DynamoDB’s LastEvaluatedKey), which is why the API leaves it out for now.
INTERVIEWER FOLLOW-UPS
THE ANSWER
“What if every user runs /usage at once?”
→ Each is one key-value read; API Gateway’s account quota of 10,000 requests a second binds first, so raise it or cache each answer for a few seconds.
“Why not show dollars?”
→ Subscribers bought a plan, not a meter. Dollars appear only for usage credits, where the user spends money.
“How does the web page authenticate?”
→ With a JWT from the user’s web sign-in. API Gateway checks it, so the browser never holds a database credential.
07
Deep dive: a stream that loses nothing
Chapter 01 asked for 28,000 usage events a second at peak, with no finished request’s usage lost (N1). The meter from chapter 04 writes two items per event inside the record call: about 56,000 writes a second, above the 40,000 one table takes by default. When a write times out, the meter can’t tell slow from lost, so it either retries and may count twice, or drops the usage. How do usage events reach the counters with none lost when the database slows down?
STEP 01 / 05
THE MAP SO FAR
Fig 7.1The bottom band is new: the meter appends to the stream, and the aggregator applies batches to the usage table.
A stream now sits between the meter and the usage table: an ordered, durable log that keeps every record it accepts. The record call appends the usage event to the stream and returns. Nothing in it waits for the table any more.
An aggregator reads the stream in batches, sums each user’s usage and adds it to the window items in the usage table. The stream is Kinesis Data Streams with 39 shards; the aggregator is a Lambda function that Kinesis triggers with each batch. The check still reads the table, so only the writes move.
STEP 02 / 05
A SLOW WRITE BECOMES A LOST ONE
Fig 7.2The red leg is the timeout: the meter can’t tell a lost write from a slow one.
Each usage event adds to two items, the session and the week, so 28,000 events a second become about 56,000 writes a second. One on-demand table accepts 40,000 writes a second by default, a quota you can raise, and a new table takes only 4,000 at once. Past that, DynamoDB throttles: it refuses the write, and the SDK retries until the call times out.
A timeout leaves the meter guessing, because the write may have landed or not. Retry it, and an ADD that did land counts twice. Give up, and a finished request is never counted. Either way the meter breaks N1, and a slow table has turned into wrong usage.
STEP 03 / 05
THE OPTIONS
Fig 7.3Same rows for each option; the accent column is the stream.
There are three ways to get an event from the meter to the counters. Write the table inside the record call, as now. Put each event on an SQS standard queue that a worker drains. Or append it to a Kinesis stream, where records stay for a day and every reader sees all of them.
Only the stream meets N1 and leaves room for chapter 09: it survives a slow table, it can replay 24 hours, and the archive can read the same records as the aggregator. It costs about $768 a month against $2,920 for SQS, which bills every send, receive and delete. The table writes, about $30,417 a month, stay in every column until chapter 08.
The price of a stream is at-least-once delivery: a retried append or a retried batch delivers a record twice, and chapter 09 deals with that.
STEP 04 / 05
39 SHARDS, KEYED BY USER
Fig 7.4Each row is one setting and the number behind it; the accent row sizes the stream.
A shard is one slice of the stream with its own write limit: 1,000 records a second. 28,000 events a second need 28 shards; with 1.4× headroom for spikes and uneven keys, 39.
The partition key is user_id. Kinesis hashes it to pick a shard, so one user’s events land on one shard in order, and 2 million daily users spread evenly across all 39.
Lambda reads each shard and calls the aggregator with up to 10,000 records, or whatever arrived within the 1 s batch window. The aggregator sums the batch per user and window and applies one ADD to each window item in the usage table.
THE STREAM AND THE AGGREGATOR’S TRIGGER (CLOUDFORMATION)
Fig 7.5Three failures, three red legs; each ends somewhere that keeps the event.
If a shard throttles an append, or the stream can’t be reached, the meter keeps the event in a spool on the task’s disk and retries. Until the meter answers 202, the gateway keeps its own copy, so a meter task that dies takes nothing with it.
If the aggregator throws on a batch, Lambda splits the batch in half and retries each half until the bad record is alone. After the last retry, it sends that record’s shard and sequence number to a failure destination, an SQS queue, and moves on.
If the usage table is down, or the aggregator stops, the records wait in the stream for up to 24 hours. The percent users see stops moving until the aggregator catches up, and no usage is lost.
COMMON QUESTIONS
Why provisioned and not on-demand?
The load follows the working day and is predictable. On-demand would cost about $1,587 a month against $768, starts a new stream at 4 MB a second of writes, and doesn’t isolate a hot key.
Does batching cut the table writes?
Not much. An active user sends a request about every 54 s, so a 1 s batch rarely holds two of theirs. What changes is that a throttled write now waits in the stream instead of being lost.
WHAT YOU NOW KNOW
The meter appends each event to a 39-shard stream and returns; nothing waits on the table.
The aggregator applies events in 1 s batches, in order per user, as one ADD per window item.
A slow table now delays usage instead of losing it; the stream holds 24 hours.
DEEP DIVE: on-demand capacity, packing small records, order and PutRecords, when Flink wins, how Stripe splits fast and slow
On-demand capacity
Kinesis on-demand mode sizes the stream for you. It absorbs up to double the highest write rate it has seen recently, and a faster jump can throttle until it adapts. A new on-demand stream starts at 4 MB a second of writes. On-demand Advantage, a newer option, lets you pay for warm throughput ahead of a launch. For a daily load you can predict, provisioned shards stay cheaper.
Packing small records
A shard takes 1,000 records or 1 MB a second, whichever runs out first. At 400 bytes an event, the record count binds first: 28 shards by records against 12 by bytes. The Kinesis Producer Library can pack many small events into one Kinesis record, so a shard carries more events a second, and the aggregator unpacks them with the deaggregation module for Lambda.
Order and PutRecords
Kinesis keeps records in order within a shard, by sequence number. A batched PutRecords call doesn’t promise order: if part of a batch fails and is retried, a user’s later event can land before an earlier one. So each event carries its end time, and the aggregator orders a user’s events by it. A sum doesn’t care about order; opening a session does.
When Flink wins
Managed Service for Apache Flink keeps windowed state with checkpoints and gives exactly-once results inside the job. It wins for joins, sliding windows and rollups across many keys. Here the aggregator only adds numbers, and Flink’s DynamoDB sink still writes at least once, so a Lambda function that sums each batch is enough, and you pay per batch instead of for capacity that runs all day.
How Stripe splits fast and slow
Stripe described its usage-based billing in “How we built it: usage-based billing” (28 January 2025). Meter events take a fast path that sums them in memory over windows of seconds, for alerts and limits, and a slow path that writes them to a ledger every few minutes, for invoices. This design makes the same split: live counters for checks, and the stream and archive for the record.
INTERVIEWER FOLLOW-UPS
THE ANSWER
“What if one user floods a single shard?”
→ One shard takes 1,000 records a second, far past any real user. The gateway rate-limits that user before the traffic reaches the meter.
“Why not write to the table from the gateway?”
→ Every gateway would need table credentials and the pricing rules. One meter service owns both and changes them without a gateway deploy.
“How long can the usage table be down?”
→ The stream holds 24 hours, and usage lands when the table is back. Chapter 09’s archive keeps a copy beyond that.
“What about a very large team?”
→ Each seat is keyed by its own user, so the team spreads across shards. The one shared key is the organisation’s credit cap.
08
Deep dive: a check in a millisecond
Chapter 01 set three targets for the check: at most 5 ms at p99 (N2), no model request refused because the meter is down (N3), and at most one request past a limit (N4). The check still reads the usage table on every request, the counter writes cost about $30,417 a month, and requests running in parallel can’t see each other. Where should the counters live so the check is fast, cheap and safe when it fails?
STEP 01 / 05
THE MAP SO FAR
Fig 8.1The accent box sits between the check and the aggregator; the usage table now holds hourly totals.
Live counters now hold each user’s open windows, and the requests still running, in memory beside the meter. The check reads and updates them in one round trip, and the aggregator adds each batch’s usage to them instead of to the table.
They run on ElastiCache for Valkey, an in-memory key-value store, as a cluster of 2 shards. Each shard is a primary and 2 replicas, one node per Availability Zone.
The usage table keeps history: the aggregator writes each open window’s totals to it every hour and when the window closes.
STEP 02 / 05
8 PARALLEL REQUESTS AT 96%
Fig 8.2The dashed line is the limit; the accent part is where 8 unchecked requests take the session.
The check reads used and compares it with the limit, but usage arrives only after a reply ends. In the illustrative example, later in usr_8c1f’s session, used stands at 4,800,000 µ$ (96%), and the agent starts 8 subagent requests at once, each costing 65,000 µ$.
All 8 checks read 96%, all 8 pass, and the session ends at 5,320,000 µ$: 106.4% of the limit, several requests past it where N4 allows one. And every check is still a table read on the p99 path, before the model starts.
STEP 03 / 05
THE OPTIONS
Fig 8.3Same rows for each option; the accent column is the cluster in memory.
There are three places for the counters. Keep them in DynamoDB and make every check a conditional write. Move them to a Valkey cluster, where one script reads and reserves in under a millisecond. Or keep them in each meter task’s memory, which works only if every request from a user reaches the same task.
Valkey wins on the rows that matter here: under a millisecond against a 5 ms budget, an atomic script per check, and about $767 a month for 6 nodes against $30,417 of table writes. In-task counters cost nothing, and lose every count on a redeploy.
STEP 04 / 05
RESERVE, THEN SETTLE
Fig 8.4Follow the 4 admitted requests to the settle; the red legs are the 4 refused.
One Lua script runs on the live counters for every check, atomically on the user’s keys. If used plus reserved is below the limit, it adds the request’s estimate to reserved and records a reservation under the request id; otherwise it refuses. When the reply ends, the aggregator settles: it adds the actual cost to used and deletes the reservation.
In the burst, the first request sees 4,800,000 µ$, the next 4,860,000 µ$, then 4,920,000 µ$ and 4,980,000 µ$: all below 5,000,000 µ$, so 4 reserve 60,000 µ$ each and run. The fifth sees 5,040,000 µ$ and gets a 429, like every request after it.
Settling adds 65,000 µ$ for each of the 4, and the session ends at 5,060,000 µ$: 101.2%, one request past the limit, as N4 allows.
CHECK AND RESERVE IN ONE SCRIPT (VALKEY, PYTHON)
import valkeycounters = valkey.Valkey(host="counters.meter.internal", port=6379, ssl=True)# Atomic on one user: both keys carry the {usr_8c1f} hash tag, so they share a slotcheck_and_reserve = counters.register_script("""local used = tonumber(redis.call('HGET', KEYS[1], 's_used') or '0')local held = tonumber(redis.call('HGET', KEYS[1], 'reserved') or '0')if used + held >= tonumber(ARGV[1]) then return 0 -- refuse: 429 with the reset timeendredis.call('HSET', KEYS[2], ARGV[3], ARGV[2]) -- the reservation, under the request idredis.call('HINCRBY', KEYS[1], 'reserved', ARGV[2])return 1""")allowed = check_and_reserve( keys=["u:{usr_8c1f}", "r:{usr_8c1f}"], args=[5000000, 60000, "req_01JD7Q"], # session limit, estimate, request id)
STEP 05 / 05
WHEN THE CACHE FAILS, REQUESTS STILL RUN
Fig 8.5The red box is the lost primary; the accent path is a request the meter still admits.
When a shard’s primary fails, ElastiCache promotes one of its replicas within seconds. Replication is asynchronous, so settles the old primary took in its last second may be missing on the new one; the hourly reconciliation in chapter 09 puts them back.
When the cluster can’t answer within 5 ms, the meter fails open: it allows the request. A guard in each meter task caps that at 20 requests a minute per user, so an outage can’t turn into unlimited use.
The record path never touches the cache, so usage keeps flowing to the stream. The aggregator’s settles fail and retry, the records wait in the stream, and they land once the cluster answers again.
COMMON QUESTIONS
Why not ElastiCache Serverless?
It fits, and it runs Lua scripts. A node cluster wins here because the load is steady and known, and Serverless bills each script by the compute it uses.
What does a request refused at 96% see?
A 429 with the session’s reset time. /usage still shows 96%, because a reservation is not usage.
What estimate does a reservation use?
The prompt’s price plus 1,000 expected output tokens: 59,750 µ$ for req_01JD7Q.
WHAT YOU NOW KNOW
Live counters move to a Valkey cluster in memory: under a millisecond per check, about $767 a month.
Each check reserves an estimate, so 8 parallel requests at 96% end at 101.2%, not 106.4%.
If the cache fails, the meter fails open under a guard of 20 requests a minute per user and counts the usage later.
DEEP DIVE: the script and its keys, hash slots and hot keys, durable writes, how LiteLLM, OpenAI and Cloudflare chose
The script and its keys
Each user has two hashes: u:{usr_8c1f} for the window totals and r:{usr_8c1f} for open reservations by request id. The part in braces is a hash tag: the cluster hashes only that part to pick a slot, so both keys sit on one shard and one script can change them atomically. Without the tag, the script fails with a CROSSSLOT error.
Hash slots and hot keys
A user’s keys live on one shard, so a heavy user loads one primary. At 28,000 checks a second spread over millions of users, no slot runs hot, and one node handles on the order of 100,000 simple operations a second. ElastiCache Serverless would fit too: it runs EVAL scripts but not FUNCTION, caps the throughput of each slot, and bills scripts by the compute they use.
Durable writes
ElastiCache copies writes to replicas asynchronously, so a failover can drop the last writes the primary took. ElastiCache also offers a durable mode for Valkey that confirms a write only once it is safely stored, so a failover loses none, at the cost of slower writes. This design doesn’t need it: the stream still holds every event, and the hourly reconciliation restores what a failover drops.
How LiteLLM, OpenAI and Cloudflare chose
LiteLLM’s proxy reserves a request’s worst-case cost against a budget before it runs and reconciles the real cost afterwards, the same reserve-then-settle. Without Redis it falls back to each instance’s memory, the in-task option (source read 21 November 2026).
OpenAI’s “Beyond rate limits” (13 February 2026) checks limits synchronously, debits usage asynchronously and refunds an overshoot afterwards. Cloudflare’s AI Gateway documents its spend limits as eventually consistent, so a burst can pass one (read 21 November 2026).
INTERVIEWER FOLLOW-UPS
THE ANSWER
“Why not cut the reply at the limit?”
→ A half-written code edit is worse than a small overshoot. N4 allows one request past the limit, and a reply is never cut.
“What if the cache loses everything?”
→ Rebuild the open windows from the usage table’s hourly totals plus the last hour from the archive, failing open under the guard meanwhile.
“How do you stop abuse while failing open?”
→ Each meter task admits at most 20 requests a minute per user while the cluster is unreachable, so the total stays bounded.
09
Deep dive: counted once, proven hourly
Chapter 01 set two targets for accuracy (N5): credits debited exactly once, and window totals within 0.1% live and exact within an hour. It also asked for 13 months of per-request records (N6). The design so far delivers at least once at every hop: the meter retries appends, Lambda retries whole batches, and nothing keeps a request once the stream’s 24 hours pass. If every hop can deliver an event twice, how is each request still counted once?
STEP 01 / 05
THE MAP SO FAR
Fig 9.1The new path runs along the bottom: stream, Firehose, archive, reconciler, and corrections back into the stream.
A copy of every usage event now lands in an archive that keeps it for 13 months. Amazon Data Firehose reads each record from the usage stream and writes it to the usage archive, an S3 bucket of Parquet files in hourly folders, ≈ 12.7 TB when full.
Once an hour, the reconciler, a Lambda function that runs Athena queries over the archive, recomputes every window. This is reconciliation. Its corrections go back into the stream as new events, so the aggregator stays the only writer of the live counters and of the usage table’s totals.
STEP 02 / 05
THREE WAYS ONE REQUEST COUNTS TWICE
Fig 9.2Each red leg is a retry that delivers req_01JD7Q a second time.
Every hop in this design retries, and a retry can repeat work that already happened. The meter’s append can time out after Kinesis stored the record; the retry stores req_01JD7Q again under a new sequence number.
The aggregator can fail partway through a batch it has half applied. Lambda then retries the whole batch, and the live counters get 57,250 µ$ for the same request twice. Firehose retries a delivery to S3 that timed out, so the archive can hold the same row twice.
The DynamoDB docs say it plainly: atomic counters are not idempotent. Each of these hops delivers at least once.
STEP 03 / 05
THE OPTIONS
Fig 9.3Same rows for each option; the accent column is exact for money and close, then exact, for limits.
The first option makes every event exact with a DynamoDB transaction: a dedupe item keyed by the request id plus both counter updates, so a duplicate fails as a whole. That is three writes per event, and a transaction bills each twice: ≈ $91,250 a month.
The second is exactly-once stream processing, such as Managed Service for Apache Flink. Its checkpoints keep the job’s own state exact, but its DynamoDB sink still writes at least once, so a restart can repeat writes to the counters.
The third is exact only where money moves: one ledger row per credit debit, plus the hourly reconciliation for limits. Debits and queries cost ≈ $1,835 a month ($1,825 and $10). A limit can be off by up to 0.1% for up to an hour.
STEP 04 / 05
ONE KEY PER REQUEST, THREE PLACES
Fig 9.4One row per place a duplicate stops; the accent row is where money moves.
A paid request’s debit is one transaction, written by the meter at record time. It puts the ledger row DEBIT#req_01JD7Q as a conditional write that fails if the row exists, and takes 57,250 µ$ off the balance, $12.00 to $11.94. A retried record fails the condition, so the whole transaction is cancelled.
The other two places catch what reaches the counters. A settle in the live counters applies only while the request’s reservation exists, then deletes it, so a retried batch adds nothing the second time. In the archive, the reconciler’s query keeps one row per request id.
A CREDIT DEBIT A RETRY CAN’T REPEAT
import boto3ddb = boto3.client("dynamodb")user = {"S": "ACCT#usr_8c1f"}ddb.transact_write_items(TransactItems=[ {"Put": { # the ledger row: one per request id, kept for good "TableName": "accounts", "Item": {"pk": user, "sk": {"S": "DEBIT#req_01JD7Q"}, "amount_u": {"N": "57250"}}, "ConditionExpression": "attribute_not_exists(sk)", # a retry fails here }}, {"Update": { # the balance, in the same transaction "TableName": "accounts", "Key": {"pk": user, "sk": {"S": "ACCOUNT"}}, "UpdateExpression": "ADD credit_balance_u :neg", "ConditionExpression": "credit_balance_u >= :cost", "ExpressionAttributeValues": {":neg": {"N": "-57250"}, ":cost": {"N": "57250"}}, }},])# A second call with the same request id raises TransactionCanceledException
STEP 05 / 05
THE HOURLY PROOF
Fig 9.5Follow 1 to 5: sum the archive, read the counters, send corrections, apply them.
Every hour, a schedule starts the reconciler. Its Athena query reads the last two hours of the archive, keeps one row per request id and sums cost per user, window and hour. It then reads the live counters’ bucket for each of those windows and hours.
Where a sum and a bucket differ, the reconciler appends a correction event carrying the difference to the stream, and the aggregator applies it like any other event. The target is drift under 0.1%; each run scans ≈ 2.7 GB of Parquet, ≈ $0.01 at $5.00 per TB.
When one of the new parts fails, nothing a user sees changes at once. If Firehose or S3 falls behind, the stream holds 24 hours and Firehose catches up. If the reconciler or Athena fails, limits drift a little longer, credits stay exact, and the next run widens its window to the hours it missed.
THE HOURLY QUERY: ONE ROW PER REQUEST ID
-- One row per request id, then exact totals per user, session and hourWITH once AS ( SELECT *, ROW_NUMBER() OVER (PARTITION BY request_id ORDER BY ended_at) AS copy FROM usage_archive WHERE dt = '2026-11-19' AND hr IN ('10', '11') -- the last two hours)SELECT user_id, session_start, hr, SUM(cost_u) AS cost_u, COUNT(*) AS requestsFROM onceWHERE copy = 1GROUP BY user_id, session_start, hr;
COMMON QUESTIONS
Why not dedupe every event in a table?
A dedupe item and both counters in one transaction cost ≈ $91,250 a month, to catch duplicates that only retries create. The ledger row does that job where money moves.
What if a duplicate pushes a user over a limit?
Their next request can be refused slightly before the true limit, until the next reconciliation subtracts the duplicate and the percent drops back.
Is the archive itself exact?
Not as stored. Firehose delivers at least once too, so a row can appear twice; the query keeps one row per request id.
WHAT YOU NOW KNOW
Each request id is checked three times: at the credit ledger, at the reservation and in the archive query.
Credits are exact at once; limits stay within 0.1% live and exact after the hourly pass.
The archive keeps 13 months of per-request rows in S3, and every correction re-enters the stream as a new event.
DEEP DIVE: dedupe windows in real systems, late events and closed periods, a hot organisation’s debits, sequence-number watermarks, three linked datasets
Dedupe windows in real systems
Billing platforms dedupe on an id the sender supplies, for a set window. Stripe treats a meter event’s identifier as unique for at least 24 hours. Metronome dedupes for 34 days and OpenMeter for 32. Lago keys its ClickHouse rows on the transaction id plus the timestamp. All four were read on 21 November 2026.
Tidewell’s ledger row has no window: it stops a duplicate debit for as long as the row is kept. The archive query has none either, because it compares every row in the hours it reads.
Late events and closed periods
An event can arrive after its hour was reconciled, when a gateway’s spool drains late or a shard falls behind. Stripe keeps an invoice open for a grace period, 1 hour by default and up to 72, so late usage still lands on it. Orb never overwrites usage; it amends it with new events.
Tidewell follows the same rule. A correction is a new event in the stream, never an edit to a counter, so the archive keeps the whole history and a bill can be recomputed as it stood at any hour.
A hot organisation’s debits
Team seats share one organisation credit cap, so every paid request in a large team updates one item, and one partition takes 1,000 writes a second. The meter can buffer a second of one organisation’s debits and write them as one transaction of up to 100 items.
If that transaction is cancelled, its CancellationReasons list says which condition failed for each item, so the retry leaves out the ledger rows that already exist.
Sequence-number watermarks
Another way to make the counters exact at once: store, with each user’s counters, the highest stream sequence number already applied, and skip any record at or below it. It works because a user’s events stay in order on one shard.
It costs a conditional write on every apply, and it misses one case. The meter’s retried append gets a new sequence number, so it looks like a new event; only an id from the request itself catches that.
Three linked datasets
OpenAI’s “Beyond rate limits” post (13 February 2026) describes three linked datasets: usage events, monetization events and balance updates, each with a stable idempotency key, so replaying any of them changes nothing. Tidewell’s archive rows, ledger rows and credit balance play the same three roles.
INTERVIEWER FOLLOW-UPS
THE ANSWER
“How do you prove a customer’s bill?”
→ Re-run the archive query for their account and period. Every row carries a request id and the price version it was priced with.
“What if the reconciler fails for a day?”
→ Limits drift a little more and credits are unaffected. The next run widens its window to the hours it missed.
“Why S3 and not the table for 13 months?”
→ ≈ 12.7 TB costs ≈ $291 a month in S3 at $0.023 per GB-month, against $0.25 per GB-month in DynamoDB.
“Why not make the limits exact at once too?”
→ Sequence-number watermarks would, at a conditional write per apply. Pay for that only if 0.1% of drift for an hour is too much.
10
Deep dive: from token to percent in seconds
Chapter 01 asked for freshness (N7): any read shows a finished request within 5 s, every CLI response carries the percent, and an open web page refreshes at least every 15 s. The counters settle a second or two after a reply ends, but the CLI asks only when you type /usage, and an open web page shows what it loaded. How does a new percent reach a terminal and a browser without a million polls?
STEP 01 / 04
THE MAP SO FAR
Fig 10.1The two accent arrows are the only change: usage headers on every reply, and a 15 s poll.
The model gateway now copies the check’s answer, both percents, into headers on every model response it sends. No new part is needed. Sending data on a message that is already on its way is called piggybacking.
The web page polls GET /v1/usage every 15 s while it is visible, through the same API Gateway route the CLI’s /usage command uses.
STEP 02 / 04
TWO WAITS OF A SECOND MAKE UP THE LAG
Fig 10.2One bar per hop on one scale; the dashed line is the 5 s target.
A lag budget adds up every hop between a reply ending and the counters showing it. The meter accepts the record in ≈ 5 ms and flushes its buffer to the stream at least every 100 ms.
Lambda polls each shard every 1 s, and the aggregator’s batch window can wait up to 1.0 s more. The settle script then takes ≈ 2 ms. In all that is ≈ 2,107 ms, about 2.1 s, against the 5 s target.
STEP 03 / 04
THE OPTIONS
Fig 10.3Same rows for each option; the accent column adds no part.
Polling everywhere is the simplest: the page and every open CLI call GET /v1/usage every 15 s. The 10,000 open pages alone make ≈ 670 requests a second, ≈ $1,607 a month. The CLIs add ≈ 33,000 more, past the account’s 10,000-a-second API Gateway quota.
Push sends each new percent over a WebSocket. With AppSync Events, connections and every publish and delivery are billed, ≈ $2,663 a month. The meter also needs a presence registry, so that it publishes only for users someone is watching, and one API takes 10,000 inbound events a second.
The third reuses what exists: the gateway puts the percent on every model response, and only the web page polls, for ≈ $1,607 a month and no new part.
STEP 04 / 04
THE PERCENT RIDES ON EVERY RESPONSE
Fig 10.4Each row is a header and where the CLI shows it; the accent row is the session percent.
The gateway copies the check’s answer into four headers on the model response: tide-usage-5h-pct, tide-usage-5h-reset, tide-usage-7d-pct and tide-usage-7d-reset. The CLI’s status line reads them after each response, so it is never more than one response behind.
The values are the check’s answer, from before this request ran: req_01JD7Q’s response says 41%, and the next one says 42%. The web page polls every 15 s while it is visible, and once more as soon as it regains focus.
If a response carries no headers, the status line keeps the last percent it saw. If a poll fails, the page keeps its last answer and tries again at the next one.
When many people watch one number change, such as a shared team budget on a wall screen. For one user watching their own percent, a 15 s poll is enough.
Why not poll every second?
10,000 open pages would make 10,000 requests a second, the whole account’s API Gateway quota of 10,000.
What does the CLI show before its first response?
It calls GET /v1/usage once when it starts. After that, the headers on each response keep it current.
WHAT YOU NOW KNOW
A finished request reaches the counters in ≈ 2.1 s, inside the 5 s target.
Every model response carries the session and week percent in headers, so the CLI is at most one response behind.
The web page polls every 15 s while visible; push would add a presence registry and cost ≈ $2,663 against ≈ $1,607.
DEEP DIVE: Claude Code’s status line, threshold warnings, WebSocket limits, when push pays
Claude Code’s status line
Claude Code runs a status line command of your choice and passes it JSON on standard input. Since version 2.1.80, that JSON includes rate_limits.five_hour and rate_limits.seven_day, each with used_percentage and resets_at in epoch seconds (docs read on 21 November 2026).
The fields appear only after the first API response, the command is debounced at 300 ms, and a window is dropped once its reset time has passed. Tidewell’s headers play the same role for the tide status line.
Threshold warnings
The check is the first place to see a user cross a threshold. At 85% of a window, the gateway adds a warning header and the CLI prints one line, once per window, in the style of Claude Code’s “You’ve used 85% of your session limit · resets 3:45pm”.
The warning costs nothing extra: the check computed the percent anyway, and the header rides on a response that was already going out.
WebSocket limits
API Gateway WebSocket APIs close a connection after 2 hours, or after 10 minutes idle, and accept 500 new connections a second per account. AppSync Events, launched on 30 October 2024, takes 10,000 inbound events a second per API.
Either way, every open page holds a connection, reconnects after a drop and must fetch the current percent again, because updates sent while it was away are gone.
When push pays
Push wins when many viewers watch one number that changes often: a shared team budget on a wall screen, or an admin page for a large organisation. One publish per organisation per second then reaches every viewer, where polling would multiply the reads by the number of screens.
INTERVIEWER FOLLOW-UPS
THE ANSWER
“Admin usage APIs lag about 5 minutes. Why is yours 2.1 s?”
→ Admin and cost reports read the archive, which may lag. The percent comes from the live counters the check uses.
“How do you warn at 85%?”
→ The check sees it first. The headers carry a warning field, and the CLI prints it once per window.
“What if a user has the page open on three devices?”
→ Each device polls on its own: three key-value reads every 15 s, small beside the model traffic.
The whole picture
Every part from the ten chapters, in place. The numbers on the map match the rows of the reference table under it.
Find a number on the map, then its row below: what the part does, the requirement it serves, and what users see when it fails.
1tide CLI (client)
sends model requests, shows /usage and the status line
serves
F3
chosen over
—
when it fails
the last percent it saw
2web app (client)
the usage page, polled every 15 s while visible
serves
F3, N7
chosen over
—
when it fails
a stale page until the next poll
3model gateway (existing)
asks the meter before inference, reports after it, adds usage headers
serves
F1, F2, N7
chosen over
—
when it fails
no model requests (outside this design)
4model servers (existing)
run the model and count tokens at prefill and decode
serves
F1
chosen over
—
when it fails
no replies (outside this design)
5usage API front door (API Gateway HTTP API)
checks the JWT and routes GET /v1/usage to the meter
the reply is already sent, and a meter outage must not fail it
transport
chose
Kinesis, 39 shards
over
SQS; on-demand Kinesis
because
replay and a second reader, ≈ $768 against ≈ $2,920 a month
live counters
chose
a Valkey cluster
over
DynamoDB counters
because
under a millisecond, and ≈ $767 against ≈ $30,417 a month of writes
admission
chose
reserve, then settle
over
check, then count
because
101.2% against 106.4% for 8 parallel requests
failure
chose
fail open with a guard
over
fail closed
because
inference is the product; a cache failover must not become an outage
exactly once
chose
a ledger row per debit, hourly reconciliation
over
a transaction per event
because
only money must be exact at once; ≈ $1,835 against ≈ $91,250 a month
freshness
chose
usage headers and a poll every 15 s
over
push over WebSockets
because
no presence registry, and ≈ $1,607 against ≈ $2,663 a month
decision
chose
over
because
record path
asynchronous, with a spool
synchronous
the reply is already sent, and a meter outage must not fail it
transport
Kinesis, 39 shards
SQS; on-demand Kinesis
replay and a second reader, ≈ $768 against ≈ $2,920 a month
live counters
a Valkey cluster
DynamoDB counters
under a millisecond, and ≈ $767 against ≈ $30,417 a month of writes
admission
reserve, then settle
check, then count
101.2% against 106.4% for 8 parallel requests
failure
fail open with a guard
fail closed
inference is the product; a cache failover must not become an outage
exactly once
a ledger row per debit, hourly reconciliation
a transaction per event
only money must be exact at once; ≈ $1,835 against ≈ $91,250 a month
freshness
usage headers and a poll every 15 s
push over WebSockets
no presence registry, and ≈ $1,607 against ≈ $2,663 a month
The monthly bill
The highlighted line, the credit ledger at $1,825, is the largest of $8,018.
The credit ledger costs the most because every paid request writes a ledger row and a balance in one transaction, and DynamoDB bills each item of a transaction twice, at $0.625 per million write units. The second line is API Gateway, which charges $1.00 per million reads for the open usage pages’ polls. Internet data out for those replies is left off: its price lives in the EC2 price list, and a reply of about a kilobyte adds little next to the total.
What you’d do next
Watch three numbers: the reconciler’s drift, the stream’s iterator age (how far the aggregator is behind), and the share of checks that failed open.
The next bottleneck past today’s numbers is one very large organisation spending credits: batch its debits into one transaction per second.
With more time, revisit accuracy: per-window sequence watermarks would make limits exact at once, not within an hour.
The design in 45 minutes
Requirements5 min
say
count, refuse, show; ≈ 28,000 events a second, a 5 ms check, credits exact, 5 s fresh
draw
the outside view: clients, model gateway, three calls in
Estimates4 min
say
≈ 28,000 events and ≈ 56,000 counter writes a second; 12.7 TB archived, 1.8 GB live
draw
the capacity bars
Entities and API5 min
say
five records; check synchronous, record asynchronous with a request id; the user from the JWT
draw
the outside view with one chip per endpoint
High-level design12 min
say
the model server counts, the meter prices, two windows, a conditional session open, credits in one transaction
draw
meter service and tables, then the check, then API Gateway
Deep dives15 min
say
a stream; Valkey with reserve-then-settle and fail open; exactly once where money moves; headers
draw
the stream band, the cache, the archive loop
Wrap-up4 min
say
the requirements check, the trade-offs, the bill, what you’d watch
draw
the whole picture, numbered
phase
min
what you say
what you draw
Requirements
5
count, refuse, show; ≈ 28,000 events a second, a 5 ms check, credits exact, 5 s fresh
the outside view: clients, model gateway, three calls in
Estimates
4
≈ 28,000 events and ≈ 56,000 counter writes a second; 12.7 TB archived, 1.8 GB live
the capacity bars
Entities and API
5
five records; check synchronous, record asynchronous with a request id; the user from the JWT
the outside view with one chip per endpoint
High-level design
12
the model server counts, the meter prices, two windows, a conditional session open, credits in one transaction
meter service and tables, then the check, then API Gateway
Deep dives
15
a stream; Valkey with reserve-then-settle and fail open; exactly once where money moves; headers
the stream band, the cache, the archive loop
Wrap-up
4
the requirements check, the trade-offs, the bill, what you’d watch
the whole picture, numbered
Ask before you draw: requirements first, as questions with answers.
Three functional requirements and one line of what’s out of scope.
Every quality gets a number, and consistency is stated per operation.
An estimate ends in a decision, or it’s left out.
The check is synchronous, the record asynchronous, and every write carries a request id.
Simple first, then break it with a number and fix what broke.
Each deep dive: options, a choice, its cost, and how it fails.
Close the loop: check every requirement against the finished design.
THREE THINGS TO KEEP
A usage meter answers twice for every request: may it start, and what did it cost.
Check in memory before the request, count from a stream after it, and prove the count from an archive.
Retries deliver some events twice. Make money exact at once, and let limits be corrected within the hour.
NEXT TIME /USAGE SAYS 42%, YOU KNOW WHAT’S BEHIND IT