A high-throughput lookup system with sub-5 ms p95 latency

Prologue Architectural drivers

Designing a system starts with establishing its boundaries: the limits the environment imposes on it. They act before any design decision. They restrict which options are available, so no other input has a more direct effect on the design.

Five architectural drivers shape this system: two fixed, in order of precedence, then three flexible, traded against each other down to floors.

Latency - a fixed 5-millisecond timeout, end to end, at p95, and the first driver in precedence: a lookup past the timeout returns nothing, so time over budget converts directly into yield loss. Every component on the request path spends a share of it, and that share shapes each component’s design. The tolerated misses grow with traffic, so the percentile caps scale: a p95 hold carries a given scale, p99 a multiple of it, and full scale sits at p99 and beyond.
Deadline - fixed by a sales season: client budgets and integrations open cyclically, and a missed window forfeits the full cycle. The binding date is integration readiness, earlier milestones tolerate partial capability, so the date moves within the cycle only. The deadline bounds the design space to what could be built in time.
Yield - flexible, down to a floor. The match rate: the share of looked-up keys that find a record. Each record returned is worth revenue, revenue per record rises as carried traffic grows, and any rate above zero is worth more than a missed deadline, so yield is the driver most often traded where the fixed drivers require it: a yield trade is recoverable, improvements can land later. The floor: a sustained low rate weakens the case for the system itself and for further investment in it.
Scale - flexible. The share of full traffic the system carries. Uncarried traffic is lost revenue, and the loss is recoverable: capacity can be added later. Each step up in rate raises cost and complexity more than the last, and some limits appear only near the full rate, so the design adapts as the carried rate grows.
Operating cost - flexible, down to a viability threshold set by the system’s revenue position; a cheaper system carrying the same load keeps a wider margin. At this scale cost tracks depth of knowledge of the system: every layer holds values whose effect repeats at the full rate. Negative margin is sustained only through compromises on the other flexible drivers, and the compromises compound.

The fixed drivers define the boundary of the design space; the flexible ones select a design inside it, traded against each other: added cost raises yield where the spend is justified, and a yield or scale improvement raises the revenue that carries the cost. Each trade is settled against the environment metrics current at the decision (margin, revenue per record, carried share), and changed metrics reopen settled trades: the drivers persist, the decisions are re-settled as the metrics move.

Chapter 1 Latency budget

latency percentiles · circuit breaker

Implementation

The budget is 5 milliseconds, end to end, at p95. The path has three layers:

the network leg: datacenter to datacenter, everything on the wire before the API’s work begins.
the API business logic: parse the payload, dispatch the lookups, build the response.
data retrieval: pull the records from the data sources, across the billions of records.

The two app-side layers are budgeted at about 1ms each, 2ms together. The network takes the remainder.

The call runs between two services in two different datacenters: the client’s service, on-prem, and the API, in AWS.

The network leg: one round trip between the client's datacenter and the API in AWS.

Rationale

The 5ms is fixed and the system is designed to fit inside it. The split follows two criteria: where the time is most at risk, and the degree of control over each layer. The two app-side stages are under direct control, and 1ms is the smallest budget enforceable there: timeouts and cancellations are configured in integer milliseconds, in the runtime and in the client libraries. Each stage is assigned that minimum. The network leg carries the most unknowns, the most complexity, and the least control, so it takes the remaining 3ms.

The 5ms budget across the round trip: ~3ms network, ~2ms logic and data fetch.

Colocating both services in the same datacenter would simplify the network infrastructure between them. The deadline set the architecture: an edge location inside the client’s on-prem datacenter means hardware procurement, vendor contracts, and third-party maintenance, and that work exceeded the time available. A cloud vendor already operates the hardware: compute, networking, and load balancing provision through API calls, in minutes. The deployment runs in AWS.

Latency targets are set as percentiles. p95 means 95% of requests finish under the limit; the slowest 5% may exceed it.

p50, p95, and p99.

A p99 target permits only 1% of requests to exceed the limit, so 4% that a p95 target tolerates must also be mitigated. That slow tail comes from rare events like GC pauses, queuing, contention, and virtualization, which is why p99 is much harder to hold.

The 5ms limit is enforced from the client side, by a circuit breaker: a guard that cuts traffic to a dependency while the dependency is slow. The client’s service runs one on its calls to the API: once more than 5% of responses exceed 5ms, the breaker opens and cuts the request rate; as latency returns under the limit, the rate recovers. A brief open is acceptable; a long one is yield lost for its duration.

The client-side circuit breaker: it opens once more than 5% of responses run over 5ms.

Chapter 2 Network validation

speed of light in fiber · network congestion

Implementation

The network leg has a 3ms budget, the most uncertainty, and the least control. It is also the one layer measurable before the system exists: an empty endpoint is enough.

The validation deployment is that empty endpoint: it receives a request, deserializes it, and returns a mocked response. No lookups, no data sources, so the round trip measures the network alone. A network that misses its 3ms is a hard limit: no other layer can win the time back.

A deployment passes when it holds under 3ms at its metro’s share of the full request rate.

The client operates in several metro areas, and each metro’s path differs in distance, peering, and routing. Each metro gets its own deployment, validated independently, serving the client’s service in that metro.

Some metros - each served by its own AWS region.
The others - each served by an AWS local zone, where no region sat close enough.

Rationale

The two services run in different datacenters within the same metro area. By default, the traffic crosses the public internet. A single round trip passes through intermediaries such as:

the client’s NIC and OS network stack
the on-prem datacenter’s LAN switches
the on-prem edge router and firewall
the datacenter’s uplink to its ISP
a local internet exchange or peering point, handing traffic off toward AWS
AWS’s edge into the region
the load balancer

The chain is representative: a real path crosses more hops, can cross several networks, and can be asymmetric. Every hop adds time, and most of them sit outside direct control. The app’s stack and the load balancer are the exception. The rest belongs to the client, the ISPs, and the public routing in between.

The round trip across the public internet: most of the hops sit outside direct control.

AWS Direct Connect is a dedicated private network link into AWS, trading the public internet’s variable routing for a stable, predictable path. Provisioning one takes procurement and installation time beyond what the deadline allowed.

The one variable that could be shaped was the fiber the signal travels, set by deployment location. Light moves through glass at about two-thirds of its speed in vacuum, roughly 5 microseconds per kilometer one way, 10 round trip: about 100 km of fiber per millisecond of round-trip latency. The figure is a floor: hardware tuning doesn’t move it, and every kilometer of fiber spends a fixed, calculable budget. The physical route can run 20–40% longer than the map distance, so the floor holds even in-metro. Deploying in the same metro as the client keeps it short.

Fiber length and latency: the speed-of-light floor grows with distance, ≈ 1 ms per 100 km round trip.

Latency above the fiber floor comes from congestion: other traffic sharing the public internet. Congestion changes with the time of day. The floor can be calculated from distance; congestion can only be measured. The measurements have to include busy hours: a path that meets the budget at a quiet hour can break it at a busy one.

End-to-end p95 is measured from the client side, the one point covering the full round trip. At each metro’s full share of the request rate, it held under 3ms, the designed target.

The measured figure differs per deployment: each metro’s network path, load balancer type, and hardware generation differ.

Chapter 3 Load balancer

load balancing · layer 4 vs layer 7 · Availability Zones and regions · TLS handshake · TLS termination · DNS resolution and caching

Implementation

Inside AWS, the first thing a request reaches is the load balancer.

The load balancer differs by deployment:

AWS regions - Network Load Balancer (NLB).
AWS local zones - Application Load Balancer (ALB). NLB is not available in local zones.

Two settings applied everywhere:

Single Availability Zone (AZ) - the load balancer is deployed in one AZ only: the one the app runs in.
TLS 1.3 - listeners created through the API, CLI, or infrastructure-as-code default to a policy negotiating at most TLS 1.2; upgraded to TLS 1.3 here. Termination stays at the load balancer.

Rationale

The app runs as many replicas; the load balancer is the one address in front of them, distributing incoming requests across the targets. Its time falls in the network leg; the app’s budget begins once traffic reaches the handler.

The load balancer fronts the replicas and picks one per request; the app budget starts at the handler.

NLB works at layer 4: it forwards TCP and inspects nothing above it. ALB works at layer 7: it terminates HTTP, parses the request, applies routing rules.

The layer also sets how load spreads. NLB distributes per connection: each TCP connection is routed to one target for its life, so with long-lived connections a few heavy clients stay on the same replicas and load can sit uneven across them. ALB distributes per request: it terminates the connection and the routing decision is per request, so load spreads across replicas regardless of how the connections are distributed.

NLB (layer 4): each connection is pinned to one replica for its life.

ALB (layer 7): the connection terminates at the load balancer; each request is routed on its own.

Operating cost selects NLB. Both charge the same base rate; the variable cost follows the work the load balancer does, and forwarding packets is the smallest unit of work. The routing rules and header inspection ALB adds go unused on this path. At this request rate, the NLB deployments cost roughly 30% less.

An Availability Zone is a single datacenter. An AWS region is a group of such datacenters, close together but physically separate. A load balancer is enabled per zone, and in each enabled zone it gets its own address. Which address a client connects to is decided by DNS, and that decision sits outside the system’s control. The app runs in one zone. A client connecting to an address in a different zone has its traffic arrive at that datacenter first, then cross to the app’s datacenter, adding time and a per-gigabyte transfer charge in each direction. Enabling the load balancer in the app’s zone only means every client connects to the app’s datacenter directly.

Single-zone operation also selects NLB in the regions: a region ALB requires at least two zones, a rule AWS enforces for the load balancer’s own availability (when one zone fails, the nodes in the other continue to route), so a region ALB would pay the cross-zone cost. In the local zones the rule doesn’t apply, and one zone is enough for the ALB too.

The tradeoff is redundancy: a zone failure takes the deployment down. The outage stays in its metro, and failover to another metro exceeds the 5ms budget: lookups in that metro stop until the zone recovers, a yield loss for the duration. Multi-AZ would remove the risk: the app and the full data store run in two or more zones, each serving its own traffic; each zone is a separate datacenter with its own network path, so each passes the network validation on its own. The price is the duplicated store, an operating cost the business scale might not justify. The local zones vary by metro, one zone or two: with one there is no decision to make; with two, the same tradeoff applies.

TLS 1.3 is about round trips. A TLS 1.2 handshake takes two before any data moves; TLS 1.3 takes one, and the per-round-trip cost on this path is significant. Connections are persistent, but they still open, and each open pays the handshake. Terminating at the load balancer keeps the cryptographic work out of the app: the handshake ends at the first hop inside AWS, and the app receives plain traffic over the internal network. None of the app’s budget is spent on it.

TLS handshake and connection reuse.

DNS sits outside the budget by design. DNS maps a name to the IP addresses behind it. Before a client can connect, it resolves the load balancer’s name to an address. An uncached resolution is a chain of round trips that alone exceeds the 5ms budget. The resolved address is cached on the client host, in the application runtime or the OS resolver. The cache is checked before any query leaves the machine, and the entry holds until the record’s TTL expires. A resolution runs at connection open, at most once per TTL window, and every request inside the window reuses the cached address, so the cost amortizes over the window. The per-request cost is negligible.

The client resolves the load balancer's name once, caches the address, then reuses it for every request until the TTL expires.

Chapter 4 Cluster

Kubernetes · Karpenter · EC2 purchase models · autoscaling · virtualization latency variance · burstable compute · CFS throttling · file descriptors

Implementation

The app runs in an EKS cluster. On the compute side:

Node size fixed - one of the standard EC2 sizes, tailored to the workload; multiple families (compute, general, memory) and architectures (ARM, AMD), all flex instance types (variable CPU by design) filtered out. Karpenter provisions the cheapest available across the set.
One pod per node - no two pods share a node.
CPU limits removed from the pod spec.
Autoscaling at 50% CPU - when average CPU crosses it, new pods are added.
File descriptor limit (nofile) - ~1 million per container, set by the node image at the container runtime level.

Rationale

The app runs on Kubernetes. EKS is AWS’s managed version of it. Kubernetes runs applications as pods: a pod is one running copy of the app, the replica the load balancer targets. Pods are placed on nodes, and a node is an EC2 machine. Kubernetes decides which pod lands on which node. The decision accepts constraints: which nodes a pod may land on, and which pods may share one.

Karpenter creates the nodes themselves: when pods need a machine, it provisions one, and the configured constraints tell it which machines it’s allowed to create.

EC2 sells the same machine under several purchase models. On-demand: the list price, no commitment. Reserved: a one- or three-year commitment on defined capacity, at a discount that grows with the term, suited to the steady baseline of a load. Spot: spare capacity at 70–90% below the on-demand price, which AWS reclaims on a two-minute notice when it needs the capacity back. Spot is preferred; when spot capacity is unavailable, on-demand is purchased.

The scheduler queues unschedulable pods; Karpenter watches the queue and provisions nodes to place them.

Autoscaling is how the deployment follows the traffic. The metrics server collects CPU usage from every pod; the autoscaler compares the average against the target and adjusts the pod count.

The 50% target is a starting point. An EC2 instance is a virtual machine on shared hardware: the hypervisor and neighboring tenants introduce latency variance that a dedicated on-prem machine doesn’t have. The variance is negligible at low utilization and compounds at high utilization. In testing, tail latency grew sharply as CPU approached 90–100%. The target keeps the pods far from that zone and leaves headroom for traffic to grow while new pods start.

The target is also a cost dial. Raising it runs fewer replicas at higher utilization: cost falls and the pods move toward the variance zone. Lowering it runs more replicas at lower utilization: cost rises and tail latency eases. The right target is the one tail latency measurements on the actual workload support.

Crossing the 50% CPU target requests a new pod; the scheduler places it on a new node.

The node size is about consistency. A fixed size gives every pod the same allocation of vCPUs and memory.

EC2 instances come in fixed sizes, each a set vCPU count. One size spans many instance types, across families (compute, general, memory), architectures (ARM, AMD), and generations. Karpenter provisions the cheapest available across the set, serving the operating cost. The types differ in absolute speed, so the size is chosen for the slowest of them: every allowed type meets the latency budget running the workload.

Flex instance types are excluded. Their CPU performance is variable by design: they target average utilization, and a p95 budget has to hold on the slowest requests. The constraint allows Karpenter only dedicated, non-burstable compute.

A wide set raises the chance of finding capacity. Spot capacity for any single instance type can run out at any moment; with many types allowed, Karpenter provisions another. In the local zones this matters most: a local zone offers far fewer instance types than a region, leaving fewer alternatives when one runs out.

One pod per node removes neighbor contention. Two pods on the same node share the CPU caches, the memory bandwidth, and the NIC. That sharing appears in tail latency and in no CPU metric. With one pod per node, the whole machine serves a single workload.

Removing CPU limits removes throttling. A CPU limit in Kubernetes is enforced by CFS bandwidth control, the kernel mechanism that caps how much CPU time a group of processes may use per scheduling period. The container gets a slice of CPU time each period, and when the slice is spent, every thread in the container stops until the next period. The stall is measured in milliseconds, larger than the entire app budget.

A CPU request is a separate setting: the CPU amount the scheduler reserves for the pod on its node. Requests stay in place, so the scheduler still sizes placement correctly. A limit protects neighbors from a pod consuming too much; with one pod per node there is no neighbor, and the limit would only stall the pod itself. The pod can use the whole node.

A CPU limit is a quota of time per period; a container that spends it early stops until the next period.

Every open socket is a file descriptor. The connections a pod maintains grow with its request rate. Latency raises the count further: a slower response holds its connection busy longer, so the same rate needs more connections open at once. A default Linux host caps open descriptors at 1024; the node image raises the cap to roughly a million. At 1024 a pod reaches its connection ceiling long before its CPU ceiling, and the deployment would need far more pods than the work requires. The high cap lets CPU govern the pod count through the autoscaling target, so one pod carries its full share of the request rate. The cap spends none of the latency budget.

Chapter 5 Hosting

web server · health checks

Implementation

The API is a .NET 10 minimal API on Linux, with two endpoints: the lookup endpoint and the health check.

The health check has three consumers:

the load balancer’s target group health checks. Both NLB and ALB run this as an HTTP check on the endpoint’s path.
the Kubernetes readiness probe.
the Kubernetes liveness probe.

Rationale

A minimal API is the smallest hosting model ASP.NET Core offers: a route maps directly to a handler. Every middleware component a request passes through spends budget, so the pipeline carries only what the contract requires.

Kestrel is the web server built into ASP.NET Core, running in the same process as the app. It opens a port and listens on it, which is what makes the process reachable. The load balancer connects to that port, and Kestrel accepts the connection. Kestrel then owns the connection: it reads the bytes off it, runs the HTTP protocol that turns them into a request, and calls the matching handler.

The handler is the app code Kestrel calls: it receives the parsed request and returns the response. Kestrel writes that response back on the connection. The division is fixed: Kestrel is the transport and the HTTP protocol, the handler is the application logic.

One replica: the load balancer connects to Kestrel, which calls the handler and returns the response.

The health check reports whether the replica can serve. Each consumer acts on it independently. The load balancer routes requests only to targets its health checks mark healthy. The kubelet runs both probes: a passing readiness probe keeps the pod in the service’s endpoints, receiving traffic; a passing liveness probe keeps the container running. An NLB health check defaults to TCP, passing when the port accepts a connection. The target groups configure it as HTTP on the endpoint’s path, so it passes only when the HTTP layer answers. The ALB runs the same check.

Chapter 6 API logic

background fill

Implementation

The API runs two ways: the endpoint flow that answers each request, and the background tasks it schedules.

The endpoint flow covers the two app-side parts of the 5ms split (1ms logic, 1ms retrieve), 2ms in total. It runs inside that:

A request arrives with its payload of signals.
Each signal yields a key, taken from the payload or derived from it.
Some keys resolve against in-memory data; the rest go to the main store as a batch lookup, multiple keys per batch, under the 1ms retrieve budget.
The records found are transformed to the output form and returned.
When a key has no record, the follow-up depends on the key type: for some, nothing; for others, the missing data has to be supplied, so the endpoint schedules a background task and returns.

The endpoint code is optimized for per-call cost throughout.

The background task runs outside the response path. What it does depends on the key type:

it reads the value from a secondary source and writes it to the main store.
it hands off to a producer service, outside this API, that generates the record and writes it to the main store when finished. A frequency check gates the handoff; after the handoff, the task writes a placeholder record for the key.

The endpoint flow looks up the main store; the background tasks supply what it misses.

Rationale

Two of the drivers govern the data-access design.

The first is the 2ms endpoint budget. It admits only the fastest retrievals: key lookups against the main store and against in-memory data. The main store holds most of the data: the large, dynamic set. In-memory holds the static set, loaded at startup and refreshed on a fixed interval. The in-memory set has a capacity bound: the allowed instance types differ in memory, so the set is sized to fit the smallest.

The second is the deadline: the work of supplying data missing from the main store stays inside the API. The workload allows it: the keys recur, so a key that misses now returns later, and the cost of supplying it is amortized over its later occurrences. Generating data carries a cost, so the frequency check gates the producer handoff: a key is handed off only after it recurs enough times to justify that cost. A cold key can schedule the same fill more than once; the frequency check and the placeholder lower that repeat rate. Once the data is in the main store, the next request for the key returns on the hot path.

Keeping the fill inside the API raises tail latency: the background work shares the CPU with the request path and competes for cycles. Bounding that competition carries the largest yield cost. The fill tasks run under a concurrency limit protecting the request path, and backpressure drops scheduled fills over that limit; a dropped fill runs on a later recurrence of its key. The key returns nothing until a fill completes. The fill also widens the dependency set: multiple data sources and producer services, all with variable latency. Building a separate fill system would have exceeded the deadline; the trade is recoverable: the fill can move to its own system later.

The request rate constrains the code itself. Every operation on the endpoint flow repeats at the full request rate, so small per-call costs accumulate into latency and CPU. The implementation is written for that path: memory reused across requests, data structures fit to the access pattern, algorithms chosen for per-call cost. As a policy, per-call cost outranks readability and general design principles.

Chapter 7 Thread model

synchronous vs asynchronous I/O · epoll · thread pool · inline vs dispatch scheduling

Implementation

The request path runs on the SocketAsyncEngine threads (the runtime’s socket I/O threads), each request on the thread that read its socket. Background work is dispatched to the thread pool.

The data store client exposes its operations two ways, synchronous and asynchronous. The request path calls the synchronous operations, on the SocketAsyncEngine thread. The background tasks call the asynchronous operations.

The compared arrangement dispatches each request to the thread pool and calls the asynchronous lookups. Measured on this workload, running requests inline with synchronous lookups takes roughly half that compute, at lower latency. The business logic compute is negligible, so scheduling and lookup I/O set the compute per request.

The API’s request-response latency sits below 1ms at p95 and around 2ms at p99, varying by region. This is measured only on the local-zone deployments, which run ALB and publish a per-request latency metric.

Rationale

A synchronous call holds its thread for the whole operation. The thread issues the request, waits for the response, and resumes. For the duration of the wait it runs nothing else.

An asynchronous call releases the thread while the operation is in flight. The call starts the I/O and the thread picks up other work. The I/O proceeds in the kernel with no thread assigned to it, and on completion the remainder of the work is queued to run on a free thread.

Synchronous: the thread is held until the I/O responds. Asynchronous: the thread takes the next item; a free thread runs the completion.

On Linux, .NET handles socket I/O on a set of threads owned by SocketAsyncEngine, separate from the thread pool and named ”.NET Sockets” at the OS level. Each runs an epoll loop (epoll is the Linux facility for watching many file descriptors at once and reporting which ones are ready for I/O), so one thread can wait on many sockets at a time. The default count is the processor count divided by a per-architecture value (30 on x64, 8 on Arm64), with a minimum of one; at these node sizes that is one.

The default configuration dispatches each ready socket’s work, the read and the request, to the thread pool. Inline scheduling runs that work on the SocketAsyncEngine thread. Both meet the p95 budget; inline costs less in latency and compute.

Default: each request is dispatched to the thread pool. Inline: the SocketAsyncEngine thread runs the handler itself.

The thread pool is built for workloads that change over time. It keeps a set of worker threads, runs queued work on whichever is free, and sizes itself to the load: a controller adds and removes threads to hold throughput at the fewest threads, adjusting at most once per completed item or per 500ms. The count moves between a configurable minimum and maximum, set to fit the workload.

That adaptation has a cost, in two places. When load rises, the pool adds threads only after a delay, and work waits in the queue until then. And each item handed to the pool changes threads on the way: a context switch, and a start on a CPU cache holding none of the request’s data. In most workloads the work items are long, so both costs are a small fraction of each item and negligible. Here each request is short, the whole budget is 2ms, so the handoff is a meaningful fraction of it. And the load is steady, so the adaptation goes unused. A fixed set of threads running one kind of work fits better: the SocketAsyncEngine threads, running each request inline.

Kestrel dispatches by default to keep a slow handler off the SocketAsyncEngine thread. Inline scheduling gives up that protection: the handler runs on the SocketAsyncEngine thread, and a call that blocks there stalls every socket behind it. The number of SocketAsyncEngine threads is a fixed setting. Enabling inline scheduling changes the default from one to the processor count. Configuration can set any other value.

The dispatch also sets where overload accumulates. The default enqueues work to the pool as fast as readiness arrives, and the pool queue is unbounded, so overload builds there as queued work; a request canceled by the client still spends pool capacity when its turn comes. Inline scheduling reads each socket only when a thread is free to run its request, so overload stays in the bounded kernel socket buffers and TCP flow control slows the client. Intake is capped by the thread count. On the request path the work on the SocketAsyncEngine thread is the data store lookup.

The data store client provides that lookup in two forms, synchronous and asynchronous, differing in whether the lookup passes through the thread pool. The asynchronous form hands the store send to the thread pool: the store client queues the send to a worker, the worker starts it, and the SocketAsyncEngine thread stays in its loop serving other sockets. The store’s response completes back on the SocketAsyncEngine thread, so the pool runs only the send. That hand-off is the cost. Under load the send waits in the pool’s queue before it runs, and the 1ms retrieve budget counts that wait: the lookup can spend its budget in the queue before any network I/O to the store begins.

The synchronous form runs the store call on the SocketAsyncEngine thread and blocks it until the store responds. A multi-node batch queries the store nodes in series on that thread, so it can block across several round trips, all sharing the one budget. The full budget goes to the store I/O. The hold is viable because the store responds in under a millisecond, and the store client enforces that bound: a 1ms total timeout, the retrieve budget, fails any lookup that exceeds it and releases the thread. A slow store response holds the thread for about 1ms.

This deployment sets the count above the processor count: with some threads held on synchronous lookups, the rest keep serving sockets. The count comes from testing. A thread blocked on a lookup is off the kernel run queue until the store responds, so the cores run only the threads with work. The compute per request is small, so at any instant most threads wait and the few runnable ones fit the cores.

Node generation is one variable in the compute per request, and it varies by deployment: a newer generation runs the same work on fewer vCPUs. Measured on this workload, inside the latency budget and at the 50% scaling target, gen 9 sustained roughly 50% more requests per second per vCPU than gen 8. The rate falls with each older generation, and in some local zones the newest available is gen 5.

The background work sits outside the budget and off the request path. When a lookup misses a key that needs filling, the request path writes the signals for that fill to an in-memory channel, a bounded in-process queue, and returns. That write completes immediately: when the bounded channel is full, the write drops the new signal and the request path continues. The keys recur, so a dropped signal returns on a later request and the fill runs then.

A hosted background service, running on the thread pool, reads the channel, builds each fill task, and runs it under a concurrency limit that caps how many run at once. The limit is adjustable at runtime. The tasks are the background fills: a read from a slower source written back to the main store, the frequency checks gating a handoff, and the handoff to that service followed by a placeholder write.

The cap protects the request path and the services the tasks call. Those services run in other regions, so each call carries the cross-region distance latency, and their capacity sits far below this request rate. The concurrency limit and per-service QPS guards hold the background load within what those services accept.

Chapter 8 Main data source

partitioning · replication · hybrid memory architecture · direct I/O · local vs network-attached storage · IOPS ceilings · network bandwidth allowances · cross-datacenter replication

Implementation

The main data source is Aerospike, deployed as edge clusters: one cluster per deployment, in the metro it serves, answering the API’s reads.

In the regions, the edge cluster is the destination of active-passive cross-datacenter replication (XDR): a source cluster in a separate region, off the serving path, ships data to it, and the edge cluster serves the reads.

In the local zones, the edge cluster runs without XDR and is populated a different way.

Rationale

Three properties select Aerospike.

Read latency: in the right setup, reads return under 1ms at p99 at this request rate; reads are what this API does.
Replication: clusters replicate across datacenters, so records supplied by other sources and regions reach the cluster the reads run against.
The local-zone bound: a local zone offers limited EC2 compute for a cluster at this load, so that case is rearchitected on a memory-only setup.

These are edge clusters: each sits close to the metro it serves and answers the API’s reads. Each cluster has a minimum of two nodes, hybrid memory, instance-local NVMe.

Two nodes is the minimum that survives a node loss. Aerospike distributes data across the nodes by partition; redundancy comes from the replication factor, which places a copy of each partition on another node. At replication factor two with two nodes, each node is master for half the data and replica for the other half, so each holds a full copy. If one node fails, the other promotes its replicas to master and serves the whole dataset at reduced capacity until the failed node returns and the cluster rebalances.

Replication factor two on two nodes: each node is master for half the partitions and holds a full copy.

Aerospike runs in its hybrid memory configuration: the primary index in DRAM, the record data on the NVMe drives. A read resolves the key against the in-memory index, which holds the exact device and offset of the record, then issues one read to that location.

Keeping the data in DRAM as well would hold the whole record set in memory, which is larger than the DRAM on the nodes. Keeping the index on flash with the data would add a device read to every lookup, raising read latency. The hybrid split keeps the lookup in memory and reads only the record body from the device, which fits the data size and the retrieve budget together.

The DRAM index resolves the key to a device and offset; one NVMe read fetches the record.

Aerospike reads the data drives as raw block devices with Linux direct I/O, bypassing the OS page cache. Every record read is a direct device read at a fixed cost, independent of which records were read before. Under a 1ms retrieve budget that predictability matters: the same read costs the same time on every request.

The data drives are instance-local NVMe, physically attached to the node, which is what holds each read at the drive’s microsecond latency and keeps the per-read cost fixed. Network-attached block storage such as EBS routes every read over the network, adding latency and variance to each one. The cost of local drives is coupling: they are ephemeral and bound to the instance, so storage and compute scale together and the data lives only as long as the node. The in-cluster replication and the source cluster behind each region edge cluster make a lost node’s data recoverable, so the coupling is acceptable.

Aerospike distributes records across a node’s drives by a device-level hash. This is separate from the partition map: the partition map distributes the 4096 logical partitions across nodes, while the device distribution operates inside a node, across its drives. Each drive serves reads independently, so read capacity scales with drive count: two drives give roughly double the capacity of one.

For this workload the NVMe drives are the first resource to saturate. The index lookup is served from memory, but the record body is read from the device, so every read that misses the in-memory set is one device read. Device IOPS scale with the operation rate against a finite per-drive ceiling, and reads and writes consume it together: XDR ingest and the background fills land on the same drives. Under a high rate of small random reads the drives reach their ceiling before CPU or network reach theirs. At the ceiling, device I/O queues and read latency rises, until transactions reach their timeout. More drives raise the ceiling; adding a node raises it further by spreading the data and the read load across more drives. In the cloud the per-node drive count is fixed by the instance type, so this capacity grows by node count.

The network ceiling is also per instance. EC2 sets a bandwidth allowance by instance type and size; on smaller sizes the documented figure is a burst ceiling above a lower baseline. Past the allowance, packets queue or drop. For a database node, the allowance bounds read throughput the same way the drive ceiling does.

Two per-instance ceilings: the per-drive device IOPS ceiling and the network bandwidth allowance.

The signal to watch is read latency and read timeouts. On NVMe the utilization percentage is unreliable for this: it can report a drive fully busy while throughput remains available.

A region edge cluster receives its data through active-passive XDR replication from a source cluster, the cluster where the dataset’s writes land. Active-passive names the shipping direction: the source ships to the edge cluster, and a write landed directly on the edge cluster stays local. The background fills are such writes, caching the fetched record on the cluster the reads run against; the producer service’s records reach the main store written directly or shipped in by the replication. XDR is asynchronous, so a region edge cluster trails the source by the replication lag, and the data it serves can be slightly behind, an accepted tradeoff.

Active-passive XDR: the source cluster ships writes to each region's edge cluster, which serves the reads.

The replication pattern is deployment-agnostic: XDR ships records over ordinary network connections, so a source cluster and an edge cluster require only network reachability between them. An edge deployment in a colocation datacenter follows the same pattern: the source cluster stays in the region, XDR ships to the edge cluster deployed in the facility, and the API runs beside it, serving the client from inside the facility. Inside the facility the API-to-cluster leg carries negligible latency and no per-byte transfer charge.

XDR ships to an edge cluster in a colocation datacenter; the client's calls and the read path stay inside the facility.

The regional architecture, the disk-based two-node edge cluster fed by XDR, is unavailable in these local zones: a local zone offers a subset of the instance types a full region does, the storage-optimized instances the setup needs are absent, and the available higher-capacity options cost three to five times the regional price.

The local edge clusters run memory-only, with no XDR. Serving from a memory-only store changes how these clusters are populated and read, so the API runs a different path for the local-zone case.

Chapter 9 Local-zone data source

hot cache · TTL eviction · working set

Implementation

The local edge cluster runs memory-only, with no XDR.

It is a hot cache. The data that the regions serve from the main data source is reached here through an extension: the nearest region’s edge cluster, which holds that data on the hybrid setup. The other data sources operate unchanged.

The data path reuses the background-fill pattern:

A read misses in the local edge cluster.
A background fill reads the record from the extension and writes it to the local edge cluster.
The key recurs, and a later request finds the record in the local edge cluster.

The cache is short-lived: records carry a TTL and are refreshed or extended on access.

A miss schedules a background fill; the key recurs and hits. TTL eviction keeps the active fraction resident.

Rationale

Operating cost drives the choice.

The hybrid memory architecture needs storage-optimized hardware. The storage-optimized sizes available in the local zones are larger than the instance the regions run, underutilized at this load. A local-NVMe instance also costs more than a memory-only instance of the same compute and memory.

The chosen setup runs memory-only instances for the local edge cluster. The extension is the edge cluster in the nearest region, so the only added cost is the local in-memory nodes, less than the larger storage-optimized instances a local-zone cluster would have required.

Which records a metro requests in a day depends on the region, the weekday, the month, the season, and other factors, on the order of 5 to 10 percent of the set. The hot cache holds that active fraction, which fits in memory where the full set would not. The fraction varies with those factors, so the in-memory nodes start from a high baseline and are resized as running memory usage rises or falls, with the resident set preserved across a resize. The TTL is the eviction mechanism: a record not accessed within it expires, so only the active fraction stays resident.

From the local zone, the extension is a slower data source, slower mainly because of the distance: it sits in another metro, and the cross-region path carries the fiber-distance latency. The background-fill logic applies without change: a miss in the local edge cluster schedules a background read from it and writes the record back. The region’s ready-to-serve set is one more slower source behind that mechanism.

The cost is lower yield. The local edge cluster returns a record only after the key has been filled and while it remains resident, so more lookups return nothing than in a region. A lower match rate can mean lost revenue or missed revenue opportunity. The tradeoff holds while the hardware cost of the full setup stays above the revenue missed by the lower match rate.

Chapter 10 Measurement points

latency decomposition

Implementation

Measurement points sit along the request path, one per layer, from the client to the store. All metrics converge on one observability tool.

Client - end-to-end latency, measured in the client’s system, outside direct observability.
Load balancer - connection counts, TLS negotiation errors, and TCP reset counts, on both load balancer types; on ALB, additionally the API’s response latency at p95 and p99.
API - inbound request counters; outbound request counters and latency timers per dependency: the main store, the secondary sources, the producer service; error rates, cancellations, and timeouts around each; the match rate; executed background tasks.
Node - the ENA allowance counters on the API and store nodes: bandwidth, packets per second, connection tracking. Every hard per-instance limit is monitored at its enforcing layer, disk ceilings included.
Aerospike - read latency and read timeouts; device read and write latency; CPU usage; free disk space and free memory; XDR replication lag on the region edge clusters.

The measurement points along the request path, and the part of it each one covers.

Rationale

High throughput makes observability mandatory. The system operates near hard limits. Per-instance network allowances, device IOPS ceilings, connection counts, scheduling and configuration bounds are the obvious ones; the full set is longer. Each of them is invisible at a low request rate, and each is a candidate for the next bottleneck here. The measurement points exist to read the distance to every one of them.

End-to-end latency is measurable only where the request starts and the response lands. Every other point covers one segment of the path, and its metrics decompose the end-to-end number by segment.

The load balancer is the first point inside AWS. Elevated reset and TLS negotiation error counts indicate a connection-level problem, typically in the API. The ALB latency metric measures the app-side share of the round trip, separated from the internet leg.

The API measures each outbound call separately: a counter, a timer, and error rates per dependency. A latency increase or an error burst is attributed to the dependency that produced it. The inbound counters measure the request rate each deployment receives, with identical instrumentation across all deployments.

Hard per-instance limits are enforced below the app. Past a network allowance, packets queue or drop, and no app timer identifies the cause; the counters at the enforcing layer are the only signal. Every such limit is monitored at its enforcing layer, disk included.

The store’s metrics are read on the cluster itself. Read latency and read timeouts are the saturation signal for the drives; device latency, CPU usage, and free disk and memory track the distance to the node-level ceilings; replication lag bounds how far an edge cluster trails its source.

Chapter 11 Operating cost

billing dimensions · compression

Implementation

The AWS bill’s lines and what each one charges on:

Load balancers - per hour, plus the largest usage dimension of that hour: new connections, open connections, processed bytes; bytes counted in both directions.
Data transfer - egress per GB; ingress from the internet unbilled.
Storage compute - the store nodes’ instance hours.
Store licensing - the commercial edition subscription.
API compute - the API fleet’s instance hours.
Internal traffic - per-byte lines between components: cross-zone transfer, NAT processing, cross-region replication.

Rationale

Traffic shape sets the bill: request size, response size, request rate, and where the traffic lands. At this request rate moving the traffic outweighs serving it: the network lines dominate and compute is the minority.

The load balancer bills its largest dimension, so the traffic decides which one pays. Persistent connections hold the new-connection dimension near zero. Open connections track concurrency, request rate times response duration, so response time is itself a billable quantity. Heavy payloads put processed bytes on top. The two balancer types share this structure at different prices: the processed-bytes threshold is identical, the capacity unit is priced about a quarter lower on NLB, and the connection thresholds run over thirty times higher, with no rule-evaluation dimension. On a byte-dominated workload the gap reduces to the unit price: the roughly 30% the NLB deployments save.

Each byte then pays by direction. A heavy request bills once, at the load balancer, in a charge visible nowhere else: ingress is unbilled as transfer, so inbound payload surfaces only through the processed-bytes dimension. A heavy response bills twice, as processed bytes and again as egress. Payload weight is the largest dial: a byte trimmed from the request format saves near one to one, and trimming it is a change in the client protocol.

The egress line is response bytes times the per-GB rate: response size and request rate move it linearly. At sustained volume the rate is negotiated, and negotiated rates cover the regional egress; local-zone egress bills at its own rate, at or above the regional list, so the deployments that remove milliseconds can carry a minority of the traffic and a majority of the network cost: the latency is priced, a trade re-measured as traffic grows.

Storage compute is sized by the dataset and the read rate together. Data volume times the replication factor sets the storage to provision: growth there means bigger drives or more nodes. Read rate spends the per-node ceilings, the device throughput and the network allowance, and adds nodes as it exhausts them. Cross-region replication ships each write per byte at the inter-region rate, so write rate and record size move a network line too. The baseline runs continuously, the case commitment pricing exists for, at a fraction of the on-demand rate; a commitment expires, so coverage is tracked as a ratio, keeping steady workloads off the on-demand rate. Licensing adds a line on top of the hardware: cross-datacenter replication is a feature of the commercially licensed edition, so the subscription is part of the store’s cost.

API compute, tuned, is the smallest line: fleet size is request rate over per-vCPU throughput, so instance generation and the scaling target move it directly; a load-following fleet suits spot pricing, spread across many families and sizes so one interruption removes little capacity. Compression shifts cost between lines: compressing requests and responses spends cycles on the cheapest line to cut bytes on the two most expensive ones.

Internal transfer growing faster than business traffic marks misplaced services, or routing through a billed hop where an unbilled one exists.

The spread between this anatomy and an untuned one is configuration knowledge: the load balancer type, the purchase models, the negotiated rates, the instance generations are each a compared choice, and each default left standing bills the difference monthly.