A high-throughput lookup system under a 5 ms timeout

An honest note before we start. A general-purpose architecture serves many workloads, at a cost on each one. A design built for one workload’s shape and scale performs better on it, often by a wide margin. This entry describes one such design. Every statement in it holds for its workload. A workload of a different shape, or ten times larger or smaller, fits a different design, usually a simpler and cheaper one.

This entry follows the request path end to end, component by component, each in three parts. The implementation says what is in place. The rationale says why, against the drivers. The background, where a chapter has one, explains the general mechanisms the chapter uses, for readers new to them. This is a technical writeup; business context appears only where it drives a decision. The figures stay approximate: no argument here rests on them, and every mechanism can be verified independently.

Everything here is described as part of one system. Most of the details and patterns stand on their own. They can be adapted to a system with different drivers, once each is checked against those drivers. Where the baseline stack, the one the other systems run on, met the requirements, it stayed. Consistency across systems, and the proficiency and experience that come with it, make a switch worth its cost only when the baseline falls short. The base a new deployment needs, accounts, networks, pipelines, and monitoring, exists for it, and a new deployment extends that base. Everything else is the best fit I found for the drivers of this one, and each chapter explains why it was chosen. I hope some of them end up useful in yours.

Prologue Context

The system is a lookup service for the client, a real-time auction platform where each auction offers an item from a seller to a pool of buyers. For each auction the client sends a request carrying the auction’s signals, and from each signal the system takes a lookup key and returns the records its dataset holds for that key. Because an auction with records clears higher, the system sells the records to the buyers.

A buyer is signed under a contract that runs for a set term on a set budget and pays for a record only on the auctions it wins with it. Revenue is real time: a record earns when its auction clears, and an auction that clears without one earns nothing. A buyer also requires a volume of auctions with records over the term, which differs from buyer to buyer, and a larger volume means a larger contract. Because the product is new, buyers are signed mostly during a sales season, the part of the year when budgets and integrations open.

The auctions vary in value. The platform groups them into segments, each with its own buyer pool, and a record returned for a high-value segment sells for more than the same record returned for a low-value one.

The client’s platform runs in its own on-prem datacenters, in several metro areas across the globe. Each calls the system through the public endpoint exposed for it, a POST over HTTP/1.1 with all traffic on HTTPS, and each request carries its signals as a JSON payload.

Keys recur, each on its own schedule, and both the time between a key’s occurrences and the number of requests each occurrence brings vary widely from key to key. A key’s records are either already in the dataset or generated for it by the system itself, once the key’s early requests show it recurs enough times to be worth the cost of generating it, in compute and data. A request returns what the dataset holds at the time.

The dataset holds single-digit billions of records in single-digit terabytes, most of it in one main source and the rest across several others, each with its own retrieval pattern. At full scale the client sends single-digit millions of requests per second, and the share the system carries is set on the client side, by choosing which segments the client sends and by capping its request rate. The client enforces a 5 millisecond timeout per request, measured client-side and including network latency. Meeting this latency target at p95 is mandatory at the intended operating scale, and meeting it at p99 is required as the carried share approaches full scale.

From the description follow five architectural drivers and the rules for trading them. Together they form the decision framework the system is built with: two drivers fixed, in priority order, then three flexible, traded against each other down to floors.

• Latency - fixed, and the first driver in priority. The client enforces the 5 ms timeout from its datacenters, so the network and each component on the request path take part of those 5 ms. The design can use only components whose times add up to under the timeout at the percentile the carried share requires.
• Delivery - fixed, and the second driver in priority. Every change to the system, the initial build and each improvement after it, is timed to the sales season, because buyers are signed then, each for a volume deliverable at that point. A change live when the season opens is sold into that year’s contracts, and a late one waits a year. A delivery at partial capability still meets the date, which is what lets yield and scale be traded for it. Because revenue is real time, an improvement earns only once it is live, so it ships in increments, each live as soon as it works. A volume sold is held for its contract’s term, so yield and scale can be traded only above it.
• Yield - flexible, down to a floor. Yield is the volume of auctions with records the system delivers, the quantity a contract is signed on, and it follows from the match rate, the share of looked-up keys that find a record, and from the scale carried. Any volume above zero earns, so yield is the driver traded first where latency or delivery requires it. A contract counts its volume over the whole term, so a period of lower yield is made up within it, because keys recur and a later improvement matches them. Each trade is recorded when it is made, so the design carries its own list of future improvements. The floor is the volume each contract requires over its term.
• Scale - flexible, and an operational and business choice. A larger share of the traffic opens opportunities to new buyers, with a yield that may grow slower than the share, and costs more at each step, in money and in technical work. The design keeps a list of the components that change by hand as traffic grows, each with the traffic level at which it changes.
• Operating cost - flexible, down to a floor. A lower cost per request widens the margin on every contract, and at any carried share a small amount per request adds up to a large bill, so cost falls as each layer is understood and tuned. The floor is where a further cut takes yield or scale with it, and each such cut may lower revenue.

These five drivers and the rules for trading them are the decision framework the system is built with. A trade is decided on the current margin, revenue per record, and traffic carried, and decided again when those numbers change. The drivers stay the same.

Chapter 1 Latency budget

latency percentiles · circuit breaker

Implementation

The 5ms budget covers three layers:

• the network: everything on the wire before the API’s work begins.
• the API business logic: parse the payload, dispatch the lookups, build the response.
• data retrieval: pull the records from the data sources, across the billions of records.

Rationale

The three layers add up to the budget. Each layer can be built in several ways, and each way takes a different amount of time, so the choice made for one layer decides how much time is left for the other two. The network has the fewest options, because its time depends only on how far apart the client and the API sit and on which hops the packets cross between them. Both are decided at deployment, and the time they cost varies with the path, so it is measured on each deployment and the measurement comes off the budget first. The business logic and data retrieval have many more options, from how the code is written to how the records are stored, so they split the time that remains.

The 5ms limit is enforced from the client side, by a circuit breaker the client’s service runs on its calls to the API. Once more than 5% of responses exceed 5ms, the breaker opens and cuts the request rate until latency returns under the limit. A p95 target allows that same 5%, so a percentile close to p99 keeps headroom under it. A brief open is acceptable, and a long one loses yield while it lasts.

Background

Latency targets are set as percentiles. p95 means 95% of requests finish under the limit; the slowest 5% may exceed it.

A p99 target allows only 1% of requests to exceed the limit, so 4% that a p95 target allows must also finish under it. That slow tail comes from rare events like GC pauses, queuing, contention, and virtualization, which is why p99 is much harder to hold.

A circuit breaker is a guard that cuts traffic to a dependency while the dependency is slow.

Chapter 2 Network validation

speed of light in fiber · network congestion

Implementation

The call runs between two services in two different datacenters: the client’s service, on-prem, and the API, in AWS. Moving the API into the client’s datacenter is on the list of components that change as traffic grows.

The network is the one layer measurable before the system exists, because an empty endpoint is enough.

The validation deployment is that empty endpoint: it receives a request, deserializes it, and returns a mocked response. No lookups, no data sources, so the round trip measures the network alone.

The round trip is measured at the metro’s share of the full request rate, and what it leaves of the 5ms is the app budget. The app budget reserves about 1ms per node a lookup spans, the store floor. The app budget places the metro on its retrieval tier by what it covers:

• two store nodes: about 2ms plus the business logic.
• one store node: about 1ms plus the business logic.
• none: less, serving the data the API loads at startup, at reduced lookup coverage.

The client operates in several metro areas. Each metro gets its own deployment, validated independently, serving the client’s service in that metro.

• Some metros - each served by its own AWS region.
• The others - each served by an AWS local zone, where no region sat close enough.

Rationale

The API can run in one of two places: in the same datacenter as the client’s service, or in a cloud vendor’s datacenter in the same metro. The five drivers decide between them.

Driver	Client’s datacenter (colocation)	Cloud vendor, same metro
Latency	Minimal and stable, the same in every metro, inside one datacenter	Higher and variable, by hour with the public internet’s congestion and by metro with the fiber route to the nearest region or local zone
Delivery	Longer initial setup and each later capacity increase, weeks to months	Fastest, minutes
Yield	A datacenter outage takes the client down with the system, so no auctions run and no yield is lost	An AWS outage drops yield for its duration, as the client’s auctions keep clearing without records
Scale	The capacity is fixed at purchase and sized for the peak; each step up is another procurement	The capacity grows and shrinks with the traffic
Operating cost	Lowest per request at a large and steady scale; hardware bought and maintenance contracted per metro, possibly across multiple vendors, adding operational complexity; the layers above the hardware run in-house	Higher per request; one vendor across metros; managed services run the hardware, network, load balancer, patching and storage, with less technical work per layer

This system takes the cloud path, on delivery and scale. The procurement takes longer than the delivery allows, the share carried is small and changes with each season’s contracts, and the baseline stack already runs on AWS: the accounts, the pipeline, the monitoring and the on-call rotation exist, and AWS runs most of the layers below the API. A region or local zone of it sits within the latency budget of every metro the clients run in, so latency holds there. The higher cost per request is the trade. Colocation goes on the list of components that change as traffic grows, with the condition that a large carried share holds across contract terms.

The two services run in different datacenters within the same metro area. By default, the traffic crosses the public internet. A single round trip passes through intermediaries such as:

• the client’s NIC and OS network stack
• the on-prem datacenter’s LAN switches
• the on-prem edge router and firewall
• the datacenter’s uplink to its ISP
• a local internet exchange or peering point, handing traffic off toward AWS
• AWS’s edge into the region
• the load balancer

A real path crosses more hops, can cross several networks, and can return by a different route. Every hop adds time, and most of them sit outside direct control. The app’s stack and the load balancer are the exception. The rest belongs to the client, the ISPs, and the public routing in between.

AWS Direct Connect is a dedicated private network link into AWS, replacing the public internet’s variable routing with a stable, predictable path. Provisioning one takes procurement and installation time beyond what the delivery allows.

The one controllable variable is the fiber the signal travels, set by deployment location. Fiber length sets a floor on round-trip latency. The physical route can run 20–40% longer than the map distance, so the fiber floor holds even in-metro. Deploying in the same metro as the client keeps it short.

Congestion can only be measured, and the measurements have to include busy hours, because a path that meets the budget at a quiet hour can break it at a busy one.

The client side is the one point covering the full round trip, so end-to-end p95 is measured there.

Each deployment is validated independently because the metros differ in network path, load balancer type, and hardware generation. A metro’s tier is set at validation and holds until the next measurement.

Background

Light moves through glass at about two-thirds of its speed in vacuum, roughly 5 microseconds per kilometer one way, 10 round trip. A millisecond of round-trip latency is about 100 km of fiber.

Latency above the fiber floor comes from congestion: other traffic sharing the public internet.

Chapter 3 Load balancer

load balancing · layer 4 vs layer 7 · Availability Zones and regions · TLS handshake · TLS termination · DNS resolution and caching

Implementation

Inside AWS, the first thing a request reaches is the load balancer.

The load balancer differs by deployment:

• AWS regions - Network Load Balancer (NLB).
• AWS local zones - Application Load Balancer (ALB). NLB is not available in local zones.

Two settings applied everywhere:

• Single Availability Zone (AZ) - the load balancer is deployed in one AZ only: the one the app runs in.
• TLS 1.3 - listeners created through the API, CLI, or infrastructure-as-code default to a policy negotiating at most TLS 1.2; the listeners negotiate TLS 1.3. Termination stays at the load balancer.

Rationale

The load balancer’s time falls in the network round trip. The app budget begins once traffic reaches the API.

At this request rate, the NLB deployments cost about a quarter less. The variable cost rises with the work the load balancer does, and forwarding packets is the smallest unit of work. This path uses none of the routing rules and header inspection ALB adds.

A load balancer is enabled per zone, and each enabled zone gets its own address. Which address a client connects to is decided by DNS, and that decision sits outside the system’s control. The app runs in one zone. A client connecting to an address in a different zone has its traffic arrive there first, then cross to the app’s zone. The crossing adds time in each direction, and on NLB a per-gigabyte transfer charge. Enabling the load balancer in the app’s zone only means every client connects there directly.

A region ALB requires at least two zones to keep routing when one zone fails, and with two zones half its clients cross to the app’s zone. NLB runs in one zone, the second reason the regions use it. In the local zones the rule doesn’t apply, and one zone is enough for the ALB too.

A failure of the one zone takes the deployment down, and the outage stays in its metro until the zone recovers, with yield lost for the duration. Multi-AZ runs a copy of the deployment in two or more zones, DNS splitting the clients across their addresses and dropping a failed zone’s address within about a minute, so the metro keeps serving. The load balancer and the app add little cost, and the store adds a full copy in each zone, keeping store calls inside it and replicating over intra-region transfer, billed per byte. A distributed counter in the store changes with it. On one zone it is one write and one read, and across zones each increments its own count, a read sums them, and the sum trails by the replication lag between their clusters. Whether that cost is worth it depends on the deployment, and a region outage takes every zone with it. A metro with one local zone has no multi-AZ option, and with two the decision is the same.

TLS 1.3 removes one round trip from the handshake, and each round trip on this path takes significant time. Connections are persistent, but they still open, and each open pays the handshake. TLS termination at the load balancer keeps the handshake and the encryption work out of the app’s budget, and the app receives plain traffic over the internal network.

DNS resolution is kept outside the budget. An uncached resolution is a chain of round trips about as long as the whole 5ms budget. A resolution runs at connection open, at most once per TTL window, and every request inside the window reuses the cached address, so the cost per request is small.

Background

The app runs as many replicas; the load balancer is the one address in front of them, distributing incoming requests across the targets.

NLB works at layer 4. ALB works at layer 7: it terminates HTTP, parses the request, applies routing rules.

The layer also sets how load spreads. NLB distributes per connection. Each TCP connection is routed to one target for its life. With long-lived connections a few heavy clients stay on the same replicas, and load can sit uneven across them. A rising request rate opens more connections. Each new connection is routed to one target chosen among all of them, so a replica added by autoscaling receives its share of the new connections. Connections on this path close and reopen within minutes, so the new replica reaches an even share of the load within a similar time. ALB terminates the connection and routes each request on its own, so load spreads across replicas however the connections are distributed.

An Availability Zone has its own buildings, power, and network. An AWS region is three or more zones, close together but physically separate. An AWS local zone extends a region into another metro, with a subset of its services.

A TLS 1.2 handshake takes two round trips before any data moves; TLS 1.3 takes one.

DNS maps a name to the IP addresses behind it. Before a client can connect, it resolves the load balancer’s name to an address. The resolved address is cached on the client host, in the application runtime or the OS resolver. The cache is checked before any query leaves the machine, and the entry holds until the record’s TTL expires.

Chapter 4 Cluster

Kubernetes · Karpenter · EC2 purchase models · autoscaling · virtualization latency variance · burstable compute · CFS throttling · file descriptors

Implementation

The app runs in an EKS cluster. On the compute side:

• Node size fixed - one of the standard EC2 sizes, tailored to the workload. It spans multiple families (compute, general, memory) and architectures (ARM, x86), with flex types and the burstable T family filtered out. Karpenter provisions the cheapest available across the set.
• Spot preferred - when spot capacity is unavailable, on-demand is purchased.
• One pod per node - no two pods share a node.
• CPU limits removed from the pod spec.
• Autoscaling at 50% CPU - when average CPU crosses it, new pods are added.
• File descriptor limit (nofile) - ~1 million per container, set by the node image at the container runtime level.

Rationale

Kubernetes is part of the baseline stack and meets the compute requirements: pods scale on CPU, placement constraints keep one pod per node, and Karpenter fills the fixed node size from the cheapest spot or on-demand type. The 50% target is a starting point. Latency variance is small at low utilization and grows at high utilization. In testing, tail latency grew sharply as CPU approached 90–100%. The target keeps the pods far from that zone and leaves headroom for traffic to grow while new pods start.

The target also sets cost. Raising it runs fewer replicas at higher utilization: cost falls and tail latency grows. Lowering it runs more replicas at lower utilization: cost rises and tail latency eases. The right target is the one tail latency measurements on the actual workload support.

The node size is about consistency. A fixed size gives every pod the same allocation of vCPUs and memory.

EC2 instances come in fixed sizes, each a set vCPU count, and one size spans many instance types and generations. The types differ in absolute speed, so the size is chosen for the slowest of them, and every allowed type meets the latency budget running the workload. Karpenter provisions the cheapest type to lower operating cost.

Flex types and the burstable T family are excluded because both have variable CPU performance, and a p95 budget has to hold on the slowest requests.

A wide set raises the chance of finding capacity. Spot capacity for any single instance type can run out at any moment; with many types allowed, Karpenter provisions another. In the local zones this matters most: a local zone offers far fewer instance types than a region, leaving fewer alternatives when one runs out. A reclaim removes one node and the one pod on it, a small share of capacity, and Karpenter provisions a replacement.

One pod per node removes neighbor contention. Two pods on the same node share the CPU caches, the memory bandwidth, and the NIC. That sharing appears in tail latency and in no CPU metric.

Removing CPU limits removes throttling, whose stalls are measured in milliseconds, larger than the entire app budget.

A CPU request is a separate setting: the CPU amount the scheduler reserves for the pod on its node. Requests stay in place, so the scheduler still sizes placement correctly. A limit protects neighbors from a pod consuming too much. With one pod per node there is no neighbor, so the pod can use the whole node, and a limit would only stall it.

Each open connection holds a file descriptor. The connections a pod maintains grow with its request rate. Latency raises the count further: a slower response holds its connection busy longer, so the same rate needs more connections open at once. At the default Linux cap of 1024 open descriptors, a pod reaches its connection ceiling long before its CPU ceiling, and the deployment would need far more pods than the work requires. The high cap lets CPU set the pod count through the autoscaling target, so one pod carries its full share of the request rate. The cap spends none of the latency budget.

Background

The app runs on Kubernetes. EKS is AWS’s managed version of it. Kubernetes runs applications as pods: a pod is one running copy of the app, the replica the load balancer targets. Pods are placed on nodes, and a node is an EC2 machine. Kubernetes decides which pod lands on which node. The decision accepts constraints: which nodes a pod may land on, and which pods may share one.

Karpenter creates the nodes themselves: when pods need a machine, it provisions one, and the configured constraints tell it which machines it’s allowed to create.

EC2 sells the same machine under several purchase models. On-demand: the list price, no commitment. Reserved: a one- or three-year commitment on defined capacity, at a discount that grows with the term, suited to the steady baseline of a load. Spot: spare capacity at up to 90% below the on-demand price, which AWS reclaims on a two-minute notice when it needs the capacity back.

Autoscaling is how the deployment follows the traffic. The metrics server collects CPU usage from every pod; the autoscaler compares the average against the target and adjusts the pod count.

An EC2 instance is a virtual machine on shared hardware. The hypervisor and neighboring tenants add latency variance that a dedicated on-prem machine doesn’t have.

Flex types run on a time-shared baseline; the burstable T family runs on CPU credits.

A CPU limit in Kubernetes is enforced by CFS bandwidth control, the kernel mechanism that caps how much CPU time a group of processes may use per scheduling period. The container gets a quota of CPU time each period, and when the quota is spent, every thread in the container stops until the next period.

Chapter 5 Hosting

web server · health checks

Implementation

The API is a .NET 10 minimal API on Linux, with two endpoints: the lookup endpoint and the health check.

The health check has three consumers:

• the load balancer’s target group health checks. Both NLB and ALB run this as an HTTP check on the endpoint’s path.
• the Kubernetes readiness probe.
• the Kubernetes liveness probe.

Rationale

.NET is part of the baseline stack and meets the request path requirements, with a lean hosting model, a web server in the app’s process, and a GC and socket engine configurable to the budget. A minimal API is the smallest hosting model ASP.NET Core offers: a route maps directly to a handler. Every middleware component a request passes through spends budget, so the pipeline carries only what the contract requires.

The health check reports whether the replica can serve, and each consumer acts on it independently. The load balancer routes requests only to targets its health checks mark healthy. Of the kubelet’s two probes, a passing readiness probe keeps the pod in the service’s endpoints, receiving traffic, and a passing liveness probe keeps the container running. An NLB health check defaults to TCP, passing when the port accepts a connection. The target groups on both types configure it as HTTP on the endpoint’s path, so it passes only when the HTTP layer answers.

Background

Kestrel is the web server built into ASP.NET Core, running in the same process as the app. It opens a port and listens on it, which is what makes the process reachable. The load balancer connects to that port, and Kestrel accepts the connection. Kestrel then owns the connection: it reads the bytes off it, runs the HTTP protocol that turns them into a request, and calls the matching handler.

The handler is the app code Kestrel calls: it receives the parsed request and returns the response. Kestrel writes that response back on the connection. The division is fixed: Kestrel is the transport and the HTTP protocol, the handler is the application logic.

The kernel keeps a receive buffer and a send buffer for each connection. It completes the TCP handshake and queues the connection until Kestrel accepts it. Request bytes arrive in the receive buffer, acknowledged by the kernel, and Kestrel reads them from there into its own memory and parses them. The handler is called once the request line and headers are parsed, and reads the body from the same memory. Kestrel writes the response to the send buffer, and the kernel transmits it after the write returns. The connection stays open, and Kestrel parses the next request on it. The TCP handshake is paid once, at connection open.

Chapter 6 API logic

garbage collection · generations · GC pauses · fragmentation

Implementation

The API runs two ways: the endpoint flow that answers each request, and the background tasks it schedules.

The endpoint flow runs inside the app budget, the 5ms less the network measurement:

• A request arrives with its payload of signals.
• Each signal gives a key, taken from the payload or derived from it.
• Some keys are looked up in memory; the rest go to the main store as a batch lookup, multiple keys per batch, about 1ms per node it spans.
• The records found are transformed to the output form and returned.
• When a key has no record, the follow-up depends on the key type. Some types need nothing. For the others, the endpoint schedules a background task to supply the missing data, then returns.

All endpoint code is written for per-call cost, with memory reused across requests and data structures fit to the access pattern.

A concurrency limit caps the background tasks running at once, and a bounded channel holds those waiting. What each does depends on the key type:

• it reads the value from a secondary source and writes it to the main store.
• it hands off to a producer service, outside this API, that generates the record, which then reaches the main store. A frequency check gates the handoff; after the handoff, the task writes a placeholder record for the key.

Rationale

The data access is designed around two drivers.

The first is the app budget. It allows only the fastest retrievals: key lookups against the main store and in-memory data. The main store holds most of the data: the large, dynamic set. In-memory holds the static set, loaded at startup and refreshed on a fixed interval. The allowed instance types differ in memory, so the in-memory set is sized to fit the smallest.

The second is the delivery: the work of supplying data missing from the main store stays inside the API. The workload allows it: the keys recur, so a key that misses now returns later, and the cost of supplying it is spread over its later occurrences. Generating data carries a cost, so a key is handed off only after it recurs enough times to be worth that cost. A cold key can schedule the same fill more than once; the frequency check and the placeholder lower that repeat rate. Once the data is in the main store, the next request for the key returns on the hot path.

Keeping the fill inside the API raises tail latency: the background work shares the CPU with the request path and competes for cycles. Limiting that competition costs the most yield. A fill scheduled against a full channel is dropped and runs when its key recurs, and the key returns nothing until a fill completes. The fill also adds dependencies: multiple data sources and producer services, all with variable latency. A separate fill system takes longer to build than the delivery allows, and moving the fill to its own system later keeps the trade recoverable.

Every operation on the endpoint flow repeats at the full request rate, so small per-call costs add up to latency, CPU, and GC pauses, and the allocation rate sets how often they occur. Per-call cost comes before readability and general design principles.

Background

The garbage collector frees heap memory held by objects the app no longer references. The API runs Server GC, the ASP.NET Core default, which collects on dedicated threads.

The heap is divided into three generations: gen 0, gen 1, and gen 2. New objects start in gen 0, which is collected most often. An object that survives a GC moves up one generation, up to gen 2. Collecting a generation includes the younger ones, so a gen 2 GC covers the whole heap.

Before a GC starts, the runtime suspends every thread running .NET code except the triggering one, so requests being processed wait. That wait is the GC pause. Gen 0 and gen 1 collections are blocking, so those threads stay suspended until the collection ends. By default, gen 2 collections run as background GC, alongside the app and with shorter pauses.

A GC starts when gen 0 fills, so a higher allocation rate means more frequent collections, and more requests wait through a pause. The length of each pause follows how much memory survives the collection. Objects that live for one request mostly die in gen 0, so little survives a gen 0 GC and it ends quickly.

Fragmentation is the free space left between live objects on the heap. When fragmentation in gen 2 is very high, the GC can move objects together in a full blocking collection, a pause that grows with the heap size.

Chapter 7 Thread model

synchronous vs asynchronous I/O · epoll · thread pool · inline vs dispatch scheduling

Implementation

The request path runs on the SocketAsyncEngine threads (the runtime’s socket I/O threads), each request on the thread that read its socket. Background work is dispatched to the thread pool. The SocketAsyncEngine thread count is a multiple of the pod’s vCPUs, set from testing.

The data store client provides its operations two ways, synchronous and asynchronous. The request path calls the synchronous operations, on the SocketAsyncEngine thread. The background tasks call the asynchronous operations.

The default configuration dispatches each request to the thread pool and calls the asynchronous lookups. Measured on this workload, both configurations meet the p95 budget, and running requests inline with synchronous lookups takes roughly half the compute, at lower latency. The business logic compute is small, so scheduling and lookup I/O set the compute per request.

The API’s request-response latency has to hold under the app budget at p95. It is measured only on the local-zone deployments, which run ALB and publish a per-request latency metric.

Rationale

On Linux, .NET handles socket I/O on a set of threads owned by SocketAsyncEngine, separate from the thread pool and named ”.NET Sockets” at the OS level. Each runs an epoll loop, so one thread can wait on many sockets at a time. The default count is the processor count divided by a per-architecture value (30 on x64, 8 on Arm64), with a minimum of one. At these node sizes that is one.

The thread pool adapts to load, at a cost in two places. When load rises, the pool adds threads only after a delay, and work waits in the queue until then. Each item handed to the pool also changes threads on the way: a context switch, and a start on a CPU cache holding none of the request’s data. In most workloads the work items are long, so both costs are a small fraction of each item. Requests on this path are short and the app budget is a few milliseconds, so the handoff takes a large share of it. The load is also steady, so the pool holds the same thread count. A fixed set of threads running one kind of work fits better, so the SocketAsyncEngine threads run each request inline.

Kestrel dispatches by default to keep a slow handler off the SocketAsyncEngine thread. Inline scheduling removes that protection, because a call that blocks on the SocketAsyncEngine thread stalls every socket behind it. The number of SocketAsyncEngine threads is a fixed setting. Enabling inline scheduling raises its default from one to the processor count, and configuration can set any other value.

The scheduling also decides where overload builds. The default queues work to the pool as soon as each socket becomes ready. The pool queue is unbounded, so overload builds there as queued work. A request canceled by the client still spends pool capacity when its turn comes. Inline scheduling reads each socket only when a thread is free to run its request. Overload stays in the bounded kernel socket buffers, and TCP flow control slows the client. The thread count caps the requests in progress. On the request path the work on the SocketAsyncEngine thread is the data store lookup.

The two forms differ in whether the lookup passes through the thread pool. The asynchronous form hands the store send to the thread pool. The store client queues the send to a worker, the worker starts it, and the SocketAsyncEngine thread stays in its loop serving other sockets. The store’s response completes back on the SocketAsyncEngine thread, so the pool runs only the send. That hand-off is the cost. Under load the send waits in the pool’s queue before it runs, and the timeout starts once it does. The wait spends the app budget before any network I/O to the store begins.

The synchronous form runs the store call on the SocketAsyncEngine thread and blocks it until the store responds. A multi-node batch queries the store nodes in turn on that thread, so it can block across several round trips, each under its own 1ms timeout. The hold works because the store responds in under a millisecond. That thread waits on the socket for the response, and the timeout fails any query that exceeds it. A slow store response holds the thread for about 1ms per node the batch spans. The queries can also run in parallel, at about 1ms whatever the count, and each then costs a handoff to the pool.

By Little’s law, the threads held at once average the request rate times the hold, about 1ms per node the batch spans plus the handler’s CPU time. At a fixed CPU target the request rate grows with the vCPUs, and so does the count. A thread blocked on a lookup is off the kernel run queue until the store responds, so the cores run only those with work.

The compute per request depends partly on the node generation, which varies by deployment. A newer generation runs the same work on fewer vCPUs. Measured on this workload, inside the latency budget and at the 50% scaling target, gen 9 sustained roughly 50% more requests per second per vCPU than gen 8. The rate falls with each older generation, and in some local zones the newest available is gen 5.

On a miss that needs a fill, the request path writes the fill’s signals to the bounded channel and returns. The write completes without waiting, on a full channel too, and the SocketAsyncEngine thread returns to its sockets.

A hosted background service on the thread pool reads the channel and runs the fills under the concurrency limit, adjustable at runtime. The services the fills call run in other regions, with capacity far below this request rate. The limit and per-service QPS guards hold the background load within what those services accept.

Background

A synchronous call holds its thread for the whole operation. The thread issues the request, waits for the response, and resumes. While it waits it runs nothing else.

An asynchronous call releases the thread while the operation is in flight. The call starts the I/O and the thread picks up other work. The I/O runs in the kernel with no thread assigned to it, and on completion the rest of the work is queued to run on a free thread.

Epoll is the Linux facility for watching many file descriptors at once and reporting which ones are ready for I/O.

The thread pool is built for workloads that change over time. It keeps a set of worker threads, runs queued work on whichever is free, and sizes itself to the load. A controller adds and removes threads to hold throughput at the fewest threads, adjusting at most once per completed item or per 500ms. The count moves between a configurable minimum and maximum, set to fit the workload.

Chapter 8 Main data source

partitioning · replication · hybrid memory architecture · direct I/O · local vs network-attached storage · IOPS ceilings · network bandwidth allowances · cross-datacenter replication

Implementation

The main store is Aerospike, deployed as edge clusters: one cluster per deployment, in the metro it serves, answering the API’s reads. Each cluster runs one or two nodes, the count set by the metro’s retrieval tier. The nodes run the hybrid memory configuration, the primary index in DRAM and the record data on instance-local NVMe.

In the regions, the edge cluster receives its data by active-passive cross-datacenter replication (XDR). A source cluster in a separate region, off the serving path, ships data to it.

In the local zones, the edge cluster runs without XDR and receives its data a different way.

Rationale

Aerospike is part of the baseline stack and meets the store requirements on three properties.

• Read latency: in the right setup, reads return under 1ms at p99 at this request rate. Reads are what this API does.
• Replication: clusters replicate across datacenters, so records supplied by other sources and regions reach the cluster the reads run against.
• Storage configuration: the store runs hybrid memory or memory-only, so a local zone, offering limited EC2 compute for a cluster at this load, runs memory-only.

Two nodes is the minimum that survives a node loss. Aerospike distributes data across the nodes by partition; redundancy comes from the replication factor, which places a copy of each partition on another node. Each node a lookup spans adds about 1ms to the store floor, so the app budget caps the count. The store client queries them in turn, each query under its own 1ms timeout. The client takes timeouts in whole milliseconds, so 1ms is the smallest it can set. At replication factor two with two nodes, each node is master for half the data and replica for the other half, so each holds a full copy. Rack-aware routing can then send a whole batch to one node, for about 1ms. The pinning costs a server-side rack id and a client per node, and holds while the replication factor equals the node count. If one node fails, the other promotes its replicas to master and serves the whole dataset at reduced capacity until the failed node returns and the cluster rebalances.

The one-node tier trades on four points: node loss, drive ceiling, cost, and latency. The record data lives on local drives and goes with a lost node. The metro then serves the static set until the node returns and the background fills refill it. Reads go to the master partition, so two nodes split the read load across their drives and one carries all of it. One node costs half. It holds the store floor at about 1ms, one round trip in turn, against about 2ms on two.

Keeping the data in DRAM as well would hold the whole record set in memory, which is larger than the DRAM on the nodes. Keeping the index on flash with the data would add a device read to every lookup, raising read latency. The hybrid split keeps the lookup in memory and reads only the record body from the device, which fits the data size and the 1ms timeout together.

Aerospike reads the data drives as raw block devices with Linux direct I/O, bypassing the OS page cache. Every record read is a direct device read at a fixed cost, whichever records were read before. Under the 1ms timeout, the same read has to cost the same time on every request.

Instance-local NVMe, physically attached to the node, holds each read at the drive’s microsecond latency and keeps the per-read cost fixed. Local drives are ephemeral and belong to the instance, so storage and compute scale together and the data lives only as long as the node. The in-cluster replication and the source cluster behind each region edge cluster make a lost node’s data recoverable, so ephemeral drives are acceptable.

For this workload the NVMe drives saturate first, because every lookup sent to the store is one device read. Every read and write uses device IOPS against a fixed per-drive ceiling. XDR ingest and the background fills land on the same drives as the reads and count toward the same ceiling. Under a high rate of small random reads the drives reach their ceiling before CPU or network reach theirs. At the ceiling, device I/O queues and read latency rises, until transactions reach their timeout. More drives raise the ceiling, and a further node brings its own. In the cloud the instance type fixes the per-node drive count. The ceiling grows with the instance type first, and by node count where the app budget holds another 1ms.

The network ceiling is also per instance. EC2 sets a bandwidth allowance by instance type and size; on smaller sizes the documented figure is a burst ceiling above a lower baseline. Past the allowance, packets queue or drop. For a database node, the allowance limits read throughput the same way the drive ceiling does.

On NVMe the utilization percentage is unreliable as a saturation signal, because it can report a drive fully busy while throughput remains available.

The source cluster is the cluster where the dataset’s writes land. Active-passive names the shipping direction: the source ships to the edge cluster, and a write landed directly on the edge cluster stays local. The background fills are such writes, caching the fetched record on the cluster the reads run against. The producer service’s records land on the source cluster and ship in by the replication. The replication runs over VPC peering, a route between the two regions’ private networks that carries the traffic on private addresses. XDR is asynchronous, so a region edge cluster trails the source by the replication lag. The data it serves can be slightly behind, which is accepted.

Background

Aerospike distributes records across a node’s drives by a device-level hash. This is separate from the partition map: the partition map distributes the 4096 logical partitions across nodes, while the device distribution operates inside a node, across its drives. Each drive serves reads independently, so read capacity scales with drive count: two drives give roughly double the capacity of one.

A read resolves the key against the in-memory index, which holds the exact device and offset of the record, then issues one read to that location.

Network-attached block storage such as EBS routes every read over the network, adding latency and variance to each one.

The replication pattern works wherever the edge cluster runs. XDR ships records over ordinary network connections, so a source cluster and an edge cluster need only a network route between them. An edge deployment in a colocation datacenter would follow the same pattern. The source cluster would stay in the region, and XDR would ship to the edge cluster in the facility. The API would run beside it and serve the client from inside the facility. Inside the facility the API-to-cluster hop would add little latency and no per-byte transfer charge.

Chapter 9 Local-zone data source

Implementation

The local edge cluster runs memory-only, with no XDR.

It is a hot cache for the data the regions serve from the main store. The data comes from an extension, the nearest region’s edge cluster, which holds it on the hybrid setup. The other data sources work the same.

The data path reuses the background-fill pattern:

• A read misses in the local edge cluster.
• A background fill reads the record from the extension and writes it to the local edge cluster.
• The key recurs, and a later request finds the record in the local edge cluster.

The cache is short-lived: records carry a TTL and are refreshed or extended on access.

Rationale

The local edge cluster runs memory-only because the hybrid setup needs storage-optimized hardware. In the local zones, the storage-optimized sizes are larger than the instance the regions run, underused at this load, at three to five times the regional price. A local-NVMe instance also costs more than a memory-only instance of the same compute and memory.

Because the extension is the region’s own edge cluster, the only added cost is the local memory-only nodes.

A metro requests on the order of 5 to 10 percent of the set in a day. The hot cache holds that active fraction, which fits in memory where the full set would not. Because the records requested depend on the region, the weekday, the month, the season, and other factors, the fraction varies, so the memory-only nodes start from a high baseline. To add memory or cut cost, a resize builds a mirror cluster at the new size and moves reads to it. The TTL keeps only the active fraction resident.

From the local zone, the extension is a slower data source, mainly because of the fiber distance to another metro, so the extension is read by the background fill.

The cost is lower yield. Because the local edge cluster returns a record only after the key has been filled and while it remains resident, more lookups return nothing than in a region. A lower match rate can mean lost or missed revenue. The memory-only setup is worth the lower yield while the full setup’s hardware costs more than the missed revenue.

Chapter 10 Measurement points

Implementation

Measurement points sit along the request path, one per layer, from the client to the store. All metrics past the client go to one observability tool.

• Client - end-to-end latency, measured in the client’s system.
• Load balancer - connection counts, TLS negotiation errors, and TCP reset counts, on both load balancer types; on ALB, also the API’s response latency at p95 and p99.
• API - inbound request counters; outbound request counters and latency timers per dependency: the main store, the secondary sources, the producer service; error rates, cancellations, and timeouts around each; the match rate; executed background tasks.
• Node - the ENA allowance counters on the API and store nodes: bandwidth, packets per second, connection tracking. Every hard per-instance limit is monitored at its enforcing layer, disk ceilings included.
• Aerospike - read latency and read timeouts; device read and write latency; CPU usage; free disk space and free memory; XDR replication lag on the region edge clusters.

Rationale

At high throughput the system operates near hard limits, so the measurement points are worth more than at a low request rate. Per-instance network allowances, device IOPS ceilings, connection counts, scheduling and configuration bounds are the obvious ones in a longer set. A low request rate reaches none of them, and at this rate any of them can become the next bottleneck. The measurement points show how close the system is to each one.

End-to-end latency is measurable only where the request starts and the response lands. Every other point covers one segment of the path, and its metrics split the end-to-end number by segment.

The load balancer is the first point inside AWS. High reset and TLS negotiation error counts point to a connection-level problem, usually in the API. The ALB latency metric measures the app-side share of the round trip, separated from the internet round trip.

The per-dependency metrics show which dependency produced a latency increase or an error burst. The inbound counters measure the request rate each deployment receives, with the same instrumentation in every deployment.

Hard per-instance limits are enforced below the app, so the app’s timers show the slowdown and only the node’s allowance counters show its cause.

The store’s metrics are read on the cluster itself. Read latency and read timeouts show when the drives saturate. Device latency, CPU usage, and free disk and memory show how close a node is to its ceilings. Replication lag measures how far an edge cluster trails its source.

Chapter 11 Operating cost

Implementation

The AWS bill’s lines and what each one charges on:

• Load balancers - per hour, plus the largest usage dimension of that hour: new connections, open connections, processed bytes. Bytes count in both directions.
• Data transfer - egress per GB; ingress from the internet unbilled.
• Storage compute - the store nodes’ instance hours.
• Store licensing - the commercial edition subscription.
• API compute - the API fleet’s instance hours.
• Internal traffic - per-byte transfer between components: cross-region replication over VPC peering.

Rationale

The bill depends on traffic shape: request size, response size, request rate, and where the traffic lands. At this request rate the network lines cost more than compute.

The load balancer bills in capacity units, each covering a fixed amount of every dimension. Usage divided by that amount gives the units per dimension, and each hour bills the largest, so traffic shape sets the billed dimension. Persistent connections hold the new-connection dimension near zero. Open connections track concurrency, request rate times response duration, so response time is itself billed. Heavy payloads make processed bytes the billed dimension. The two balancer types share this structure at different prices. One unit covers 1 GB of processed bytes per hour and 3,000 open connections on both types. For TLS listeners it covers 50 new connections per second on NLB and 25 on ALB. ALB adds a rule-evaluation dimension. The hourly base rate is equal, and the capacity unit is priced about a quarter lower on NLB. On a byte-dominated workload the gap is the unit price.

Each byte bills by direction. A heavy request bills once, at the load balancer, in a charge visible nowhere else: ingress is unbilled as transfer, so inbound payload bills only as processed bytes. A heavy response bills twice, as processed bytes and again as egress. These lines move most with payload weight. A byte trimmed from the request format saves about one billed byte, and trimming it changes the client protocol.

The egress line is response bytes times the per-GB rate. Response size and request rate move it directly. At steady high volume the rate is negotiated, and negotiated rates cover the regional egress. Local-zone egress bills at its own rate, at or above the regional list. The deployments that remove milliseconds can carry the smaller part of the traffic and the larger part of the network cost. Lower latency costs a higher rate, re-measured as traffic grows.

Storage compute is sized by the dataset and the read rate together. Data volume times the replication factor sets the storage to provision: growth there means bigger drives or more nodes. Read rate spends the per-node ceilings, the device throughput and the network allowance, and adds nodes as it reaches them. Cross-region replication ships each write over VPC peering, billed per byte at the inter-region rate, so write rate and record size move a network line too. The baseline runs continuously, so commitment pricing covers it at a fraction of the on-demand rate. A commitment expires, so coverage is tracked as a ratio, keeping steady workloads off the on-demand rate. Cross-datacenter replication is a feature of the commercially licensed edition, so the subscription adds a line to the store’s hardware cost.

API compute, tuned, is the smallest line. Fleet size is request rate over per-vCPU throughput, so instance generation and the scaling target move it directly. A fleet that follows the load suits spot pricing. Compressing requests and responses shifts cost between lines. It adds CPU work to API compute and cuts bytes on processed bytes and egress, the charges behind the two largest lines. The trade favors compressing where bytes bill per GB. Inside a colocation facility the client’s calls would cross a fixed-capacity port with no per-byte transfer charge. The remaining reason to compress is a link near its bandwidth limit.

Internal transfer growing faster than business traffic points to misplaced services, or routing through a billed hop where an unbilled one exists.

Three discounts apply to the bill, each reaching its own lines. Commitment pricing is bought per instance hour and reaches the store nodes, as reserved capacity in the regions and as a savings plan in the local zones. An enterprise agreement commits to a yearly spend for several years and takes one percentage off list across most of the bill. The load balancer has one list price at every volume, so its rate drops through this agreement alone. Egress is negotiated on its own, against a committed monthly volume. The network lines carry most of this bill and both move through negotiation, so the agreements matter more here than the purchase models.

The spread between this bill and an untuned one comes from configuration choices: the load balancer type, the purchase models, the negotiated rates, the instance generations. Each is chosen by comparing options. Each default left unchanged raises the monthly bill.