The compiled read model

Context

A general-purpose architecture carries the cost of its generality; for a workload of a specific shape and scale, a design built on that workload’s properties outperforms it, often by a wide margin. The compiled read model is one such design, for reads of data that may be a cycle old at a rate and latency budget where request-time assembly is the dominant cost.

A conventional read over distributed data assembles its answer at request time from queries against data sources, and its latency is the sum of the hops: network round trips, storage retrieval, calls to other services. Lowering that latency means adding infrastructure. Caches sit in front of storage, replicas spread the load, retries and timeouts guard the network. Each layer reduces the cost of the layer below it, and each layer adds its own operational cost.

A compiled read model removes that infrastructure from the read. The assembly runs ahead of the request, on a schedule: data is fetched from the sources, compiled into the layout the reads need, and loaded as one unit into the memory of every serving process. At request time a read runs against a local structure that already has the shape it needs.

Read performance becomes an algorithmic property. Latency is decided by the lookup, the number of memory accesses and how local the layout is. Data that takes several round trips across several systems on the conventional path takes a single lookup in a data structure.

Workflow

Fetch. The current data is read from the same sources a conventional read path queries at request time.

Build. The data is compiled into the model, joined, derived, and indexed into the layout its reads need.

Publish. The finished model is written to object storage as a new immutable version.

Update. Each serving process loads the new version.

Read. Each read runs in process against the loaded model.

Traits

• Linear read scaling. Each process holds the whole model, making it a complete read path in itself. Read capacity scales by process count alone.
• Eventually consistent. The model is built from a snapshot of the data sources. Each cycle rebuilds it from a newer snapshot, so the data a read serves is at most one cycle old. Freshness is set by the cycle length.
• Algorithmic reads. A read is solved in memory, an algorithm over data structures built to fit it. The structures take whatever shape serves the workload, and the read logic carries as much of the work as the query demands. The split between what the build precomputes and what the read computes is a design choice per workload.
• Immutable and versioned. A published model never changes. Change arrives only as the next version, loaded whole. An answer traces to the version that served it.
• Narrow dependencies. The model store is the single external dependency, and access to it is cyclic: one load per cycle, off the request path.

Requirements

• Consistency tolerance. Reads serve data up to one cycle old, and during an update processes swap versions independently, so concurrent reads can answer from different versions. The workload has to tolerate both.
• Memory capacity. The process must hold the whole model in memory, with room left for its own working set and for the update mechanism’s cost. Memory bandwidth is a limit too. At extreme read rates, reads together can require more data per second than the machine’s memory can deliver.
• Model growth. Keeping the model in memory as the data grows is design work. The layout and the read logic evolve with the size.
• Memory management. Each update allocates and reclaims memory at model scale in a live serving process, a recurring cost whatever the mechanism. The choice of mechanism is open. It decides which resource carries the cost, and the process needs headroom there.
• Longer startup. A serving process is ready only after loading the model, so startup lengthens by the load time. Readiness checks must reflect that.
• Backward compatibility. The read code and the model schema deploy separately, so a process can hold a model one schema version apart from its code. A schema change ships with read code that handles both versions until the transition completes.
• Complexity. The fetch, build, and publish cycle is a new module in the platform. Building and operating it is standing overhead a compiled read model adds.

Example

The system to build is a matching service. Each request in a stream gets one answer: which of the configured entries apply to it. Requests arrive at hundreds of thousands per second into several datacenters, matching runs in each of them, and each answer has a latency ceiling. The entries come from a control panel, where hundreds of operators edit their own entries on their own schedule.

Each entry holds conditions on request attributes. Some attributes are on the request. Others have to be looked up in mapping tables in another system. Matching is a chain: filter on the attributes the request has, look up the missing ones, filter again.

The chain needs the current entries and mapping tables at request time. Serving those reads at the stream’s rate raises the infrastructure questions: whether the panel’s store takes the read load, whether a cache goes in front, how the data replicates across datacenters, and what each layer costs to run in every one.

The system divides into two parts: managing the entries and the mapping tables, mostly a baseline CMS, and running the matching stream on the data the CMS manages.

The latency ceiling is the constraint to solve first. Every system on the request path adds variability and an operating cost sized for the full stream in every datacenter. A compiled read model reduces the request path to the serving process. Matching runs in process against a model compiled from the entries and the mapping tables ahead of the request, so latency is decided by the lookup. The panel’s store serves one fetch per cycle, and the model reaches every datacenter as each process loads the same version.

Whether the pattern fits is a check against its requirements. Some have to hold, and the rest are costs to accept. Some costs are product concessions, and enough of them degrade the product even when each is tolerable alone.

Strong consistency at this read scale costs far more than an edit taking effect within minutes. In that window requests match against either version. The panel presents an edit as taking effect within minutes, and operators work on that expectation.

Model size restricts the hardware, and the bound appears in the product as caps on the data operators can create. The model grows with the operators and the sources, and a build and load designed around megabytes may need a redesign at gigabytes. A cap rises after the model fits at the new size.

The fetch, build, and publish cycle is a new module beside the panel and the matching service, and its cost grows with the model.

The deployed system has four components.

The panel is a CMS over a relational store. Operators edit their entries there, and an entry’s conditions reference mapping tables, held in the panel’s store or in linked external sources.

The compiler is a separate service on its own schedule. Each cycle it reads the entries and the mapping tables and compiles them into structures shaped for the matching chain: dictionaries for the key lookups, and an index from attribute value to the entries it matches for the filter stages. The result is written to object storage as one immutable versioned file.

Object storage replicates the file to every datacenter.

The serving processes in each datacenter poll object storage, load a new version into memory, and run the matching chain in process against it.

Related work

• Materialized views store a query’s precomputed result inside the database, refreshed on demand, on a schedule, or incrementally as the sources change. The result stays in the store, and reads reach it through the store’s query path. A compiled read model serves the result from the memory of the serving process.
• Netflix Hollow is documented as a “total high-density near cache”. A producer distributes an in-memory dataset to many consumers. Total means the entire dataset is on every node, so every read hits. High-density means encoding, bit-packing and deduplication shrink the footprint. Near means the data is in RAM on every instance that needs it. The density comes from encoding against the whole dataset, since deduplication and bit-packing choose their layout with every value visible.
• Baked data (Simon Willison, 2021) bundles a read-only copy of the data beside the application code, as part of the same deployment, typically as a SQLite file. The refresh unit is a redeploy, binding the model’s lifecycle to the code’s; a compiled read model refreshes on its own cycle, independent of deploys.
• Voldemort read-only stores bulk-load immutable datasets built offline. A Hadoop job repartitions the source per serving node and prebuilds the store files and their indexes, which are pushed out and swapped in atomically. Each node holds its own partition of the data.