The compiled read model

Context

A general-purpose architecture carries the cost of its generality; for a workload of a specific shape and scale, a design built on that workload’s properties outperforms it, often by a wide margin. The compiled read model is one such design, for reads of data that may be a cycle old at a rate and latency budget where request-time assembly is the dominant cost.

A conventional read over distributed data assembles its answer at request time from queries against data sources, and its latency is the sum of the hops: network round trips, storage retrieval, calls to other services. Lowering that latency means adding infrastructure. Caches sit in front of storage, replicas spread the load, retries and timeouts guard the network. Each layer reduces the cost of the layer below it, and each layer adds its own operational cost.

A compiled read model removes that infrastructure from the read. The assembly runs ahead of the request, on a schedule: data is fetched from the sources, compiled into the layout the reads need, and loaded as one unit into the memory of every serving process. At request time there is nothing left to fetch or wait for; a read runs against a local structure that already has the shape it needs.

Read performance becomes an algorithmic property. Latency is decided by the lookup, the number of memory accesses and how local the layout is. Data that takes several round trips across several systems on the conventional path takes a single lookup in a data structure.

Workflow

Fetch. The current data is read from the same sources a conventional read path queries at request time.

Build. The data is compiled into the model, joined, derived, and indexed into the layout its reads need.

Publish. The finished model is written to object storage as a new immutable version.

Update. Each serving process loads the new version.

Read. Each read runs in process against the loaded model.

Traits

• Linear read scaling. Each process holds the whole model, making it a complete read path in itself. Read capacity scales by process count alone; the serving fleet is the only tier that grows with read traffic.
• Eventually consistent. The model is built from a snapshot of the data sources. Each cycle rebuilds it from a newer snapshot, so the data a read serves is at most one cycle old. Freshness is set by the cycle length.
• Algorithmic reads. A read is solved in memory, an algorithm over data structures built to fit it. The structures take whatever shape serves the workload, the read logic carries as much of the work as the query demands, and the split between what the build precomputes and what the read computes is a design choice per workload.
• Immutable and versioned. A published model never changes; change arrives only as the next version, loaded whole. An answer traces to the version that served it.
• Narrow dependencies. The model store is the single external dependency, and access to it is cyclic: one load per cycle, off the request path.

Requirements

• Consistency tolerance. Reads serve data up to one cycle old, and during an update processes swap versions independently, so concurrent reads can answer from different versions. The workload has to tolerate both.
• Memory capacity. The process must hold the whole model in memory, with room left for its own working set and for the update mechanism’s cost. Memory bandwidth is a limit too: at extreme read rates, reads together can ask for more data per second than the machine’s memory can deliver.
• Model growth. Keeping the model in memory as the data grows is design work; the layout and the read logic evolve to keep the model in memory at the new size.
• Memory management. Each update allocates and reclaims memory at model scale in a live serving process, a recurring cost whatever the mechanism. The choice of mechanism is open; it decides where the cost lands, and the process needs headroom there.
• Longer startup. A serving process is ready only after loading the model, so startup lengthens by the load time. Readiness checks must reflect that.
• Backward compatibility. The read code and the model schema deploy separately, so a process can hold a model one schema version apart from its code. A schema change ships with read code that handles both versions until the transition completes.
• Complexity. The fetch, build, and publish cycle is a new module in the platform. Building and operating it is standing overhead a compiled read model adds.

Example

The system to build is a matching service. Each request in a stream gets one answer: which of the configured entries apply to it. Requests arrive at hundreds of thousands per second into several datacenters, matching runs in each of them, and each answer has a latency ceiling. The entries come from a control panel, where hundreds of operators edit their own entries on their own schedule.

Each entry holds conditions on request attributes. Some attributes are on the request; others have to be looked up in mapping tables in another system. Matching is a chain: filter on the attributes the request has, look up the missing ones, filter again.

The chain needs the current entries and mapping tables at request time, and serving those reads at the stream’s rate raises the infrastructure questions: whether the panel’s store takes the read load, whether a cache goes in front, how the data replicates across datacenters, and what each layer costs to run in every one.

The system divides into two parts: managing the entries and the mapping tables, mostly a baseline CMS, and running the matching stream on the data the CMS manages.

The latency ceiling is the constraint to solve first. Every system on the request path adds variability and an operating cost sized for the full stream in every datacenter. A compiled read model reduces the request path to the serving process. Matching runs in process against a model compiled from the entries and the mapping tables ahead of the request, so latency is decided by the lookup. The panel’s store serves one fetch per cycle, the model reaches every datacenter as each process loads the same version, and read capacity scales by process count alone.

Whether the pattern fits is a check against its requirements: the preconditions have to hold, and the rest are costs to accept. Some costs are product concessions, and enough of them degrade the product even when each is tolerable alone.

Strong consistency gets expensive at this read scale; accepting an edit taking effect within minutes costs far less, and in that window requests can match against either version. The delay becomes part of the product: the panel presents an edit as taking effect within minutes, and operators work on that expectation.

The entries and the mapping tables have to fit in the memory of every serving process, with headroom for the working set and for the update, which allocates and reclaims memory at model scale each cycle while the process holds its latency ceiling. Model size restricts the hardware, and the bound surfaces in the product as caps on the data operators can create, sized to what the model holds. The model grows with the operators and the sources, and a build and load designed around megabytes may need a redesign at gigabytes; raising a cap is that work on a schedule: the model fits at the new size first, and the cap moves after.

The matching code and the model schema deploy separately, so a schema change ships as a transition, with matching code handling both versions until it completes. The fetch, build, and publish cycle is a new component beside the panel and the matching service, and its cost grows with the model: fitting memory, the update’s headroom, and the schema transitions all get harder at a larger size.

The deployed system has four components, one per stage of the workflow.

The panel is a CMS over a relational store. Operators edit their entries there, and an entry’s conditions reference mapping tables, held in the panel’s store or in linked external sources.

The compiler is a separate service on its own schedule. Each cycle it reads the entries and the mapping tables, compiles them into structures shaped for the matching chain, dictionaries for the key lookups and an index from attribute value to the entries it matches for the filter stages, and writes the result to object storage as one immutable versioned file.

Object storage replicates the file to every datacenter.

The serving processes in each datacenter poll object storage, load a new version into memory, and run the matching chain in process against it.

Related work

• Materialized views store a query’s precomputed result inside the database, refreshed on demand, on a schedule, or incrementally as the sources change. The result stays in the store, and reads reach it through the store’s query path; a compiled read model serves the result from the memory of the serving process.
• Netflix Hollow describes itself as a “total high-density near cache”: a producer disseminates an in-memory dataset to many consumers; total, the entire dataset on every node with no eviction and no misses; high-density, encoding, bit-packing and deduplication shrinking the footprint; near, in RAM on every instance that needs the data. The density comes from encoding against the whole dataset: deduplication and bit-packing choose their layout with every value visible.
• Baked data (Simon Willison, 2021) bundles a read-only copy of the data beside the application code, as part of the same deployment, typically as a SQLite file. The refresh unit is a redeploy, binding the model’s lifecycle to the code’s; a compiled read model refreshes on its own cycle, independent of deploys.
• Voldemort read-only stores bulk-load immutable datasets built offline: a Hadoop job repartitions the source per serving node and prebuilds the store files and their indexes, which are pushed out and swapped in atomically; each node holds its own partition of the data.