Link Search Menu Expand Document
The System Design Checklist
Table of contents
  1. Ensure data consistency requirements.
  2. Ensure data replication and partitioning requirements.
  3. Consider concurrency requirements.
  4. Consider decomposing a monolithic database into domain-driven schemas.
  5. Consider architecture patterns to improve scaling, security and performance.

Databases

Ensure data consistency requirements.

Transactions are logical units made of multiple read and write operations.

It must be in one of two states:

  1. Committed to the database in case it succeeds
  2. Rolled back in case of failure

OLTP vs OLAP

OLTP systems offer strong ACID guarantees.

  • Atomicity: all transactions succeed or are rolled-back
  • Consistency: after each transaction the system is structurally sound
  • Isolation: transactions don’t interfere with one another and appear to run sequentially
  • Durability: commited transactions are permanent even after system failures

By contrast, most Online Analytical Processing (OLAP) systems offer BASE guarantees.

  • Basic Availability: the databases work most of the time
  • Soft State: data stores don’t offer reading consistency across replicas
  • Eventual Consistency: at some point the data stores become consistent

Consistency vs scalability

ACID vs BASE is mostly about consistency vs scalability.

Consistency means if the data on the nodes is the same at any point in time.

  • Strong consistency guarantees that when we read data after we successfully complete a write we see the value just written.

  • Weak consistency means that when the data is replicated on multiple nodes, depending which node we read from, we might not see that value immediately.

  • Eventual consistency is a type of weak consistency which guarantees that if we wait a while, eventually the data on the nodes will converge to be the same.

We cannot guarantee both because of the CAP theorem.

The CAP Theorem

A distributed system can provide only two out of three guarantees:

  1. Consistency: every read receives the most recent write or an error
  2. Availability: every request receives a response
  3. Partition tolerance: the system operates even if messages are dropped between its nodes
  • Consider ACID transaction model when strong consistency is a key requirements
  • Consider BASE transaction model when high availability is more important than strong consistency

Ensure data replication and partitioning requirements.

  • Choose the partition key to avoid hot data that might prevent uniform load distribution

Consider concurrency requirements.

Optimistic vs pessimistic concurrency

  • Optimistic: before updating the data, the application checks if the cached data has changed since last retrieved. If it’s the same, the data is updated. If it’s not, the application needs to decide if/how to update it, either via automatic conflict resolution or asking the user to solve the conflict.
  • Pessimistic: upon data retrieval, the application locks the cache to prevent other instances from changing the same data.

Trade-offs

  • Optimistic concurrency is suitable for less frequent updates or ones with less likelihood of collisions.
  • Pessimistic concurrency is suitable for transactionality, e.g. if multiple items need to be updated at the same time in a consistent manner. However, locking impacts latency.

Conflict resolution

Conflict resolution is extremely tricky. Even companies like Amazon get it wrong, like in the case of a famous bug they had where users sometimes saw deleted items being added back to the shopping cart. Read more in this AWS whitepaper.

Examples where conflicts can happen:

  • apps with locally cached data on multiple user devices syncing state, e.g. calendars, reminders, notes etc.
  • collaborative software with several users concurrently modifying the same data, e.g. Google Docs, Notion, Miro etc.
  • distributed databases and caches with multiple replicas

When and how conflict strategies are applied:

  1. On write: a background process that detects conflicts in replicated databases and allows the application to execute a bit of code.
  2. On read: every time a conflict is detected, the data is stored as another version. When the data is read, all the versions are returned and the application (potentially through UX) is asked to solve them.

Automatic conflict resolution strategies:

  1. Conflict-free replicated datatypes (CRDTs) for sets, maps, lists, counters etc.
  2. Mergeable persistent data structures similar to Git version control with a three-way merge function
  3. Operational transformation algorithm used by Google Docs for concurrent editing an ordered list of items like a text document.
  • Ensure read-after-write consistency, e.g. if the user reloads the page they will see any updates they’ve submitted themselves
  • Ensure monotonic reads, e.g. if the user requests the same data multiple times in a row, they won’t see the data moving back in time because of stale updates
  • Consider optimistic (conflict resolution) vs pessimistic concurrency (locking), i.e. ensuring that updates made by one instance of your services don’t overwrite changes made by another

Consider decomposing a monolithic database into domain-driven schemas.

There are several problems with shared databases at scale:

  1. Prevents information hiding, making it unclear who uses what data across a monolithic application or a distributed monolith.
  2. Reading and writing directly to a shared database makes the database schema become a public-facing contract that can’t change without pain.
  3. Breaks the principles of Domain-Driven design, because microservices can’t totally encapsulate their own data storage and retrieval mechanisms.
  4. Having the same data controlled by multiple services leads to lack of business logic cohesion because behaviour and state can become inconsistent across services.

However, there are situations when a monolithic database makes sense, for example:

  • When it’s used as a read-only, static reference, i.e. a schema holding highly stable data structures such as post codes, country codes etc.
  • In a Database-as-a-Service pattern, where a service exposes a database as a read-only endpoint for multiple consumers. 💡 To find out more about this pattern, read Monolith to Microservices by Sam Newman.
  • Consider using a database view projecting a subset of an underlying schema, as the first step towards information hiding
  • Consider using a service to wrap around a database, to stop adding data to the underlying schema while decomposition is in progress
  • Consider using a database-as-a-service as an external, read-only endpoint to abstract away the internal database from consumers/clients

Consider architecture patterns to improve scaling, security and performance.

💡 I prefer Microsoft Azure’s collection of cloud design patterns explained in plain English.