Book
Designing Data-Intensive Applications
The Big Ideas Behind Reliable, Scalable, and Maintainable Systems
Martin Kleppmann
Summary
Martin Kleppmann's Designing Data-Intensive Applications maps the tradeoffs behind storing, querying, and moving data at scale: data models and query languages, storage engines, replication, partitioning, transactions, and the harder problems of consistency and consensus in distributed systems. It closes with batch and stream processing, tying operational and analytical data paths together. Widely used as a reference for engineers designing systems that must scale reliably, it favors first-principles explanations over any single technology stack, so the concepts hold even as the underlying tools change.
Target Readers
- Backend and infrastructure engineers choosing between consistency models, replication strategies, or partitioning schemes for a system in production
- Engineers preparing for system design interviews who want the reasoning behind tradeoffs, not just a list of technologies
- Developers moving from single-database applications to distributed data systems and encountering consistency or partitioning problems for the first time
Tags
Colophon
- Publisher
- O'Reilly Japan
- ISBN
- 978-4873118703
- Published
- Jul 2019
- List price
- ¥5,060incl. taxMay differ from the actual selling price on Amazon
Get this book
* The link above is an advertisement via Amazon Associates.Related Books
Prerequisites
- Prerequisite
Database Internals
A Deep Dive into How Distributed Data Systems Work
Alex Petrov
Reason: Only after understanding single-node storage-engine internals do discussions of consistency, replication, and partitioning across multiple nodes become grounded. Kleppmann's 'Designing Data-Intensive Applications' is the capstone that systematizes the principles of distributed data systems, placing internals knowledge as a required prerequisite.
- Recommended
Shippai kara Manabu RDB no Tadashii Arukikata (The Right Way to Work with RDBs, Learned from Failure)
Taketomo Sone
Reason: Having understood the design of a single relational database, you broaden your view to the big picture of large-scale systems involving multiple data stores. Designing Data-Intensive Applications systematizes the fundamentals of data systems—replication, partitioning, consistency, faults—without being swayed by passing trends.
- Recommended
Microservices Patterns
With examples in Java
Chris Richardson
Reason: Learning microservice implementation patterns, you inevitably hit the hard problem of keeping data consistent across services. Designing Data-Intensive Applications supplies the theory of distributed-data consistency, replication, and transactions, underpinning why those patterns work.
- Recommended
Understanding Distributed Systems, 2nd Edition
What every developer should know about large distributed applications
Roberto Vitillo
Reason: The author of Understanding Distributed Systems says himself that the book favors breadth over depth and may not suit readers with years of experience. Replication, partitioning, transactions and consistency models each get a chapter or a section. Designing Data-Intensive Applications spends a whole book on those subjects, and is easier to enter once the first book has given you the vocabulary and a sense of where each piece sits.
- Recommended
Designing Distributed Systems
Patterns and Paradigms for Scalable, Reliable Services
Brendan Burns
Reason: After learning structural 'patterns' of distributed systems from Burns's book, advance to the hard part beyond them—data consistency and fault tolerance. Kleppmann's 'Designing Data-Intensive Applications' digs theoretically into replication, distributed transactions, and consensus, providing the data-layer principles that underpin distributed patterns.
- Recommended
Building Microservices
Designing Fine-Grained Systems, 2nd Edition
Sam Newman
Reason: Once you split services, the next hard problem becomes the distribution, consistency, and reliability of data. DDIA systematically provides the foundational theory and trade-offs of distributed data systems—replication, partitioning, and transactions.
- Recommended
Software Architecture: The Hard Parts
Modern Trade-Off Analyses for Distributed Architectures
Neal Ford, Mark Richards, Pramod Sadalage, Zhamak Dehghani
Reason: When the hard parts of distributed architecture force decisions about decomposing data, you descend to the underlying theory of distributed data. The principles of consistency, replication, and storage engines become the grounds for your design decisions.
- Related
System Kaihatsu, Sasshin no Tame no Data Model Taizen (The Complete Book of Data Models for System Development and Renewal)
Kozo Watanabe
Reason: Once you master business modeling on a single RDB, you hit the question of how design changes when data becomes distributed and large-scale. Kleppmann's 'Designing Data-Intensive Applications' covers replication, partitioning, and consistency models across the board, serving as the starting point for extending modeling knowledge into distributed environments.
- Related
PostgreSQL Tettei Nyumon, 4th Edition (A Thorough Introduction to PostgreSQL)
From Installation to Features and Internals, Application Building, and Administration and Operations
Yuta Kondo, Yuta Masano, Kiyoshi Sakai, Tatsuhito Kasahara, Tatsuo Ishii, et al.
Reason: Once you understand operating and transactions on a single RDBMS (PostgreSQL), step into the distributed world of what breaks when you scale to multiple nodes. Kleppmann's 'Designing Data-Intensive Applications' covers replication and consistency trade-offs, bridging single-node operational knowledge into distributed design.
- Related
Data-Oriented Programming
Reduce software complexity
Yehonathan Sharvit
Reason: The mindset of treating data as immutable connects deeply with how distributed systems handle consistency and event history. Kleppmann's 'Designing Data-Intensive Applications' covers log-centric architectures and event sourcing, extending the immutable-data idea into the context of distributed infrastructure.
- Related
Patterns of Enterprise Application Architecture
Martin Fowler
Reason: After mastering enterprise patterns that assume a single database, you widen your view to consistency, replication, and partitioning of data spread across many nodes. The assumptions of scale change.
Next Books
- Recommended
Building Applications with AI Agents
Designing and Implementing Multiagent Systems
Michael Albada
Reason: Having grasped the theory of consistency, reliability, and scale in large-scale data systems, you see it as the foundation for putting AI agents into production. Practical AI Agent Development bridges the robustness principles learned in data-intensive systems to designing agent state management and tool integration.
- Related
Dragon Quest X o Sasaeru Gijutsu (The Technology That Supports Dragon Quest X)
Behind the Scenes of a Large-Scale Online RPG
Koji Aoyama
Reason: Having grasped the theory of data systems, you flesh it out with a concrete example of how it's applied in a real service. 'The Technology Behind Dragon Quest X' concretely describes the operation, scaling, and incident response of a large-scale online game, giving abstract principles a tangible feel.
- Related
Database Reliability Engineering
Designing and Operating Resilient Database Systems
Laine Campbell, Charity Majors
Reason: After understanding the theoretical trade-offs of distributed systems in Kleppmann's 'Designing Data-Intensive Applications', step into the reality of keeping them running. Based on Google's SRE principles, 'Database Reliability Engineering' (by Laine Campbell and Charity Majors) provides concrete practices for operating distributed data infrastructure reliably in production.