designing data intensive applications the big ideas behind reliable

Two ways of implementing those operations:

Home timeline: a user can view tweets posted by the people they follow (300k req/sec).

Post tweet: a user can publish a new message to their followers (4.6k req/sec, over 12k req/sec peak).

We need to succinctly describe the current load on the system only then we can discuss growth questions. This is how do we cope with increased load.

Implement good management practices and training.

Set up detailed and clear monitoring, such as performance metrics and error rates ( telemetry).

Quick and easy recovery from human error, fast to rollback configuration changes, roll out new code gradually and tools to recompute data.

Provide fully featured non-production sandbox environments where people can explore and experiment safely.

Minimising the opportunities for error, peg: with admin interfaces that make easy to do the 'right thing' and discourage the 'wrong thing'.

Configuration errors by operators are a leading cause of outages. Software errors are a systematic error within the system, they tend to cause many more system failures than uncorrelated hardware faults. It is unlikely that a large number of hardware components will fail at the same time. A system that tolerates machine failure can be patched one node at a time, without downtime of the entire system ( rolling upgrade). There is a move towards systems that tolerate the loss of entire machines. As data volumes increase, more applications use a larger number of machines, proportionally increasing the rate of hardware faults. Until recently redundancy of hardware components was sufficient for most applications. You should generally prefer tolerating faults over preventing faults. Systems that anticipate faults and can cope with them are called fault-tolerant or resilient.Ī fault is usually defined as one component of the system deviating from its spec, whereas failure is when the system as a whole stops providing the required service to the user. Application performs the function the user expected.To work correctly even in the face of adversity. Periodically crunch data ( batch processing) Send a message to another process asynchronously ( stream processing)

Reliable, scalable, and maintainable applicationsĪ data-intensive application is typically built from standard building blocks.

Data structures that power up your database.Reliable, scalable, and maintainable applications.

Designing data intensive applications the big ideas behind reliable