CAP Theorem applied to Organizations

It’s well understood today that as you scale out internet platforms you can’t have perfect data consistency and high availability at the same time.

Like system architects that are now forced to adopt alternative eventually-consistent data stores, big organizations are facing a similar dilemma as the demands for information consistency, speed and scale square off. Increasingly, organizations are taking on distributed, high-decision-throughput formations, where control rests within small functionally-integrated teams – often spanning marketing, engineering and operations.

CAP Theorem
The CAP theorem was presented by Eric Brewer back in 2000 based on his findings while working at Inktomi in the late 90’s.  It’s a key tenet of the big platform architectures that emerged in the last decade, including Amazon’s Dynamo and Google’s MapReduce, and the discrediting of traditional hub-and-spoke architectures with a relational database at the center. CAP comes from:

  • Strong Consistency – every node has the same version of data at all times
  • High Availability – nodes allow read and write access to the data at all times
  • Partition-tolerance – the system still works when its divided into parts (as big systems must be)

The theorem states that you can only require two of the three properties in a system. For large systems and for large organizations alike, partitioning is inescapable. Big problems have to be broken down into small problems, large organizations into small teams. But once an organization is partitioned, you have the problem of communication between teams. When having the exact same information and shared goals is the absolute priority for an organization, there are generally two solutions:

1. Centralized decision-making (prone to bottlenecks)

2. Committee-based decision-making (prone to delays)

But when the requirement for strong alignment is relaxed, another approach becomes available:

3. Distributed, parallel decision-making with ongoing conflict reconciliation

While it’s true this option introduces the possibility of organizational disharmony, decisions aren’t stuck in committee or caught in hierarchical bottlenecks. This flexibility makes teams more productive and shortens time to market. Many organizations want to expand, go fast, and be consistent, but given the choice of only two, availability and partition-tolerance is the formula for success. Like the distributed systems that power the internet, these organizations expect decision conflicts, and have processes in place to resolve them as they emerge.