You Might Be A Data Radical

Alex Popescu disagrees with my post on the distinction between partition-obliviousness and partition-tolerance, and brings up an interesting discussion:

The post presents a very dogmatic and radical perspective on what the requirements of both applications and distributed databases must be. I cannot agree with most of it if only for the reason it’s using the “bank example”.

Dogmatic radicals of the data management world, they might prevail! Or not! I wouldn't know because I'm not one of them.

While I'll be the first to admit that I use colorful language and metaphors, my perspectives typically come straight out of Distributed Systems 101. Half the reason why I started this blog is that there is a big discrepancy between what people who have studied distributed systems know, and what is being said in the marketing materials of various vendors. And every time I say anything that is slightly different from the orthodox academic view, my colleagues jump on my case, like they did when I suggested that examples and tutorials could form part of a specification (Oh the heresy! I can see that I will never get a named chair at Cornell with that kind of talk. Keep on talking like this, and I'm destined for a lifetime of university committee work!). So call me anything, but a "radical" is just way off.

Take the bank example. It is mentioned within the first 10 pages of every single database book I know of. It's part of our common DNA. To pretend that it does not exist, or that the NoSQL space is so different that we should forget what we know in terms of simple applications, seems odd.

Now, Alex has a very good point, even though he's not as explicit about it as I would like; namely, many first-generation NoSQL stores are not meant for, and should never be deployed in any application that is trying to maintain a high-level service invariant in the presence of partitions. Why? Because their weak properties render them wholly unsuitable. Does that mean that NoSQL vendors are clearly admitting what Alex is saying? No, even Alex isn't explicit about the point he's conceding. Does it mean that Alex's message is well-understood in the developer community and people are avoiding such deployments? Sadly, not at all. Here's a whole lot of word soup, for instance, on how to use Couchbase to simulate "transactions". I'll leave it up to the readers to point out the different ways that the suggested pattern will fail, but the bottom line is that people are actively mis-deploying NoSQL solutions. The bank example is useful because everyone understands how it ought to work, and can see how Banko Dynamo will run into problems when it's partitioned.

And to be clear, I provided some other applications as well. One was Ticketmaster. The other was EBay. Yet another was data analytics.

Perhaps some view NoSQL as a niche solution that is forever limited to non-demanding applications. I disagree with this and believe that NoSQL has the potential to supplant RDBMSs and capture the bulk of the database market. The "daddy knows best" attitude that RDBMSs bring to data management, the way they strip all information that the developer had about her data and force her to write a declarative specification of what she wants, only to try to then come up with an efficient evaluation plan in the query optimizer, all reflect a klunky aesthetic to system design that the lean and mean NoSQL movement can and will supplant. NoSQL is to RDBMSs what Unix was to Multics. But if we are to see this happen in our lifetimes, we should not hesitate to think big. Why shouldn't it be possible for NoSQL to tackle the big boys and serve as the sole data store for Ticketmaster or EBay or Bank of America or Goldman? Sure, one cannot do this with Mongo, but there are alternatives.

And even when we look at other, easier applications, we see the same problems; they just take a lot longer to set up and describe because they're not part of our DNA. For example, take a user discussion forum, like reddit. Processing a user post typically involves many lookups and multiple writes. Everyone realizes that there is one write to the subreddit and a corresponding update to the user's summary page. One might claim that, perhaps, it's quite ok for the user summary page to not be in perfect agreement with the discussion forum if the reddit cluster is partitioned. But what about duplicate URL detection? Post ID generation? Spam detection? The problem is that most people don't know how reddit checks for dupes or performs spam detection, or how it would do the same when it gets moved to NoSQL, so the examples get bloated. The bank example isolates the same issues without a complicated back story.

It should be clear that almost every non-trivial application will have some high-level invariants to maintain through partitions. And a partition-oblivious data store, while it may itself operate through a partition, makes it difficult to build partition-tolerant applications on top.

If you're a CTO and you use a partition-oblivious data store, you might need his assistance for a good night's sleep.

And the real trouble with partition obliviousness is that it happens silently. If I'm running data analytics, and my NoSQL data store decides in the middle of an 8-hour run that some nodes are unreachable, do I even get to find out that I miscomputed my analysis? If I want to know when my computation is not computing the right results, am I a radical? By that logic, are you, the dear reader, not a radical?

Let's take a benign scenario, one that most decidedly does not involve a bank or transactions: my team is currently working with a startup that has a very very very large data corpus. Their application issues a large number of queries and searches to this database. A partition-oblivious data store would return incomplete results, something that cannot be tolerated in this application. I can't imagine how their CTO could sleep well at night if a transient network failure during a long run could cause some GETs to fail silently.

Alex says:

NoSQL databases provide the knobs to tune the availability and consistency to the levels required by many applications. Applications can define more fine grained knobs on top of that.

One can always use various settings to improve a lobotomy.

This is certainly true. In the limit, what you get from a first-gen NoSQL vendor is a data management library, and you can then use all sorts of settings to tweak its behavior. A system like Mongo has more settings than it has modules. This makes it difficult to pin down a discussion, and enables the salesmen to be even more mercurial than they normally would be. It's also true that computers are general-purpose Turing machines, and a careful engineer can take a NoSQL engine, no matter how scrappy, and tweak it (with the aid of settings, or in the limit, by patching it) until it more or less does what she needs. The problem is that these discussions tend to get fuzzy. Good engineers work with a well-understood substrate and they build guarantees on top; they do not handwave. Which is partly why I picked on Dynamo as the prototypical example; it has been published and documented, and therefore acts as a common, non-moving point of reference.

As an aside, there are too many NoSQL systems, and too many settings for each, for me to be "fair" to each individual data store. Some Dynamo-clones, like Riak, are designed by careful engineers who thought deeply about their technological choices, and are sold by people who are careful to specify their limitations -- I suspect that they have happy users because their users are not surprised by sudden, catastrophic let downs of overblown expectations. And by no means is Riak alone, so if you're a good guy and I neglect to mention your system by name, sorry, the space is too crowded to do a thorough job. But the default in the industry seems to be to have documentation lying (pun intended) all around the web, with claims a mile high and the nudge-nudge wink-wink implications piled even higher. Yet the system implementations fall far short. Ask any developer if he understood that what is billed as a partition-tolerant data store is exactly what would keep him from building a partition-tolerant application, and we can then have a discussion over the responses.

In my experience, when a system cannot definitively claim "we provide you with a guarantee on X," instead does a complex song and dance around the topic of X, and requires you to understand its internal implementation to correctly implement what you need, it usually fails at X altogether.

In this case, X is partition management. And I think there's value in being first to point out the difference between the phrase used by salesmen to sell their system (partition tolerance), and the proper term for what the resulting applications end up experiencing (partition-obliviousness).