The latest HyperDex 1.6 release has a surprising feature: HyperDex can now act as a stand-in for MongoDB. HyperDex now supports a new Python interface, a Mongo veneer, that is API-compatible with its namesake. Many applications written for MongoDB can now work seamlessly with HyperDex.
Surprisingly, HyperDex-pretending-to-be-Mongo is 1-4X faster than Mongo itself. It's also strongly consistent, tolerates faults up to a user-specified threshold without data loss, and enables ACID transactions. It can be set up in under 15 minutes, and it simplifies dev-ops with seamless, cluster-wide, consistent backups.
Let's take a brief look at HyperDex's new interface.
Switching from Mongo to HyperDex is trivial. Suppose you have some application code that uses MongoDB:
>>> import pymongo # the old import
>>> client = pymongo.MongoClient('localhost', 27017)Switching to HyperDex involves simply replacing this snippet with the following:
>>> import hyperdex.mongo as pymongo
>>> client = pymongo.HyperDatabase('localhost', 1982)The rest of the code, like the snippet below, remains unaffected and continues to work:
>>> # Here's the code you don't change
>>> collection = client.db.profiles
>>> collection.insert({'_id': 'jane', 'name': 'Jane Doe', 'sessioncount': 1})
'jane'
>>> collection.update({'name': 'Jane Doe'}, {'$inc': {'sessioncount': 1}})
{'updatedExisting': True, u'nModified': 1, u'ok': 1, u'n': 0}Just as in MongoDB, there's no need to explicitly declare schemas or hyperspaces. The Mongo veneer automatically creates and manages the necessary data spaces, and translates MongoDB queries and updates to HyperDex calls.
Let's examine the document type that this veneer supports, and delve into the operations supported on documents. All of the examples in this post can be run interactively against a HyperDex shell using HyperDex's quickstart docker image. To follow along with this post, we can set up a cluster with these two commands:
$ docker pull hyperdex/quickstart
$ docker run --net=host -t -i hyperdex/quickstartWith the running cluster, we can then play with our temporary cluster from within Python using the same Docker image. Look at the output of the running cluster for a statement like, "You can connect to this cluster at address=172.17.0.15, port=1982." Use that address and port in the sample below:
$ docker run --net=host -t -i hyperdex/quickstart /usr/bin/python
>>> import hyperdex.mongo as pymongo
>>> client = pymongo.HyperDatabase('address from cluster', 1982)
>>> collection = client.db.profilesThe new release builds on HyperDex's existing documents. This datatype allows us to store, retrieve and operate on JSON documents directly. HyperDex imposes few limits on the structure of documents -- they can be deeply nested, contain any number of fields, and so forth; it requires only that they be valid JSON, not contain more than 200 levels of nesting, and not exceed 32MB. These limits are double what is supported by the latest release of MongoDB.
The HyperDex document API supports many of the operations and operators supported by MongoDB. The following code examples work identically under both MongoDB and HyperDex, and will produce the output shown. Many MongoDB tutorials can now be executed, word for word, on the HyperDex Mongo Veneer instead.
Of the features present in MongoDB, this latest release (1.6) supports operations like:
insert: Store a document into the database. For example, this will insert a document with the key "andy1":
>>> collection.insert({'_id': 'andy1', 'name': 'Andy', 'friends': []})
'andy1'save: Overwrite a document, changing its value, possibly inserting it if it does not exist. For example, this overwrites the Andy object with a new one, where his name is "Andy", and he has an Age of 17:
>>> collection.save({'_id': 'andy1', 'name': 'Andrew', 'age': 17, 'friends': []})
'andy1'find: This will find all documents that match a particular predicate. So we can find every user named Andrew:
>>> list(collection.find({'name': 'Andrew'}))
[{u'age': 17, u'_id': u'andy1', u'friends': [], u'name': u'Andrew'}]findOne: This will find at most one document that matches the predicate. So we can find any one user named Andrew:
>>> collection.find_one({'name': 'Andrew'})
{u'age': 17, u'_id': u'andy1', u'friends': [], u'name': u'Andrew'}update: This will update the documents that match a particular predicate. The predicate is the same form as the selection used for "find". For example, this sets value "is_andrew" to true for every user named Andrew:
>>> collection.update({'name': 'Andrew'}, {'$set': {'is_andrew': True}})
{'updatedExisting': True, u'nModified': 1, u'ok': 1, u'n': 1}
>>> collection.find_one({'name': 'Andrew'})
{u'age': 17, u'_id': u'andy1', u'friends': [], u'name': u'Andrew', 'is_andrew': 1}The new veneer supports many of the modification operators, such as:
"$set": This will set a field within a document to the specified value. For example, we can change Andrew's name back to Andy with this:
>>> collection.update({'_id': 'andy1'}, {'$set': {'name': 'Andy'}})
{'updatedExisting': True, u'nModified': 1, u'ok': 1, u'n': 1}
>>> collection.find_one({'name': 'Andy'})
{u'age': 17, u'_id': u'andy1', u'friends': [], u'name': u'Andy', 'is_andrew': 1}"$inc": Increment a particular field. If Andy acquires ten trinkets, we can atomically increment his trinket count. Since the field doesn't exist, the field will be created and set to 0 + 10, like this:
>>> collection.update({'name': 'Andy'}, {'$inc': {'trinkets': 10}})
{'updatedExisting': True, u'nModified': 1, u'ok': 1, u'n': 1}
>>> collection.find_one({'name': 'Andy'})
{u'age': 17, u'_id': u'andy1', u'friends': [], u'name': u'Andy',
u'is_andrew': 1, 'trinkets': 10}"$mul": Multiply the existing value by the provided value. For example, to add 1% interest to his existing value, we could multiply it by 1.01 like this:
>>> collection.update({'name': 'Andy'}, {'$mul': {'trinkets': 1.01}})
{'updatedExisting': True, u'nModified': 1, u'ok': 1, u'n': 1}
>>> collection.find_one({'name': 'Andy'})
{u'age': 17, u'_id': u'andy1', u'friends': [], u'name': u'Andy',
u'is_andrew': 1, 'trinkets': 10.1}"$div": Divide is very similar to the "$mul" operator. For example, we can reduce Andy's holdings of trinkets by half like so (MongoDB does not provide $div):
>>> collection.update({'name': 'Andy'}, {'$div': {'trinkets': 2.0}})
{'updatedExisting': True, u'nModified': 1, u'ok': 1, u'n': 1}
>>> collection.find_one({'name': 'Andy'})
{u'age': 17, u'_id': u'andy1', u'friends': [], u'name': u'Andy',
u'is_andrew': 1, 'trinkets': 5.05}"$push": The push operator appends to an array. We can add Buzz to the list of Andy's friends like this:
>>> collection.update({'name': 'Andy'}, {'$push': {'friends': 'Buzz'}})
{'updatedExisting': True, u'nModified': 1, u'ok': 1, u'n': 1}
>>> collection.find_one({'name': 'Andy'})
{u'age': 17, u'_id': u'andy1', u'friends': [u'Buzz'], u'name': u'Andy',
u'is_andrew': 1, 'trinkets': 5.05}"$rename": Move a field within a document. We can change the "is_andrew" field to "is_andy" like this:
>>> collection.update({'name': 'Andy'}, {'$rename': {'is_andrew': 'is_andy'}})
{'updatedExisting': True, u'nModified': 1, u'ok': 1, u'n': 1}
>>> collection.find_one({'name': 'Andy'})
{u'age': 17, u'_id': u'andy1', u'friends': [u'Buzz'], u'name': u'Andy',
u'is_andy': 1, 'trinkets': 5.05}"$bit": Manipulate fields in a bitwise fashion. For instance:
>>> collection.update({'name': 'Andy'}, {'$bit': {'perms': {'and': 488}}})
{'updatedExisting': True, u'nModified': 1, u'ok': 1, u'n': 1}
>>> collection.find_one({'name': 'Andy'})
{u'age': 17, u'_id': u'andy1', u'friends': [u'Buzz'], u'name': u'Andy',
u'is_andy': 1, 'trinkets': 5.05, 'perms': 0}
>>> collection.update({'name': 'Andy'}, {'$bit': {'perms': {'or': 055}}})
{'updatedExisting': True, u'nModified': 1, u'ok': 1, u'n': 1}
>>> collection.find_one({'name': 'Andy'})
{u'age': 17, u'_id': u'andy1', u'friends': [u'Buzz'], u'name': u'Andy',
u'is_andy': 1, 'trinkets': 5.05, 'perms': 45}HyperDex also supports many of the same predicates, such as:
"$eq": Check that the values are equal. Here we find everyone named Andy:
>>> list(collection.find({'name': {'$eq': 'Andy'}}))
[{u'trinkets': 5.05, u'name': u'Andy', u'is_andy': 1, u'age': 17,
u'perms': 45, u'_id': 'andy1', u'friends': [u'Buzz']}]"$lt", "$lte", "$gte", "$gt": Less than, lesser-equal, greater-equal, and greater-than. These operators will work to compare numbers like this:
>>> # Find people who are underage
>>> list(collection.find({'age': {'$lt': 18}}))
[{u'trinkets': 5.05, u'name': u'Andy', u'is_andy': 1, u'age': 17,
u'perms': 45, u'_id': 'andy1', u'friends': [u'Buzz']}]
>>> # Find people who qualify for AARP
>>> list(collection.find({'age': {'$gt': 65}}))
[]
>>> # Find people in the [25-35] demographic
>>> list(collection.find({'age': {'$gte': 25, '$lte': 35}}))
[]The surprising part of HyperDex's Mongo veneer is that the resulting system not only has stronger consistency and fault-tolerance guarantees as well as features such as ACID transactions over multiple keys, but it's also faster than Mongo.
Let's go through some microbenchmarks and compare the performance of the two data stores. These benchmarks explore both writing complete documents as well as modifying small subsets of a document in place. They were all executed on Amazon EC2 c3.2xlarge instances. In each experiment, the documents average 16KB in size with an average of 1,000 top-level fields. We use a single client to write the data, so that we may quantify the throughput a typical single-threaded application will see; in practice, many threads may run in parallel and performance will increase accordingly.
In our first deployment scenario, we'll look at a small two-node cluster configured to provide fault tolerance such that either node may fail without introducing data loss. With such a cluster, data will be written to each node before a write is considered complete in order to ensure that data cannot be lost. With HyperDex, these writes will always be strongly consistent, inline with HyperDex's strong consistency guarantees. For these benchmarks, the MongoDB client was configured to wait for a write to propagate to all replicas to ensure that writes can never be rolled back.
Loading into an empty database.
Bulk Loads To the right is the result of loading one hundred thousand documents, or 1.6GB, into both systems. Documents are inserted in sorted order by their identifier starting from an empty state of the database. We can see that HyperDex provides more than twice the throughput of MongoDB. In this same benchmark, the measured 99th percentile latency for HyperDex operations is just 2.6ms, while for MongoDB the same measurement is 5.6ms. The speedup is due partly to the optimized HyperLevelDB backend HyperDex uses to store its data.
Throughput to 'save' new documents in a populated database.
Random Writes This second benchmark, shown to the right, measures the rate at which new documents may be inserted in a random order. The documents have the same size as those inserted in the first benchmark, but have keys chosen at random, rather than in a sequential order. Again, we see that HyperDex provides twice the throughput of MongoDB on the same workload. Our optimizations to HyperLevelDB are robust across workloads, not special-cased for a specific insertion pattern.
Throughput of the '$inc' operator in an 'update' call.
Atomic Increment The third benchmark measures the rate at which a single field may be atomically incremented using the "$inc" operator. This benchmark differs from the first two in that the amount of traffic between the client and the server is much smaller than the whole document, and proportional to the single field to be mutated. As with the first two benchmarks, HyperDex provides more than twice the throughput of MongoDB.
Throughput of the '$set' operator in an 'update' call.
Set Operator A fourth benchmark is similar to the third, but uses the "$set" operator in place of the "$inc" operator. Similar to the third benchmark, this one uses small amounts of client-to-server communication to atomically mutate documents that reside on the server. In this benchmark, HyperDex provides approximately one and a half times the throughput of MongoDB.
Throughput of the 'remove' call.
Deletion The final benchmark measures the performance of removing documents one at a time by their primary key. In this benchmark, HyperDex provides more than four times the throughput of MongoDB.
To summarize, two clusters deployed with the same fault tolerance guarantees, and the same benchmark code, were able to provide a single client thread with the following throughput.
Performance summary of a single client on a fault-tolerant cluster.
Up to this point, we've only investigated the performance under a replicated deployment. Historically, MongoDB has incorporated optimizations that improve single-node performance, where concerns for data safety and fault tolerance across replicas are secondary to raw performance. For an example of such optimizations, see our article on why MongoDB is broken by design. These optimizations have the potential to improve MongoDB performance beyond the linear speedup expected by removing the replication component.
Indeed, this is exactly what we see when running both HyperDex and MongoDB in a single-server without fault tolerance or replication. The figure below shows the same workloads described above, but without any replication. Here, MongoDB's single-node performance optimizations can take effect and provide higher throughput. Even under such a scenario, HyperDex provides similar throughput to the MongoDB cluster on a variety of conditions. And it does so without weakening any consistency guarantees or special-casing for non-fault tolerant clusters.
Performance summary of a single client on a non-fault-tolerant, single-server.
Of course, applications can always get an additional performance boost by using the native HyperDex API instead of the Mongo veneer. But a 1-4X performance improvement without trying is a good start for most applications.
A common refrain in conversations about distributed systems is that "everything is a tradeoff." So it might seem a little hard to believe that a data store can be better than another one across every metric. Surely, people will say, there must be something that HyperDex is worse at. What is it that you're not telling us? So let's talk a little bit about this line of reasoning.
It's definitely the case that there are many tradeoffs in distributed system design, and that features often come at the expense of other features within the same code base.
But to claim that one system must always have some weakness compared to another is to assume an equivalence among implementations, and there is no sound basis for such an assumption. The same way quick sort is better than bubble sort, a second generation computer program can easily be, and often is, better than first-generation alternatives across the board.
That said, there are some things to keep in mind:
HyperDex is free and open source. It supports Debian, Ubuntu, Fedora, CentOS, Mac OSX, and docker with prepackaged binaries.
Users looking to quickly try out HyperDex can use the "hyperdex/quickstart" Docker image for experimentation. A simple one-node HyperDex cluster can be created with just these two commands:
$ docker pull hyperdex/quickstart
$ docker run --net=host -t -i hyperdex/quickstartFor production, HyperDex provides binary packages for Debian, Ubuntu, Fedora and CentOS, and source downloads for other platforms. A Mac OSX port for development is also available via Homebrew.
Upgrading from MongoDB to HyperDex brings several advantages:
PhD student at Cornell. His current research focuses on building scalable distributed systems with high integrity. more...
Hacker and Ph.D. student at Cornell, focusing on distributed systems and systems in general. Current projects include HyperDex, Replicant, and applications built on top. more...
Hacker and professor at Cornell, with interests that span distributed systems, OSes and networking. Current projects include HyperDex, OpenReplica and the Nexus OS. more...
bitcoin / security / ethereum / hyperdex / release / nosql / selfish-mining / blocksize / dao / surveillance / privacy / mongo / broken / weaver / nsa / meta / leveldb / blockchain / 51% / voting / smart contracts / graph stores / decentralization / bitcoin-ng / vaults / snowden / satoshi / philosophy / mt. gox / mining pools