What Did Not Happen At Mt. Gox

The world's biggest Bitcoin exchange recently declared bankruptcy, with close to $400M in Bitcoins missing. There has been much talk about what may have happened at Mt.Gox, with speculations running wild. I want to quickly go over what did not happen at Mt. Gox, and how to avoid that which did happen in the future.

It's Not Transaction Malleability

Thief

The initial claim from Mt. Gox was that they lost money to a problem with Bitcoin known as "transaction malleability." Lots of people jumped on this explanation (for one of the better ones, see this one, though its depiction does not match what actually happened). I stayed away from this topic, because to even accept the format of the discussion would have been to lend credence to Mt. Gox's ludicrous claims about transaction malleability. But it was inescapable, and I started getting inquiries from Wall Street about what transaction malleability is and what it implies.

Essentially, transaction malleability refers to the fact that an attacker can ask Mt. Gox to transfer some Bitcoins, capture the transaction order Mt. Gox issues, and modify it in a way that causes the money transfer to take place yet confuses Mt Gox about whether or not it actually did. This confusion then enables the attacker to contact Mt Gox, claim that the transaction did not go through, and get issued a second payment, thereby stealing money. It's outright fraud and/or theft.

There are two obstacles to a successful malleability attack, one technical, and one social.

The technical obstacle is that the attacker has to either front-run Mt Gox or wait for it to slip up. Specifically, she has to capture a legitimate transaction issued by Mt. Gox, modify it, and have her modified transaction accepted by Bitcoin miners in lieu of the original one. Evidently, this technical feat is not hard to do, since we know that people have successfully launched transaction malleability attacks. Interestingly, this fact has a dire implication for Bitcoin that has nothing to do with malleability; namely, a successful malleability attack involves a technical step similar to what happens in selfish mining when the selfish miner goes tete-a-tete against a block found by honest miners, in that it involves a race in the peer-to-peer network. If it is indeed the case that an attacker can front-run a transaction so easily, then that suggests that a selfish miner can outdo the honest miners with high probability. As a co-author of selfish mining, I welcome this bit of news.

In the case of Mt. Gox, the attacker does not even have to front-run the exchange, because Mt. Gox seems to have guaranteed a win for the attacker by generating bad transactions. Specifically, Mt. Gox occasionally produces signed-but-ill-formed transactions that will never go through the Bitcoin system (like a bank check missing the date field), which gives someone else the opportunity to patch things up and make the transaction go through (by the equivalent of filling in the date field). But Mt. Gox then gets very confused about whether or not its check is accepted, because it is monitoring for checks that are identical to what it issued, with the missing date. No one knows how frequently Mt. Gox issued such bad transactions that an attacker could take advantage of, and no one knows how frequently an attacker could front-run Mt. Gox's good transactions. So there are some technical challenges here for the attacker to overcome.

But the real obstacle is that the attacker needs to get Mt. Gox to reissue the transaction that she modified. Some people seem to think that just modifying the transaction is sufficient to get paid twice. It isn't, because every Bitcoin transaction exhausts all of its inputs, so once the attacker's transaction goes through, Mt. Gox's original transaction will not. To be paid more than her fair share, the attacker has to socially engineer Mt. Gox to issue a brand new payment. This likely requires some contact with Mt. Gox, like an actual phone call. Doing this at scale requires a lot of contact. Doing it to the tune of $400 million dollars, and draining cold wallets, requires the equivalent of the attacker twerking the Mt. Gox support hotline. There is no practical way it can be done without raising eyebrows.

Sure enough, people who have analyzed malleable transactions have found only very modest evidence of malleability attacks. There has not been a comprehensive analysis of the 9 sources of malleability in Bitcoin, but a preliminary study shows that the volume of malleable transactions is very small, too small to account for $400M.

It's Not Lost Keys

Thief

Some people have speculated that Mt. Gox may have lost the keys to their wallets. There are two variants of this theory, the "dude, where are my keys" theory and the "bad php" theory. Neither seem likely.

The "dude, where are my keys" theory postulates that Mt Gox people realized one fine morning that they had digitally misplaced their keys, prohibiting them from accessing their Bitcoin wallets. This is the equivalent of "the dog ate my homework and your $400M" excuse, and is about as believable. People who have analyzed the blockchain have observed activity in accounts that are linked to Mt Gox (though not with 100% certainty), so that observation argues against this theory. But it's not surprising to hear this idea tossed around. Part of the Bitcoin creation myth is that Satoshi and friends have mined a huge number of the initial coins and promptly lost the keys to the fruits of their own labor. People parrot this line even though no one knows who Satoshi is, what he stands for, what his long-term plans and motivations are. So if we are to believe that the tooth fairy took Satoshi's keys, we can also believe that the easter bunny took Mt. Gox's.

The "bad php" theory is a bit more savvy. It postulates that Mt. Gox's code for computing Bitcoin addresses ended up transferring their cash to an account number to which they lack the private key. Apparently, Mt. Gox's CEO made some statements on an IRC channel to indicate that he lacks access to the money at this time. The theory is that they moved the money to an address for which they lack the key, or parts of the key, and therefore they are busy at work, having rented a huge datacenter trying to brute-force the parts of the key that they lack. It's like someone installed a new lock on their front door, and instead of pocketing the key that came with the new lock, placed some other key on their keychain, and now they're outside the door, metal file in hand, modifying the incorrect key and trying it repeatedly, hoping to break into their own apartment.

As ridiculous as it may sound, this kind of technical error is actually plausible. So I downloaded the most commonly used PHP library for elliptic curve cryptography, donned the tyvek suit and welding mask I use when approaching PHP, and looked into just how plausible this is.

There are certain domains where code is inherently full of complicated corner cases. Such code involves systems with huge, typically exponential, numbers of states, and correspondingly complex code paths with many conditionals. For instance, fault-tolerance in distributed data stores is one of these cases, and it remains a topic that certain kinds of software can never hope to conquer, and yes, I'm looking at you MongoDB.

But elliptic curve crypto is not one of these topics. If the code can generate a handful of Bitcoin addresses and corresponding keys correctly, there is hardly any reason why it cannot do so for all addresses and corresponding keys. My colleagues Nate Foster, Michael Greenberg and Benjamin Pierce argue that unit tests are big correctness theorems: elliptic curve key generation and Bitcoin transaction prep are very big theorems over a fairly uncomplicated code path, so if you can pass that bar for some random addresses, you can likely pass all bars of equivalent height for all addresses. This badly generated key theory seems quite implausible.

It's Not "Hackers"

Thief

The most recent claim by Mt. Gox is that its internal holdings were stolen by computer hackers.

When I was in middle-school, we'd get chided by a teacher for some misbehavior, like making a mess in the cafeteria. And our response was always the same, delivered with the crackling voice of a boy in the midst of puberty, mixed with the whine of a little kid: "we're not the ones making the meeeesssss, it's the kids from upper classsssseeeees." It was always the kids from upper classes. And in the days before cameras, constant surveillance, and zero tolerance, the teachers had to accept this ridiculous claim. But it was always us. Not once did a kid from an upper class come and make a mess.

This claim makes absolutely no sense for Mt Gox because they lost the coins in cold storage. "Cold storage" is a fancy way of saying that the keys, required to move the funds, are not online. Someone has to physically touch something to gain access -- a USB stick, a computer, and perhaps a safe. It takes a thief with a corporeal presence to do this.

It's Not the Web Server or Database

Thief

I'm actually shocked that Mt. Gox did not lose money to a database screwup. There are so many flawed NoSQL databases out there that, if you adopt the technologies advertised as "hip" on techcrunch, you'll most likely end up with a broken exchange (more on this in subsequent blog posts, because there are many funny examples that deserve their own discussion). It is quite easy for well-meaning developers to build an exchange on a database that loses transactions, or to restore their database from backups and find themselves in a state where the accounts don't match up. It's safe to guess that their losses were not due to their database, because if they had been, Mt. Gox would have played it up already.

As an aside, quite a few of the Bitcoin exchanges are, technologically speaking, one giant cluster-love-affair-without-feelings-involved. Coinbase and others tried to assure the public with a very nicely spun PR release, but the bottom line is that many of these exchanges have had recurrent problems. Historically, Wall Street has had a crisis every seven years, because the entire system is based on a broken premise. Again, objectively, Bitcoin exchanges have had a crisis every six months, because they use terrible databases whose code quality matches that of a masters project. I went to the Cornell dairy barn, I drank the fizzy milk drinks that the agriculture students designed for their masters projects. The code being peddled under the first-generation NoSQL rubric is analogous in its construction and quality. This broken-by-design technology is, amazingly, not yet implicated in Mt. Gox's demise. If I had to guess, I'd venture that Mt. Gox is based on something like MySQL -- strongly consistent and not broken by design, though perhaps slow and hard to scale.

It's maybe the US Government?

Thief

Some people claim that the "men in black took 97% of the cash." Perhaps. One would have to believe that they took the keys with the help of the Japanese government, that they seized the coins because of their involvement with Silk Road, that they served a gag order, and that the gag order is still in effect. This is, in theory, possible, but there is no evidence for it, except for a link to 9gag that the Mt. Gox CEO dropped in an IRC chat room. Get it? 9gag? We're punning on URLs now.

I'd continue the discussion, but I just got a brain aneurysm from the sheer amount of IRC stupidity coming from this CEO who held close to half a billion dollars. I knew that some companies employ developers not to write code but to create noise on social media. But the CEO? On IRC? And some people entrusted their life savings to this guy? Ok, so you may not have known that he had faced computer fraud charges, but it seems like everyone except me was on IRC with this fellow, ROFL'ing when Bitcoin was going to the moon. I have harbored doubts about the solvency of my upstate-NY-hicksville bank on occasion (specifically, when I tried to purchase Russian sovereign bonds, and their "investment advisor," an older gentleman, said "we don't know how to purchase those, but if you're looking for high yield bonds, I'd recommend GM." I pointed out that Russia has nuclear weapons and isn't going anywhere soon, except perhaps towards the warmer seas, and in contrast, his chosen junk-bond company manufactures steel death traps for humans. He said "well, you claim that Russia will be around forever, and some people would claim the same about GM." Couple of years later, Russia paid 14% on their bonds, GM needed a bailout). If I were to encounter my bank's CEO on IRC, I would not think "how cool and edgy, lemme get a screen cap, LOL," I'd put my money in a mattress and short the bank's stock.

In any case, there is not enough data to rule this explanation in or out. I am suspicious because it plays so well into Bitcoiners innate distrust of the government. Bitcoiners are deathly afraid of governments, fiat, and inflation, even though most of them have never seen two-digit interest rates in their entire lifetimes, don't hold much fiat, and are trying to minimize their tax-related societal obligations to the government. The blame-Obama explanation is too convenient, far too pandering to the masses. Can the Feds seize funds in Japan? Possibly. Can Mt. Gox be storing all of their cold storage keys in safe deposit boxes? Unlikely, but possibly. Can the Feds, combined with the Japanese police, seize the keys from those safe deposit boxes? Plausibly. Can they decrypt them? Not easily, but let's just say they can. Can they go beyond the funds implicated in the Silk Road and seize almost all the funds at Mt. Gox? Yes on the Feds side, unknown on the Japanese side. Can they place an indefinite and binding gag order that no one would dare violate? Maybe. Can they sit on the coins without charging Mt. Gox, even after charging the Dread Pirate Roberts and his money-laundering accomplices? Perhaps. But multiply out those probabilities and we're exploring an unlikely scenario. Add the fact that Mt. Gox initially blamed transaction malleability, it becomes even less likely. If Mt. Gox is solvent, but had its coins seized, it could simply publish its Bitcoin addresses which held the cash, to show that they indeed have some funds. Since this did not happen, I'm inclined to not lend too much credence to this scenario.

Where Does That Leave Us?

Thief

Human history is full of people who were entrusted with valuables, who then absconded with them. Whenever anyone is in a position of trust, whenever the illegal gains to be obtained from breaking that trust exceed the value of one's reputation, there will be a temptation to steal. Jail is not quite a deterrent in this case, where the jurisdiction is Japan and the technology is too new for the justice system. Chances are that this is a simple case of theft, involving at least one insider.

And it's important to not get too carried away with reading the tea leaves in the blockchain when performing forensic analysis. For instance, some people believe that Mt. Gox transferred $75M three months after they were sued for $75M for breach of contract. Let me go on a limb and claim that in the long and fabled history of lawsuits, not a single one for $75M was settled within 3 months for $75M. That's just not how the justice system works. You sue for X, with the expectation that you'll get less than X, and the other side, even if they are completely wrong, has tons of legal tricks to slow down the process. The blockchain contains many transactions, and the laws of small worlds imply that one can always find a short path between a wallet and a transaction of the desired size. These connections are not always meaningful.

Overall, Bitcoin has been an ongoing massive online course on economics and distributed systems for the libertarian masses. It's ironic that Mt. Gox turned into a chapter on fractional reserve banking.

How Do We Avoid The Next Mt. Gox?

Thief

So, this is where I could score easy points with the crowd by delivering some well-accepted "wisdom" such as "do not risk more than you can afford to lose." I'll do no such thing; I hate repeating content-free advice. I took stupid risks when I was younger. I'd get seriously depressed if young people were not taking equally stupid risks these days. So I don't expect that anyone's wise words will have any impact, and as well they should not.

I could also repeat the party line which goes "do not trust your wallet to an exchange." I won't say this, either. The people who parrot this line know full well that a normal person is incapable of running their own wallet. And it's not even good advice: your Macbook or Android phone is not a secure device. The Bitcoin demographic is 98% male, between the ages of 20-30. I base this on official online polls, confirmed by pictures of the Bitcoin Christmas party at Bitcoin HQ in NYC, which somehow managed to look more depressing than my high-school dances. This demographic uses their laptops for viewing highly questionable content and their phones for installing flapping bird software from Jimmy Bob's Software Inc. These are the same people who have started but not yet finished reading The Fountainhead, the same people who grant permissions to make phone calls, activate cameras and send SMSs to phone apps whose sole function is to act as a flashlight. Running a wallet is the last thing they should be trusted with.

My Personal Approach

Thief

I wanted to end on a note of cautious optimism, similar to those visionary futurists from the 70s who were all full of hope for mankind, extrapolating from their experience after a decade of awesome parties. They expected a future filled with tech wonders, strawberries the size of watermelons, and carefree love with strangers. But it's just not possible. Cryptocurrencies are here to stay, but future systems will look nothing like the currency systems we have today. Before Bitcoin, we had Karma, and before Karma, we had millicent, with plenty of others before and in between. There will be others.

What Nigerian scams are to your grandfather, Bitcoin exchanges are to the 20-30 semi-tech-savvy libertarian demographic. Even if the Bitcoin protocol were perfect, and it isn't, our computing infrastructure is not up to the task of handling high-value transactions. The exchanges are built on the latest hyped technologies that have incredibly poor guarantees, and routinely run into technical problems. They require full trust for their operation and are open to attacks from insiders and out. In a world where secret agents are hopping across machines and networks, keeping coins safe in a computer is a losing battle. Even if you keep everything in cold storage, laptops and phones can be infected with malware that steals coins when they come out of cold storage.

Then there is the increasingly disconcerting social side. I've always steered my own life towards doing fun and interesting things with smart people in happy and positive communities. Bitcoin, at the moment, is in a slump, with a community that has become its own parody. While the underlying cryptocurrency is quite interesting and the wallet software is fairly good, the exchanges are based on layers upon layers of bad software, run by shady characters. The Bitcoin masses, judging by their behavior on forums, have no actual interest in science, technology or even objective reality when it interferes with their market position. They believe that holding a Bitcoin somehow makes them an active participant in a bold new future, even as they passively get fleeced in the bolder current present. And as if the world does not have enough schmucks with Macbooks who call themselves entrepreneurs, we have the term "Bitcoin entrepreneurs" used unironically by mainstream media. The community has designated a Nobel leaurate as its nemesis, solely because he asked some inevitable questions every thinking person in his profession ought to ask. As far as I'm concerned, the only winning move is to not play this game. Sure, you may make money, perhaps lots of it, on the inevitable ups and down that are sure to come, but you'll be associating with the wrong kinds of people. If your life goals did not include some amount of pride and self-respect for you and your community, there were tons of other, easier ways of making money fast that you could have taken.

The only "positive" news about Mt. Gox is that money lost at Gox is apparently tax deductible. This is so wrong on so many different levels that I don't know where to begin.

If one must pick a cryptocurrency, the lowly dogecoin, of all things, is doing everything right. It's based on economic principles that provide the right incentives for a healthy economy. The community does not take itself seriously. Most importantly, no one pretends that Doge is an investment vehicle, a slayer of Wall Street, or the next Segway. No one would be stupid enough to store their life savings in Dogecoins. And people freely share the shiba goodness by tipping others with Doge. So, young people who are excited about cryptocurrencies and want to get involved: Dogecoin is where the action is at. Much community. So wow.

Share on Linkedin
Share on Reddit
comments powered by Disqus