Bitcoin, as any system of man, exhibits together both the highest ideals of utopia, and the lowest residual trash of society.

[Note: some names & sensitive details have been changed]

This is the story of how ShapeShift, a leading blockchain asset exchange platform, was betrayed. Not once, not twice, but three times in less than a month.

In total, nearly two-hundred thousand dollars in cryptocurrency was stolen by thieves within and without, not to mention the significant resources expended in its wake. Nevertheless, no customer funds were ever lost or at risk, a milestone for an industry pocked with past tragedy, and ShapeShift itself has adapted and rebuilt, humbled by the experience learned, and ever more resolute in its mission of safe, frictionless asset exchange.

In the spirit of Bitcoin's openness, we wanted to share this story with the community; may you be informed, entertained, reflective, and ever-diligent in your own affairs.

The Backstory

Since its inception in the Spring of 2014, ShapeShift has been an evolving creature. What began as a quick experimental way to swap between Bitcoin and Litecoin grew into an advanced engine for the effortless exchange of all major blockchain assets, each one into the other, with no user friction. No user accounts. No signup process. It is the Google Translate of cryptocurrency.

And we've always been playing catch-up. Trying to build at the speed of this industry, not only along the vertical of Bitcoin proper, but along the breadth of all crypto, is a challenge.

Last Fall, we realized the "minimum viable product" server architecture established originally for ShapeShift was insufficient. We needed a professional to join the small team, and craft a scalable, and secure, server apparatus upon which our technology could grow.

We hired such a person, and patted ourselves on the back for our proactive decision. On paper, he looked great; the reference we called confirmed his prior role and responsibility. He'd even been into Bitcoin since 2011/2012 and had built miners in his room. Awesome. We'll call this new employee Bob… indeed his real name starts with a B.

Over the next months, Bob built and managed ShapeShift's infrastructure. He did okay, nothing special, but we were content to have a professional taking care of devops at least well enough to enable our engineers to build upon the architecture.

In the first quarter of this year, as the market discovered what we already knew – that our world will be one of many blockchain assets each needing liquidity with the other – exchange volumes surged at ShapeShift. Ethereum was on the rise, specifically. Our infrastructure was not ready for the pace of growth. It was like riding a bicycle upon which jet engines suddenly appear full-thrust

Unfortunately, Bob did little to be helpful. He puttered around aimlessly while the team worked long hours to keep the ship together.

Scratch that, actually, Bob was not aimless.

He was preparing to steal from us.

The Genesis Betrayal

On the morning of March 14th, in the midst of one of our heaviest volume weeks ever, I get a call from our Head of Operations, Greg. "Erik, our hot wallet is missing 315 Bitcoin." Why did we have so much in a hot wallet, you ask? Well, with volumes surging, our hot wallet would be drained through normal business in an hour at that level, which then required constant manual rebalancing. Are there ways to automate and reduce that risk? Absolutely… but hindsight of one's development priorities is always 20/20.

So 315 Bitcoin was gone.

To those who have experienced such incidents, the feeling of sickness is profound. It's a deep, dismal state, that doesn't stop at the edge of financial loss, but permeates down to one's core. When systems are breached, systems that one has engineered and cared for deeply, obsessively, that violation of what one considers safe and secure is very, very uncomfortable. And then there's the loss itself. 315 Bitcoin… roughly $130,000. That's college tuition, part of a house, food for ten years… a couple months of payroll. It's a lot of money for a pre-profit startup.

I rushed to the office, hoping there was some mistake. The only comforting thought was that the loss was only our own money. With no customer accounts, neither customer funds nor personal information were at risk from the hack. That was by design from the beginning of ShapeShift; one of our tenets. But even if nobody nearby is harmed, a punch in the face still hurts like hell.

Myself, Greg, and our two lead engineers poured through logs and servers, trying frantically to figure out what had happened. The 315 BTC went to an unfamiliar Bitcoin address, and was sitting there.

Indeed, it sits there still: https://blockchain.info/address/1LchKFYxkugq3EPMoJJp5cvUyTyPMu1qBR

Despite our note to all employees to come into the office urgently, Bob, our head IT guy, the one responsible for security and infrastructure, arrives at 11:30am.

We ask Bob to join our discussion. We reveal the hack to him. We ask him if he had logged in at all that morning, to which he responded no (on several occasions). On the new of the theft, he seems neither particularly shocked nor outraged, yet it was his security that failed us. Immediately, he starts pointing to red herring explanations, "It must be one of the exchanges that got hacked, that happens all the time." Umm, our exchange accounts are fine, Bob.

"Well, look at the IP address, it happened somewhere off west Africa." Umm, IP addresses on block explorers indicate only the first node that noticed a transaction, and are generally meaningless in the context of Bitcoin, Bob. (What kind of Bitcoiner doesn't know that?)

Very quickly, we realize he is pretty much useless. Here we have our "server guy" and he has zero intelligent comments about a hack against his own infrastructure.

While pouring over logs we noticed, however, a couple SSH keys (belonging to Bob) that had logged into the breached server that morning an hour before the rogue transaction, and then logged off two minutes after. Not nefarious, necessarily, for indeed Bob's keys would be expected to log in periodically, though the timing was strange (6am-ish in the morning). We also discovered the breach occurred over the VPN, meaning someone in the office, or someone with access to our VPN, committed the theft.

We ask everyone with server access to provide the fingerprints of their SSH keys so we can start comparing them to logs. Everyone does so, but another strange thing: the fingerprint of the key handed in by Bob doesn't appear in any logs. It appears brand new. Strange that the key of the server admin would never have been seen on any server…

Soon after, Bob decides it's time for his lunch break, and we don't see him for an hour, during the worst incident in ShapeShift's history. We frankly didn't care that much, he wasn't helpful and suspicions were starting to creep in. He tells all of us that he's leaving his laptop open to download some logs, and makes sure we see that the laptop is left open. He's being a little weird.

Upon his return an hour later, he is sitting down with other engineers still investigating what occurred. I'm in the other room on a call. When I finish my call, I come check on the progress. Bob appears to receive a call "from his mother who needs to go to the hospital." He packs up his stuff, grabs his dog who was at the office, and heads out. We're all half relieved for his departure and half in awe… did our server admin really just leave for the second time during our investigation, which he should be leading?

He says, "I'll be back within an hour." This was at about 3pm, March 14.

We never saw him again

Shortly after he leaves, one of our engineers pulls myself and Greg aside, and says, "While you were on your call, we were all sitting around the table, and we saw in the logs that Bob deleted two SSH keys while he was sitting there with us, then he grep'd several times for them [a server command to find specific text], and then he left. Those two keys matched the two keys we saw in the log this morning which accessed the Bitcoin server just prior to the hack."

He just deleted his keys from the server?? Well fuck. Guns don't get any smokier than that.

We all immediately move to the assumption that Bob stole the funds. He is out of the building, and so we start locking everything down. All keys are changed in haste (well, almost all).

We work for a few more hours, no word from Bob. No calls, no texts, nothing. By the end of the day, it had been 3-4 hours since he left to "take his mother to the hospital." We decide to call him, without letting on our suspicions just yet.

"Hey Bob, where are you?."

"Oh hey, I just decided to go home."

"You're at home?"

"Yeah, just here, working on some stuff."

WTF?

That call is innocuous, but we recorded it. We also recorded the next one 30 mins later, in which we confront him with some of the evidence.