The pitfalls of hosting in the cloud – or the fall of codespaces.com

I  get into debates frequently about the pros and cons of hosting in the cloud vs having your own environment co-located in a datacentre.

My argument generally revolves around the idea that these things come and go in cycles and we can benefit from a hybrid solution since there are benefits to each. One cycle pushes the power of computing to mainframes with multi-tenancy, the next moves processing power closer to the end-user with desktops and local network file servers. Another cycle still, takes us back to client-server architectures and centralizes processing at the server, yet another exposes web services through APIs.

Well, this week comes an example of how to do cloud hosting wrong, and what can happen to your company as a result (hint: it doesn’t end well).

On Tuesday, Code Spaces (a code hosting provider) replaced their homepage with this:

www.codespaces.com

 

If you don’t want to read the whole thing, it goes something like this:

A hacker put codespaces out of business after he logged onto their EC2 panel and started deleting all EBS snapshots, S3 buckets, all AMI’s, some EBS instances and several machine instances.

There’s a bit more to it, but in a nutshell they had no offline
backups of anything, and as a result lost EVERYTHING. Now they’re out of business.

Now, if you’ve ever been in any type of sysadmin role, you’ll understand how traumatizing the above two paragraphs are. This is the digital equivalent of going into a village, and indiscriminately murdering everyone – men, women and children. It’s senseless, it’s barbaric and it’s just plain wrong.

That said, Code Spaces failed to apply some basic common sense to their infrastructure and as a result, I put the blame squarely on them for this.

The crux of the problem is that they had NO offline backups that were off-limit to the EC2 control panel. As a result, even their DR strategy was within the attacker’s reach – in other words, as good as useless.

They could’ve prevented this disaster – which cost them their business – by following any of these compensating controls:

  • Have offline backups that are inaccessible from AWS. Remember, offsite means nothing if it’s not also offline!
  • Have a DR solution that isn’t administered by the AWS control panel
  • Contact Amazon support to freeze the compromised account
  • Use two-factor authentication

I know for fact that many IaaS users follow the same model as these guys. They simply haven’t gotten popped yet.