Bulletproof Encrypted Cloud Storage

While searching for secure online storage, you'll run across several issues with some providers opening documents, or having a history of using content to generate targeted ads, or outright selling data to advertisers or any other third party willing to pay. Add that to difficult or misleading access controls, or services designed to share data rather than keep it private, it's rather difficult to find a trustworthy cloud storage provider.

So to start with, lets come up with some goals...

1. Data should be encrypted at the provider
2. The provider shouldn't have access to the encryption keys
3. Data needs to be protected in transit

The best solution I've found is to run a virtual machine instance on the cloud provider of your choice. Have the virtual machine expose a block device on the network, this will be the storage used for the data, so make sure to allocate enough for your needs. Restrict access to the network to your client machine. Attach the block device to your client machine, and apply an appropriate disk encryption layer to the block device. This makes sure the data is encrypted, and also keeps the encryption keys away from the provider.

For this case I'm going to use Amazon Web Services EC2. The EC2 instance can run for $3-4 a month, so most of the cost will go for the storage. Now we need to chose a network storage system to share the storage drive. I'm going to use NBD, it's pretty easy to setup, and the 3.x branch has built-in ssl/tls support which we will need later. We could also use iSCSI, to be more platform independent for our client server, but I mostly run Linux so NBD works fine. Unfortunately Amazon's Linux distribution doesn't have the latest nbd package, so we have to install that manually. We also need some SSL/TLS keys, I like the OpenSSL Command-Line HOWTO at madboa.com as a guide.

Amazon EC2 Setup

• # Install dependencies
• yum install gcc glib2-devel gnutls-devel
• # Download the source since amazon has an old 2.x version
• wget https://sourceforge.net/projects/nbd/files/nbd/3.15.3/nbd-3.15.3.tar.xz
• tar xJf nbd-3.15.3.tar.xz
• cd nbd-3.15.3
• # Source dance
• ./configure
• make
• make install
• # Create the config file (see below for contents)
• vi /usr/local/etc/nbd-server/config
• # Create a symlink for the nbd-server so it shows up in $PATH
• ln -s /usr/local/bin/nbd-server /usr/bin/nbd-server
• # Create the init script
• vi /etc/init.d/nbd-server
• vi /etc/sysconfig/nbd-server
• chmod +x /etc/init.d/nbd-server
• # Start the service
• /etc/init.d/nbd-server start
• # Open the firewall for the service
• iptables -A INPUT -p tcp -s 1.1.1.1 --dport 10809 -j ACCEPT

/usr/local/etc/nbd-server/config

• [generic]
  • allowlist = true
  • cacertfile = /usr/local/etc/nbd-server/cacert.pem
  • certfile = /usr/local/etc/nbd-server/cert.pem
  • force_tls = true
  • keyfile = /usr/local/etc/nbd-server/key.pem
  • port = 10809
  • tlsonly = true
• [default]
  • exportname = /dev/xvdf
  • authfile = /usr/local/etc/nbd-server/allow

/etc/sysconfig/nbd-server

• # Command line options for nbd-server
• OPTIONS="-C /usr/local/etc/nbd-server/config -l /usr/local/etc/nbd-server/allow"

/etc/init.d/nbd-server

• #!/bin/bash
• #
• # nbd-server This shell script takes care of starting and stopping
• # nbd (NBD daemon).
• #
• # chkconfig: - 58 74
• # description: nbd-server is the NBD daemon. \
• # The Network Block Device server (NBD) is used to provide block device \
• # access for remote clients.
•  
• ### BEGIN INIT INFO
• # Provides: nbd-server
• # Required-Start: $network $local_fs $remote_fs
• # Required-Stop: $network $local_fs $remote_fs
• # Should-Start: $syslog $named
• # Should-Stop: $syslog $named
• # Short-Description: start and stop nbd-server
• # Description: nbd-server is the NBD daemon. The Network Time Protocol (NBD)
• # Description: nbd-server is the NBD daemon. The Network Block Device server \
• # (NBD) is used to provide block device access for remote clients.
• ### END INIT INFO
•  
• # Source function library.
• . /etc/init.d/functions
•  
• # Source networking configuration.
• . /etc/sysconfig/network
•  
• prog=nbd-server
• lockfile=/var/lock/subsys/nbd-server
•  
• start() {
  [ "$EUID" != "0" ] && exit 4 [ "$NETWORKING" = "no" ] && exit 1 [ -x /usr/local/bin/nbd-server ] || exit 5 [ -f /etc/sysconfig/nbd-server ] || exit 6 . /etc/sysconfig/nbd-server
  •  
  • # Start daemon.
  echo -n $"Starting $prog: " daemon $prog $OPTIONS RETVAL=$? echo [ $RETVAL -eq 0 ] && touch $lockfile return $RETVAL
}
•  
• stop() {
  [ "$EUID" != "0" ] && exit 4 echo -n $"Shutting down $prog: " killproc $prog RETVAL=$? echo [ $RETVAL -eq 0 ] && rm -f $lockfile return $RETVAL
}
•  
• # See how we were called.
case "$1" in start) start ;; stop) stop ;; status) status $prog ;; restart|force-reload) stop start ;; try-restart|condrestart) if status $prog > /dev/null; then stop start fi ;; reload) exit 3 ;; *) echo $"Usage: $0 {start|stop|status|restart|try-restart|force-reload}" exit 2 esac

Now on the the encryption details, we need to figure out how we want to encrypt the data being sent to the cloud. I'm going to use cryptsetup on a gentoo server to access the cloud storage. You will need to refer to your distribution for installing nbd on your client. Check for nbd, or nbd-client in your package manager. Also, It's important to make sure you have some network encryption between the two servers. nbd-3.x has built in ssl/tls support, but if you are planing on using a different system, make sure you have some ssl/tls or even ipsec encryption in place. Next we need to choose the encryption method. I've chosen to go with twofish in xts mode for the encryption, this is a common setup for local disk encryption. One important distinction is that while some encryption methods are fine for local storage, we are transferring data over untrusted channels rather than a local pci/sata bus. In this case xts has an issue with data manipulation in transit, which probably isn't a huge concern over a local pci/sata bus, but transferring data over the internet it becomes a major issue, which is why the ssl/tls support is so important. If you plan on choosing a different cipher, be sure to research it before hand. The other thing to note, is I'm not using LUKS. LUKS has a standard header defined which can be detected by anyone attempting to read the disk. It also stores the master key that's used to encrypt the drive in the header. The good news is the master key is also encrypted by the user password, but this would still be vulnerable to an offline attack by anyone who can read the header. Going with plain mode, the contents of the disk will appear more random as there is no header or format for the data. The bad news is this makes it less portable between systems. I mostly run linux based systems, so the portability isn't an issue.

Now that we have a chosen encryption method, and the tools installed, time to setup the drive on the client server. For this I'm using a simple script to manage everything. I recommend running these commands manually while setting things up to debug any issues.

/root/nbd-crypt.sh

• #!/bin/bash
• #
• # nbd-crypt - setup and tear down a nbd encrypted device
•  
• case "${1}" in
  • mount)
    • nbd-client -N default -certfile /etc/nbd/cert.pem -keyfile /etc/nbd/key.pem -cacertfile /etc/nbd/cacert.pem nbd.server.com 10809 /dev/nbd0
    • cryptsetup open --type plain --hash=sha512 --cipher=twofish-xts-plain64 /dev/nbd0 nbdcrypt
    • mount /dev/mapper/nbdcrypt /mnt/nbdcrypt
    • ;;
  • umount)
    • umount /mnt/nbdcrypt
    • cryptsetup close nbdcrypt
    • nbd-client -d /dev/nbd0
    • ;;
  • *)
    • echo "Usage: $0 {mount|umount}"
    • exit 1
• esac

I've been using this setup to backup my local git repositories for a while now and it's been working great.