Thinking through some security concerns recently, I found myself wondering if it was possible to achieve full system Linux encryption in the cloud — running GNU/Linux off of an encrypted root partition (using LUKS). I thought that it should have been possible — it was achieved easily running with a local virtualization platform (VirtualBox, VMWare Fusion, etc.).
Since we have used Linode for a few projects, I figured that I would try to setup Linux encryption on Linode, but a quick Google search for “linux encryption on linode” didn’t turn up anything regarding root partition encryption with LUKS. I decided to try and figure it out myself. It turned out to be a bit of a challenge, but one which I’m glad I undertook as I learned a tremendous amount about Linux disk encryption, and how Linode manages the Linux boot process.
In order to achieve this, I consulted several very different resources, iterated several times on my setup process, and learned a great deal about GRUB configuration. Since I think others might find this information useful, I’ve compiled my setup process. The process assumes a working understanding of GNU/Linux, GNU utilities, and dm-crypt (cryptsetup).
I. Linode Preparation
1. Initial Decisions
I decided on the following system:
- OS: Ubuntu
- Version: 12.04 LTS
- Archictecture: 64-bit
While I choose this setup, certain variations (e.g. 32-bit, Ubtunu 12.10) should also work without too much extra effort. Just tweak parameters where necessary.
Linode boots custom systems (like the one I created) somewhat differently than your typical Linode system. Instead of having a single disk with multiple partitions, I needed to create multiple disks, each with the filesystem directly on the disk (no partitioning). This was necessary in order for the Linode GRUB loader to actually recognize my boot configuration.
For consistency, I always refer to my boot disk, root disk, and swap disks as my boot, root, and swap partitions. So for example, I refer to
/dev/xvdb as my root partition even thought it is just my primary disk mounted at
3. Disk Image Preparation
I created a new Linode with a three images of type “raw/unformatted”. The first was for the boot partition (
/boot), the second for the root partition (
/), and the third for the swap partition.
4. Configuration Preparation
I created a new Linode configuration which assigned the disk images that I just created to block devices. I decided to made my boot parition
/dev/xvda, my root partition
/dev/xvdb, and my swap partition
/dev/xvdc. Since I wanted to use a custom boot process and kernel, I needed to select
pv-grub-x86_64 as my “Kernel”. I set the “root device” to
/dev/xvda, as I intended to configure it as the boot device. I also disabled the “Xenify Distro” option. These specific details were provided by Linode in their Run a Distribution-Supplied Kernel with PV-GRUB how-to article.
II. System Preparation
1. Booting Into Rescue Mode
Linode allows you to enter a “Rescue Mode” that boots you into a Finnix Live CD environment with access to a Linode via Lish (the Linode Shell). This sort of access to a working Linux environment is necessary in order to prepare the encrypted root partition. I booted into the Finnix rescue environment, making sure that my disk image was associated with the same block device as in my Linode configuration.
2. Remote Lish Access
I was able to access Lish via SSH using the information on the “Remote Access” tab of the Linode manager. The section under “Console Access” has the SSH command you can run: “Lish via SSH”.
First I created a new encrypted partition (
luksFormat) using the
aes-xts-plain64 cipher, a key size of 512 bits (
-s 512), and hash algorithm of sha512 (
-h sha512) on my root partition (
/dev/xvdb). I entered my passphrase when requested (making sure that the passphrase was sufficiently secure). This sets up the encrypted disk.
Next, I needed to open the new encrypted partition so that I could actually access it (
luksOpen). This opened the encrypted partition and used device-mapper to map it as an unencrypted block device with a given name. I opened my root disk (
/dev/xvdb) and mapped it to
crypt-xvdb. This made the encrypted partition available to use as a normal block device at
root@hvc0:~# cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha512 /dev/xvdb WARNING! ======== This will overwrite data on /dev/xvdb irrevocably. Are you sure? (Type uppercase yes): YES Enter LUKS passphrase: somethingsecure Verify passphrase: somethingsecure root@hvc0:~# cryptsetup luksOpen /dev/xvdb crypt-xvdb Enter passphrase for /dev/xvdb: somethingsecure Enter passphrase for /dev/xvdb: somethingsecure
I formatted my boot partition as
ext2 and my root partition as
Note that I used the new mapped device (
/dev/mapper/crypt-xvdb) to create the filesystem for
root@hvc0:~# mkfs.ext2 /dev/xvda root@hvc0:~# mkfs.ext4 /dev/mapper/crypt-xvdb
III. System Setup
1. Installing Ubuntu
The Finnix Live CD comes with the
debootstrap utility which allows you to download and bootstrap minimal Debian-based systems (including Ubuntu). I mounted my new
ext4 formatted root partition to a folder, and started the bootstrap process. I choose a 64-bit architecture (
--arch amd64) and Ubuntu 12.04 LTS (
root@hvc0:~# mkdir newsystem root@hvc0:~# mount /dev/mapper/crypt-xvdb newsystem root@hvc0:~# debootstrap --arch amd64 precise newsystem/
deboostrap finished, I mounted several devices/filesystems which I would need in a chrooted environment (e.g.
/sys), and then chrooted:
root@hvc0:~# mount /dev/xvda newsystem/boot/ root@hvc0:~# mount -o bind /dev newsystem/dev root@hvc0:~# mount -o bind /dev/pts/ newsystem/dev/pts root@hvc0:~# mount -t proc /proc/ newsystem/proc/ root@hvc0:~# mount -t sysfs /sys newsystem/sys/ root@hvc0:~# chroot newsystem/ /bin/bash
3. Getting Started
After chrooting in my new system environment, I set the root password, and ran
apt-get to retrieve current package information and install a few packages that I needed (
cryptsetup) and wanted (
root@finnix:/# passwd root@finnix:/# apt-get update root@finnix:/# apt-get install cryptsetup vim
IV. System Configuration
1. Encrypted Swap
I decided that I also wanted to encrypt the swap partition. I did this using
cryptsetup (installed in previous step). This created a device mapping similar to the one for the root partition, but used basic (plain) dm-crypt instead of LUKS.
Instead of providing a passphrase on the command line, I specified a keyfile (
-d /dev/urandom). This, essentially, used a randomized key to encrypt the partition since swap didn’t need to persist across reboots.
Finally, I actually formatted the new encrypted partition for swap with
root@finnix:/# swapoff -a root@finnix:/# cryptsetup -d /dev/urandom create crypt-swap /dev/xvdc root@finnix:/# mkswap /dev/mapper/crypt-swap
crypt-swap, I received an error similar to:
modprobe: FATAL: Error inserting padlock_sha. According to Ubuntu Launchpad comments, the error message is benign and indicates that the kernel attempted to use a dedicated hardware accelerator which wasn’t present, so a software implementation was used instead.
2. Setting up /etc/crypttab
/etc/crypttab specifies any encrypted filesystems present on the system. I needed to populate this file so that the kernel would recognize encryption modules were necessary during boot, and to properly mount the encrypted filesystem at boot. For more details, see
The format for
- Target mapped device name (
- The source block device (The UUID for
- The keyfile to use (
nonecauses a passphrase prompt at boot)
- Options (
luksto enable LUKS extensions)
I included the following information in my
# crypt-xvdb UUID=5bd616e2-7caa-4648-8f8c-5a00b747cbd2 none luks crypt-swap /dev/xvdc /dev/urandom swap
Note that you can use
blkid to retrieve the UUID’s for block devices. In my
/etc/crypttab, I used the UUID for the target block device which holds my LUKS encrypted partition (
/dev/xvdb). Your source device should have a
crypto_LUKS when running
blkid. Note that I used a keyfile of
/dev/urandom for the swap partition as described earlier.
For more details, see
man crypttab and
3. Setting up /etc/fstab
/etc/fstab specifies filesystems to be mounted by the system (normal behavior for
/etc/fstab). The exception is that instead of specifying the normal block devices for the encrypted partitions, I had to specify the mapped devices. For example, I didn’t mount
/etc/xvdb, but at
I included the following information in my
/dev/xvda /boot ext2 defaults 0 2 /dev/mapper/crypt-xvdb / ext3 noatime,errors=remount-ro 0 1 /dev/mapper/crypt-swap none swap sw 0 0 proc /proc proc defaults 0 0
Following my configuration scheme from earlier, I set
/dev/xvda to be mounted at
/dev/xvdb mapped as
/dev/mapper/crypt-xvdb to be mounted at
/dev/xvdc mapped as
/dev/mapped/crypt-swap to be mounted as swap. The rest of the file (e.g.
/proc) and other options were fairly typical.
For more information, see
4. Setting up /etc/mtab
/etc/mtab usually lists currently listed filesystems. Because of the way that I entered my system environment (chrooting), this file was not properly populated (in fact, it didn’t exist). I used
/proc/mounts to populate this file as certain applications (e.g.
update-initramfs) make use of it.
root@finnix:/# cat /proc/mounts > /etc/mtab
rootfs on / type rootfs (rw) tmpfs on / type tmpfs (rw,relatime,size=804960k) /dev/xvdh on /cdrom type iso9660 (ro,relatime) /dev/loop0 on /FINNIX type squashfs (ro,relatime) unionfs on /UNIONFS type aufs (rw,relatime,si=1bbcee677d8e663e) none on /proc type proc (rw,relatime) none on /sys type sysfs (rw,relatime) tmpfs on /dev type tmpfs (rw,relatime,size=10240k,mode=755) devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000) devshm on /dev/shm type tmpfs (rw,relatime) /dev/mapper/crypt-xvdb on / type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered) /dev/xvda on /boot type ext2 (rw,relatime,errors=continue,user_xattr,acl) tmpfs on /dev type tmpfs (rw,relatime,size=10240k,mode=755) devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000) /proc on /proc type proc (rw,relatime) /sys on /sys type sysfs (rw,relatime)
5. Setting up hvc0 and /etc/init/hvc0.conf
hvc0 is the Xen hypervisor console. It allows access to the Linode via Lish, and is very important. On non-custom Linodes,
hvc0 is setup for you. However, on my custom Linode, I had to set it up manually.
Without properly configuring
hvc0, booting your Linode with an encrypted root partition will be impossible.
hvc0 access allows you to type in the LUKS passphrase that opens the encrypted partition. Once booted, a
hvc0 allows you to access your Linode via Lish since you do not have actual terminal (
I added the following Upstart script so that my system would maintain a
# hvc0 - getty # # This service maintains a getty on hvc0 from the point the system is # started until it is shut down again. start on stopped rc RUNLEVEL= stop on runlevel [!2345] respawn exec /sbin/getty -8 38400 hvc0
These specific details were provided by Linode in their Run a Distribution-Supplied Kernel with PV-GRUB how-to article.
V. System Boot
1. Installing GRUB and the Linux Kernel
Next, I actually went about installing GRUB (GRand Unified Bootloader) and the Linux kernel. While I could have installed these packages earlier, getting key configuration files finalized (i.e. the various
/etc/*tab files) prevents errors during the installation, and automates the otherwise tedious creation of several configuration files. Note that I installed GRUB legacy (version 1), not GRUB2. This is required for booting a custom system on Linode.
root@finnix:/# apt-get install grub linux-server
2. Configuring GRUB
Configuring GRUB was challenging due to uncertainty regarding the proper command line options for specifying the encrypted root partition. The Arch Linux Wiki had an excellent section on booting to a LUKS partition using GRUB Legacy which was instrumental in figuring everything out.
I generated the initial GRUB configuration (e.g.
/boot/grub/menu.lst) by running:
root@finnix:/# update-grub -y
-y prevents prompting for whether or not to create the
After the initial files were generated, I edited
/boot/grub/menu.lst to provide the necessary options for booting my system.
I changed the following lines from
## timeout sec # Set a timeout, in SEC seconds, before automatically booting the default entry # (normally the first entry defined). timeout 3
## timeout sec # Set a timeout, in SEC seconds, before automatically booting the default entry # (normally the first entry defined). timeout 10
in order to give me more time for catching the boot menu.
I changed the following lines from
## default grub root device # groot=(hd0,0)
## default grub root device # groot=(hd0)
in order to set the “root” device for GRUB to
/dev/xvda (instead of
I changed the following line from
## ## Start Default Options ## ## default kernel options ## default kernel options for automagic boot options # kopt=root=UUID=14fec84b-7054-4c49-aff4-40946c0bb61b ro
## ## Start Default Options ## ## default kernel options ## default kernel options for automagic boot options # kopt=root=/dev/mapper/crypt-xvdb cryptdevice=/dev/xvdb:crypt-xvdb console=hvc0 ro
in order to specify that the “root” device (the device holding the root partition) will be on
/dev/mapper/crypt-xvdb, and that this is a cryptdevice which is mapped from
console=hvc0 indicates the
hvc0 is the console device to use (as referenced earlier).
ro indicates that GRUB will treat the partition as read-only.
3. Finalizing GRUB Configuration
After making the above changes to
/boot/grub/menu.lst, I needed to re-run the
Next, I needed to do a little bit of file manipulation within
/boot. The Linode GRUB loader expects to find the
menu.lst file at
/boot/grub/menu.lst on the root device
hd0. This would be all fine and good if my
/boot directory wasn’t actually a mount-point to the root of
/dev/xvda. When the Linode GRUB loader looks at
/dev/xvda, it will only find a
/grub directory, not
To address this, I created a directory in
/boot/boot/grub and symlinked
root@finnix:/# cd /boot root@finnix:/# mkdir boot/ root@finnix:/# mv grub boot/ root@finnix:/# ln -nfs boot/grub grub
4. Updating Initrd
Before attempting to boot, I rebuilt the initrd image (initramfs) for my system. This is stored in
/boot, and serves as the initial root filesystem for the kernel (early user space).
The initrd image contains necessary helpers for device detection and module loading for the kernel. Updating the initrd image ensures that it contains the necessary kernel encryption modules so that the kernel can discover and open encrypted disks.
root@finnix:~# update-initramfs -c -v -k all
And now, approaching the moment of truth (the reboot), I backed out of my chroot and unmounted all of the filesystems that I had mounted in preparation:
root@finnix:~# exit root@hvc0:~# umount newsystem/proc/ root@hvc0:~# umount newsystem/dev/pts root@hvc0:~# umount newsystem/dev root@hvc0:~# umount newsystem/sys/ root@hvc0:~# umount newsystem/boot root@hvc0:~# umount newsystem
2. The Moment of Truth
Finally, I rebooted… and hoped that my system would boot the first time.
root@hvc0:~# shutdown -r now
But, of course, my system didn’t boot the first time. At least not like it should have. My initial attempts at getting the system running were hampered by several issues, including silly mistakes I had made. It took several attempts to resolve all of the issues, and finally get a fully running system.
3. Debugging GRUB
One particular process was tremendously helpful in getting the system to run: manually booting from the GRUB console. This allowed me to verify that my GRUB options were correct and see any verbose output from the process to help debug any issues (the default GRUB menu items use the
quiet option which supresses output).
For example, when the system would start to boot, I would get the message:
Press 'ESC' to enter the menu… at which point I would hit
ESC, and then immediately hit
c to drop to the GRUB console.
From there, I could try commands like the following to attempt a boot and see any errors:
grubdom> root (hd0) grubdom> kernel /vmlinuz-3.2.0-23-generic root=/dev/mapper/crypt-xvdb cryptdevice=/dev/xvdb:crypt-xvdb console=hvc0 ro grubdom> initrd /initrd.img-3.2.0-23-generic grubdom> boot
This helped me catch errors I made in specifying device names, as the GRUB process would usually indicate something helpful.