Postgres on an Encrypted EBS Volume

There are only two kinds of people, those who encrypt and those who wish they encrypted. The way EC2 instances boot, it is almost impossible to do full disk encryption. Here we will store the postgres database files on a separate encrypted EBS volume, and thus tweak the default postgres installation along the way.

Starting an EC2 Instance

Security Groups

In a previous post, we have seen how to get started with Docker on AWS Elastic Beanstalk. Here we will go quickly through the steps of creating a security group, starting an EC2 instance inside it and ssh into the instance.

Let's create an SSH Key Pair to log into our EC2 instances. I recommend to generate two set of key pairs, one for customer facing machines and one for sandbox / staging / development machines.

(Unless specified otherwise, all commands are run on the local developer machine.)

# It seems Amazon only supports 1024 bits RSA PEM at the time of writing.
# (update: it looks like 2048 bits are also supported now)
$ ssh-keygen -q -f ~/.ssh/ec2-prod-key -b 1024 -t rsa

We verify the email address is a valid email account in the generated .pub file and edit the file to fix it if it isn't. Once done, we import the public key to AWS. This is the key that will be used to SSH into our EC2 instances.

$ aws ec2 import-key-pair --key-name ec2-prod-key \
    --public-key-material file://~/.ssh/ec2-prod-key.pub

AWS security is tight and by default will deny all access. We will need to create a security group and open port 22 (SSH) to connect to our EC2 instances.

$ aws ec2 create-security-group --description "front-end machines" \
     --group-name castle-gate
$ aws ec2 authorize-security-group-ingress --group-name castle-gate \
     --protocol tcp --port 22 --cidr 0.0.0.0/0

The security group is ready and we have a key that will be deployed on created EC2 instances.

The EC2 instance

First we need to choose an image to start from. Eagerly we first tried the command line tools to list available AMIs but the list is very (very) long. It is just better to search through the aws website, or better the ubuntu cloud images locator, or fedora in the cloud.

Here we will start an EC2 instance running Fedora 20 in the us-west-2 region.

# command line tools
$ aws ec2 run-instances \
    --security-group-ids castle-gate --key-name ec2-prod-key \
    --image-id ami-cc8de6fc
INSTANCE i-XXXXXXXX

The instance number will be useful later on. For now let's retrieve the public IP address of our instance and ssh into it.

$ aws ec2 describe-instances --instance-ids i-XXXXXXXX
INSTANCE  i-XXXXXXXX ... {'publicIp': 'PUBLIC_IP', ...
$ ssh -i ~/.ssh/ec2-prod-key fedora@PUBLIC_IP

Great, we are in! Let's move on to creating an encrypted EBS Volume.

Creating an Encrypted EBS Volume

Encrypting your data on Amazon EC2 is a good starting post. To mention, 13 Practical and Tactical Cloud Security Controls in EC2 is an abstraction level-up, a very good refresh on security in the AWS Cloud.

We always need an availability zone to create the EBS volume into but more importantly, we must create the volume in the same availability zone the instance will attach to is in. As long as the device is not already mounted, any name will do. We use /dev/sdf here.

$ ssh -i ~/.ssh/ec2-prod-key fedora@PUBLIC_IP ls -ld /dev | grep '/sd'

$ aws ec2 describe-availability-zones
AVAILABILITYZONE    us-west-2a  available
AVAILABILITYZONE    us-west-2b  available
AVAILABILITYZONE    us-west-2c  available
$ aws ec2 describe-instances --instance-ids i-XXXXXXXX
... us-west-2b ...
$ aws ec2 create-volume --availability-zone us-west-2b --size 20
VOLUME vol-XXXXXXXX
$ aws ec2 attach-volume \
    --instance-id i-XXXXXXXX \
    --volume-id vol-XXXXXXXX --device /dev/sdf

I am not sure why when we use the /dev/sdf device name, the EBS volume is mounted at /dev/xvdf. That seems to consistently be the case an x prefix is added.

From here on encrypting the volume is pretty much the same as we would do on our local Linux system.

(The following commands are run on the EC2 instance)

$ ls -ld /dev/* | grep 'df'
... /dev/xvdf
$ sudo yum update
$ sudo yum install cryptsetup
$ sudo cryptsetup -y luksFormat /dev/xvdf
Are you sure? (Type uppercase yes): YES
Enter passphrase:
sudo cryptsetup luksDump /dev/xvdf
...
Key Slot 0: ENABLED
...
$ sudo cryptsetup luksOpen /dev/xvdf my_enc_fs
$ sudo mkfs.ext4 -m 0 /dev/mapper/my_enc_fs
$ sudo mkdir -p /mnt/encvol
$ sudo mount /dev/mapper/my_enc_fs /mnt/encvol
$ diff -u prev /etc/grub.conf
...
- kernel /boot/vmlinuz ro root=UUID=uuid console=hvc0 LANG=en_US.UTF-8
+ kernel /boot/vmlinuz ro root=UUID=uuid console=hvc0 LANG=en_US.UTF-8 rd_NO_LUKS
...

Before we move any further, let's now run an "unscheduled" reboot and see what happens... After reboot, we ssh into the instance and re-mount the encrypted volume.

$ sudo shutdown -r 0

(Back running commands from the local machine)

$ aws ec2 describe-instances --instance-ids i-XXXXXXXX
... 'publicIp': 'PUBLIC_IP' ...
$ ssh -i ~/.ssh/ec2-prod-key fedora@PUBLIC_IP
$ sudo cryptsetup luksOpen /dev/xvdf my_enc_fs
Enter passphrase for /dev/xvdf:
$ sudo mount /dev/mapper/my_enc_fs /mnt/encvol

Everything looks good.

Setting up Postgres

After we install postgresql* on the EC2 instance, the main configuration steps involve SELinux and systemd scripts.

$ sudo yum install postgresql-server postgresql
$ sudo mkdir -p /mnt/encvol/var/db/pgsql/backups
$ sudo mkdir -p /mnt/encvol/var/db/pgsql/data
$ sudo chown -R postgres:postgres /mnt/encvol/var/db/pgsql
$ semanage fcontext -a -t postgresql_db_t "/mnt/encvol/var/db/pgsql(/.*)?"
$ sudo restorecon -R -v /mnt/encvol/var/db/pgsql

I believe it is a bug somewhere between initdb, /usr/bin/postgresql-setup and systemd but passing a non-default PGDATA does not always propagate correctly. The only commands that worked were to create a separate systemd script where we override PGDATA in the environment and run postgresql-setup on it.

$ cat /etc/systemd/system/sqldb.service
.include /usr/lib/systemd/system/postgresql.service

[Service]
# Location of database directory
Environment=PGDATA=/mnt/encvol/var/db/pgsql/data

$ sudo sh -c 'PGSETUP_DEBUG=1 /usr/bin/postgresql-setup initdb sqldb'

Not a great idea but since we run a lot of psql commands while debugging, we turn local authentication to trust.

$ sudo diff -u prev /mnt/encvol/var/db/pgsql/data/pg_hba.conf
 # "local" is for Unix domain socket connections only
-local   all         all                               peer
+local   all         all                               trust
$ sudo service sqldb start

We then create a database and connect to it.

$ sudo -u postgres createdb db_name
$ ls /mnt/encvol/var/db/pgsql/data
... new files have been created here as expected ...
$ ls /var/db/pgsql/data
... nothing appeared here as expected ...
$ sudo -u postgres psql \
    -c "create role role_name login password 'role_password';"
$ psql -U role_name -d db_name

Everything looks in order. We are ready to populate the database.

Detaching the EBS Volume

Since there will be a time where we are required to update the EC2 instance the volume is attached to, let's unmount the EBS volume cleanly and detach it from the EC2 instance - just to make sure it works.

(The following commands are run on the EC2 instance)

# First stop postgres database
$ sudo service sqldb stop
$ sudo umount /mnt/encvol
$ sudo cryptsetup luksClose my_enc_fs
$ exit

(Back running commands from the local machine)

$ aws ec2 detach-volume --volume-id vol-XXXXXXXX

$ aws ec2 stop-instances --instance-ids i-XXXXXXXX
$ aws ec2 create-snapshot --volume-id vol-XXXXXXXX
SNAPSHOT    snap-XXXXXXXX    vol-XXXXXXXX
$ aws ec2 detach-volume --volume-id vol-XXXXXXXX
$ aws ec2 create-volume --availability-zone us-west-2b --size 4 \
    --snapshot snap-XXXXXXXX
VOLUME    vol-XXXXXXXX    4    snap-XXXXXXXX    us-west-2b
$ aws ec2 attach-volume \
    --instance-id i-XXXXXXXX \
    --volume-id vol-XXXXXXXX --device /dev/sda1
$ aws ec2 start-instances --instance-ids i-XXXXXXXX
$ aws ec2 describe-instances --instance-ids i-XXXXXXXX
... 'publicIp': 'PUBLIC_IP' ...
$ ssh -i ~/.ssh/ec2-prod-key fedora@PUBLIC_IP
$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      4.0G  1.9G  2.1G  48% /