The idea was to build my own EBS-backed Gentoo AMI on which I could run docker with a btrfs backend. So putting down my experience here, of making docker work on a Gentoo AMI (Amazon machine image). The Gentoo AMI should be able to run as a PV-HVM (para-virtualized hardware virtual machine) on AWS, with enhanced networking (SRIOV) enabled on the virtual machine.
Let me first break down the steps to achieve the final goal of docker+Gentoo+AWS. The process involves three stages:
- Create an instance-store backed Gentoo AMI, which can run as PV-HVM and has enhanced networking enabled.
- Upload instance-store backed Gentoo AMI to AWS S3 and convert it into an EBS-backed AMI.
- Install docker on the Gentoo AMI EBS-backed instance, and run docker with btrfs backend.
We can do part of step 3 along with step 1, i.e., we can create a Gentoo AMI with docker pre-installed, but the only drawback of creating such an AMI is that we won’t have a generic Gentoo AMI that we might want to use for a different purpose at a later date. If we specifically want a docker-installed Gentoo AMI we can always create one from a snapshot after step 3. So, without further adieu lets get started with the steps.
Step 1: Creating an Instance store Gentoo AMI:
There are quite a few blogs, such as this, which give a very thorough description of creating Gentoo AMI. However, these blogs primarily tell us how to create a PV (para-virtualized) AMI. A main constraint with the PV AMI is that you cannot use your own custom kernel with these AMI, which is a requirement for installing docker. The two alternatives to PV AMI, for running custom kernels, is to run the instance as PV-GRUB AMI, or a PV-HVM AMI. You can read about the different types of virtual machines that AWS allows you to run here .
PV-GRUB simply uses a specialized Xen-grub to boot your custom kernel, but still uses para-virtualized drivers with Xen-aware (domU) kernel. PV-HVM allows an unmodified kernel (wihtout Xen domU), but with para-virtualized drivers for components that are not hardware virtualized (disks for example). Amazon is moving all its AMI to HVM or PV-HVM because of the obvious benefits of hardware virtualization (SRIOV, virtualized GPU and CPU using Intel-VT) and hence we will also aim to create a PV-HVM AMI.
While most of the instructions listed below are similar to the ones you would use to create a PV AMI, you have to jump through a few more hoops (primarily installing your own grub) to make a PV-HVM work. I couldn’t readily find the additional instructions to get a PV-HVM to work at any one single place (especially for Gentoo) so thought will try and consolidate them here. Lets get started….
Creating the virtual disk
- Create a virtual loop-back device. Since we are going to create an instance-store backed AMI from this device, the image will be store in Amazon S3. Amzon S3 has a limit of 10G on uploads so we can’t have a device greater then 10G. Create a 10G loop back device as :
dd if=/dev/zero of=gentoo_aws.fs bs=1M count=10000
- Mount the loop back devcie :
losetup /dev/loop0 gentoo_aws.fs
- Now partition the loop-back device using fdisk or partd. Create three partitions on the disk. The first partition of size 1G can be the boot partition, the second partition can be a swap partition of 512M and the 3rd partition will be our root partition of 8.5G. Once you are done partitioning the disk you should see the partitions on /dev/loop0 as follows:
Disk /dev/loop0: 10 GiB, 10737418240 bytes, 20971520 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0xe9fba698 Device Boot Start End Blocks Id System /dev/loop0p1 * 2048 2099199 1048576 83 Linux /dev/loop0p2 2099200 3147775 524288 82 Linux swap / Solaris /dev/loop0p3 3147776 20971519 8911872 83 Linux
- Using kpartx, create a a device for each partition under /dev/mapper as follows:
$ kpartx -a /dev/loop0
$ ls /dev/mapper/
control loop0p1 loop0p2 loop0p3
- Create an ext4 partition on loop0p1 and loop0p3 using mkfs.ext4 and swap on /dev/loop0p2
$ mkfs.ext4 /dev/loop0p1 $ mkfs.ext4 /dev/loop0p3 $ mkswap /dev/loop0p2 $ swapon /dev/loop0p2
- Mount loop0p3 to /mnt/gentoo as follows:
mkdir /mnt/gentoo mount /dev/mapper/loop0p3 /mnt/gentoo
- We are ready to install gentoo on /dev/mapper/loop0p3. You can “chroot” into /mnt/gentoo and follow the Gentoo Handbook to create a fresh gentoo install .
- A note on grub-bootloader: While you can follow all the steps of the Gentoo Handbook to install the Gentoo file-system and the kernel, skip the step for installing the boot-loader. We will install the boot-loader separately in /dev/loop0p1.
- A note on kernel configuration:
- While configuring and compiling the Gentoo kernel make sure to compile the intel ixgbevf driver as a module. You can do this by running “make menuconfig” and selecting Device Drivers->Network device support->Ethernet driver support->Intel(R) 10GbE PCI Express Virtual Function Ethernet support . This is required for enabling enhanced networking on the PV-HVM. Also add the ixgbevf module to be loaded at startup in /etc/conf.d/modules
- Turn on Xen para-virtualized drivers for using the Xen front-end for Xen’s para-virtual disk drivers. This link provides a nice description of the options that need to be turned on. You can skip the network device driver, guest human interface, and ACPI section since we will be using SRIOV for networking, and won’t be needing any special devices or ACPI on virtual machine.
- A note on user-management: Make sure to create a user which you will use to access your AMI at a later date. You can also pre-install public keys in the users account to gain PKI-based ssh access to your AWS instance, based on this AMI.
- A note on allowing AMI to install AWS-keys at bootup: Rich0’s gentoo blog gives a nice way to install AWS keys, that you select during instance creation. Follow step 21 of Rich0’s gentoo blog to allow instances created of this AMI to use AWS keys instead of pre-installed keys.
- We are now ready to install grub2 on /dev/mapper/loop0p1 . To install grub2 on a loopback device, you first need to create a device-map as follows:
echo "(hd0) /dev/loop0" > /tmp/device.map
- mount /dev/mapper/loop0p1 on /mnt/boot
mount /dev/mapper/loop0p1 /mnt/boot
- To install grub2 on /dev/mapper/loop0p1 run the following command
grub2-install --no-floppy
--grub-mkdevicemap=/tmp/device.map
--modules=" part_msdos ext2 "
--root-directory=/mnt/boot
/dev/loop0
- This should have installed grub2 on the /dev/loop0 MBR .
- NOTE:, its important to include the modules part_msdos and ext2 for making grub2 read the ext4 boot partition and to allow grub to read an MBR partition table.
- Now go to /mnt/boot/boot/grub and create a grub.cfg and add the following:
timeout=0
menuentry 'Gentoo' {
root=hd0,3
linux /boot/vmlinuz-3.14.14-gentoo root=/dev/xvda3 console=ttyS0
}
NOTE: Replace vmlinuz-3.14.14-gentoo with your kernel. This is assuming you are keeping your kernel image in a directory called boot in the /dev/loop0p3 disk.
At this point our disk is ready to be booted as an instance-store backed AMI PV-HVM. Now lets create the AMI image, bundle it, upload it to Amazon S3 and register the AMI.
Step 2: Upload Gentoo instance-store AMI and convert it to EBS-backed AMI
In order to upload and register the Gentoo AMI created in step 1, we need to have an AWS account, and setup access keys to run the EC2 AMI tools and EC2 CLI tools. You can find the instructions for installing and using EC2 AMI tools here. You can find the instructions for installing and using EC2 CLI tools here.
Creating, uploading and registering the AMI bundle
- Create the AMI bundle as follows:
ec2-bundle-image --image gentoo_aws.fs --prefix base_gentoo --cert <AWS X509 cert> --privatekey <AWS private key> --user <AWS account number> --destination out/ --arch x86_64 --ec2cert /etc/ec2/amitools/cert-ec2.pem
- Upload the AMI bundle as follows:
ec2-upload-bundle --manifest out/<XML manifest file> --bucket <S3 bucket> --access-key <AWS access key> --secret-key <AWS secret key> --region us-west-2
The <XML manifest file> is the manifest created in the previous step. Before uploading the image you will need to setup an S3 bucket before uploading the image.
- Register the AMI image:
ec2-register <full path of manifest in S3> --name <image name> --description <verbose description> --architecture x86_64 --region us-west-2 --virtualization-type hvm --sriov simple
Pay special attention to the –virtualization-type option and the –sriov option. These two options tell AWS to launch the instance a PV-HVM and enable enhanced networking on these instances. At this point we should be able to launch images on AWS.
Converting instance-store PV-HVM to EBS-backed store PV-HVM
- Create an instance of the AMI image from the previous steps:
- A note instance types: to use PV-HVM with enhanced networking you will have to launch the instance as a C3 or C4 instance type. Instances that support enhanced networking are given here.
- A note AWS keys: As mentioned in STEP 1, the keys could be either pre-installed when you were creating the virtual disk for the AMI or you can install a script (step 21 in Rich0’s gentoo blog) to install the AWS keys selected during instance creation.
- Make sure that the instance store is able to launch by looking at the system log of the AWS console.
- Once the instance has started and you are able to login to the instance, create an EBS volume as follows:
ec2-create-volume -s <volume size> -z <availability zone> --region <your region>
While creating the EBS volume make sure to keep the volume size >= to virtual disk size of your instance store. The cool thing about EBS is that you can grow this volume at a later stage, so you need not be overtly concerned about creating too small a volume size. At the same time you need to be aware that you are paying for each gigabyte of EBS storage so you don’t want to create too large a volume either.
- Note the volume ID created in the last step. Attach the volume to your instance:
ec2-attach-volume <volume ID> -i <instalce ID> -d <device name (/dev/sdg)> --region <your region>
The volume should show up on your instance as block device with a name such as xvdg . NOTE: The block device name you see in your instance might not be the same as what you gave when you attached the volume to your instance. The device name within the instance is allocated by the kernel and not by AWS. You can check the block devices in your instance by running the lsblk command:
~ # lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 50G 0 disk
└─xvda1 202:1 0 50G 0 part /
xvdg 202:96 0 50G 0 disk
- Create an ext4 file system on this volume
mkfs.ext4 /dev/xvdg
- Mount the volume to /mnt/ebs
- Copy current root file system to /mnt/ebs as follows
rsync -avHx / /mnt/ebs - Now we need to install grub2 in the ebs volume. Create a folder called /boot/ and run the following command
grub2-install --no-floppy --modules=" part_msdos ext2 " --root-directory=/mnt/ebs/boot /dev/xvdg
- Finally, copy a kernel image (from /mnt/ebs/usr/src/linux/arch/x86/boot/) to /mnt/ebs/boot and create the following grub.cfg under /mnt/ebs/boot/grub :
timeout=0 menuentry 'Gentoo' { root=hd0,1 linux /boot/<kernel image> root=/dev/xvda1 console=ttyS0 }
Replace the kernel image and note the root device name given. There are critical for the system to boot.
- Take a snapshot of the EBS volume from your instance-store backed VM. This can be done from AWS console WEBUI
- Once the snapshot has been stored, got the snapshot and using the AWS console WEBUI opt to create an EBS-backed image from this snapshot. While creating the image make sure to choose the virtualization type as “HVM”.
- “Once the image has been created, launch a C4/C3 instance using this image.
- Once the instance has been launched you want to enable enhanced networking on this instance for this you want to set the SRIOV attribute of the instance:
ec2-modify-instance-attribute <instance ID> --sriov simple - At this point we have a EBS-backed Gentoo PV-HVM instance running on ehanced networking (SRIOV)
Now lets get the docker (btrfs backend) working.
Step 3: Install docker with btrfs backend
- Go through the installation of docker on Gentoo as described here . I prefer using the official way (using emerge) . For using the btrfs backend you will have to explicitly turn on “btrfs” USE flag in /etc/portage/package.use for app-emulation/docker . Also make sure the btrfs support is turned on in your linux kernel.
- The installation will check whether the relevant kernel modules are present by looking at the .config in /usr/src/linux . If the installation fails complaining about missing kernel config, please turn on the correct kernel config and re-compile the kernel.
- Once you have successfully installed docker, go ahead and create a new EBS volume of a specified size and attach it to your instance:
ec2-create-volume -s <volume size> -z <availability zone> --region <your region>ec2-attach-volume <volume ID> -i <instalce ID> -d <device name (/dev/sdh)> --region <your region> - Make sure that the new volume comes up as block device using lsblk
- Lets assume the block device came up with the name /dev/xvdh . Create a btrfs file system on this new volume.
mkfs.btrfs /dev/xvdh
- Ensure that the btrfs file system has been installed correctly by running command
~ # btrfs filesystem show Label: none uuid: d581f67c-8bc3-439c-8157-6eaf94e11b0f Total devices 1 FS bytes used 8.34MiB devid 1 size 50.00GiB used 3.04GiB path /dev/xvdg
- Mount the btrfs volume at /var/lib/docker
mount /dev/xvdg /var/lib/docker
- In /etc/conf.d/docker set the DOCKER_OPTS to use btrfs as the backend storage driver:
DOCKER_OPTS="--storage-driver=btrfs "
- Restart docker using /etc/init.d/docker restart
- Check /var/log/docker to verify that docker started successfully with btrfs as its backend.
At this point we have achieved our goal of docker (btrfs backend) + Gentoo + AWS (PV-HVM)
kerneldynamics 