Sharing an EFS Filesystem Across ECS Instances, Services, and Tasks

In application hosting scenarios, it is common to have data that must be available to all application instances. Often, this data is stored in a database (e.g. AWS RDS), or some other centralized repository (e.g. AWS S3). However, situations arise where third-party software expects data to be on the filesystem, or where application instances generate files which must be immediately available to all other application instances.

In these situations, a shared filesystem may be the easiest (or perhaps only) way to ensure multiple application instances have simultaneous access to the same data. The situation becomes more complex when the application instances may be distributed across multiple containers (AWS ECS services), as well as multiple hosts (AWS ECS instances).

Fortunately, with AWS ECS (Docker), CloudFormation, and EFS, it is very easy to share a filesystem across all application instances that may be running as ECS services or tasks.

Terminology:

• ECS: EC2 Container Service; hosted container service utilizing Docker
• EC2: Elastic Compute Cloud; hosted virtual machines and associated resources
• ECS Instance: EC2 instance with ECS support which runs the Docker daemon
• ECS Cluster: a cluster of ECS instances all running the same set of ECS services or tasks
• ECS Service: a persistently running Docker container
• ECS Task: a single run of a Docker container
• EFS: Elastic File System; hosted NFS server
• CloudFormation: System for automating EC2 resource provisioning and configuration

EFS

EFS is AWS’ implementation of NFS which shares filesystems across TCP/IP networks. The NFS server hosts the filesystem and provides NFS clients access over the network as if the filesystem were mounted locally. Naturally, there are major performance considerations, but on a local area network with NFS client file caching, the performance is often acceptable.

Once an EFS filesystem has been provisioned, it can be mounted directly. However, ECS instances in a cluster are generally provisioned and managed automatically with CloudFormation templates and EC2 auto-scaling groups. The process to mount the EFS filesystem must be added to the CloudFormation template.

CloudFormation

AWS provides a convenient way to prepare new EC2 instances by allowing “UserData” to be specified which may contain arbitrary scripts and other customizations. In CloudFormation, a template may contain a “UserData” section that is passed to a new EC2 instance at time of provisioning. The specific CloudFormation template created to manage ECS instances in an ECS cluster can be updated to include a script which automates the EFS mounting process.

Mount Script

The script to mount EFS itself is quite straightforward:

# Make sure all packages are up-to-date
yum update -y

# Make sure that NFS utilities and AWS CLI utilities are available
yum install -y jq nfs-utils python27 python27-pip awscli
pip install --upgrade awscli

# Name of the EFS filesystem (match what was created in EFS)
EFS_FILE_SYSTEM_NAME="my-efs-filesystem"

# Gets the EC2 availability zone for the current ECS instance
EC2_AVAIL_ZONE="$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)"
# Gets the EC2 region for the current ECS instance
EC2_REGION="$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r '.region')"

# Creates the mount-point for the EFS filesystem
DIR_TGT="/mnt/efs"
mkdir "${DIR_TGT}"

# Get the EFS filesystem ID.
EFS_FILE_SYSTEM_ID="$(/usr/local/bin/aws efs describe-file-systems --region "${EC2_REGION}" | jq '.FileSystems[]' | jq "select(.Name==\"${EFS_FILE_SYSTEM_NAME}\")" | jq -r '.FileSystemId')"

if [ -z "${EFS_FILE_SYSTEM_ID}" ]; then
    echo "ERROR: variable not set" 1> /etc/efssetup.log
    exit
fi

# Create the mount source path
DIR_SRC="${EC2_AVAIL_ZONE}.${EFS_FILE_SYSTEM_ID}.efs.${EC2_REGION}.amazonaws.com"

# Actually mount the EFS filesystem
mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,soft,timeo=600,retrans=2 "${DIR_SRC}:/" "${DIR_TGT}"

# Create a backup of the existing /etc/fstab
cp -p "/etc/fstab" "/etc/fstab.back-$(date +%F)"

# Add the new mount point to /etc/fstab
echo -e "${DIR_SRC}:/ \t\t ${DIR_TGT} \t\t nfs \t\t nfsvers=4.1,rsize=1048576,wsize=1048576,soft,timeo=600,retrans=2 \t\t 0 \t\t 0" | tee -a /etc/fstab

# Stop the ECS agent container
docker stop ecs-agent

# Restart Docker
/etc/init.d/docker restart

# Start the ECS agent container
docker start ecs-agent

Dependencies for this script:

• Default ECS cluster configuration (Amazon Linux ECS AMI).
• The ECS instance must have a IAM role that gives it at least read access to EFS (in order to locate the EFS filesystem ID).
• The ECS instance must be in a security group that allows port tcp/2049 (NFS) inbound/outbound.
• The security group that the ECS instance belongs to must be associated with the target EFS filesystem.

Notes on this script:

• The EFS mount path is calculated on a per-instance basis as the EFS endpoint varies depending upon the region and availability zone where the instance is launched.
• The EFS mount is added to /etc/fstab so that if the ECS instance is rebooted, the mount point will be re-created.
• Docker is restarted to ensure it correctly detects the EFS filesystem mount.

CloudFormation Template

The “UserData” section of a CloudFormation template is also fairly simple, although there are some YAML complexities at work. (For concision, the comments in the Bash script were omitted.):

Fn::Base64: !Join
  -
    ""
  -
    - !If
      - SetEndpointToECSAgent
      - !Sub |
         #!/bin/bash
         echo ECS_CLUSTER=${EcsClusterName} >> /etc/ecs/ecs.config
         echo ECS_BACKEND_HOST=${EcsEndpoint} >> /etc/ecs/ecs.config
      - !Sub |
         #!/bin/bash
         echo ECS_CLUSTER=${EcsClusterName} >> /etc/ecs/ecs.config
    - |
       yum update -y
       yum install -y jq nfs-utils python27 python27-pip awscli
       pip install --upgrade awscli
       EFS_FILE_SYSTEM_NAME="my-efs-filesystem"
       EC2_AVAIL_ZONE="$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)"
       EC2_REGION="$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r '.region')"
       DIR_TGT="/mnt/efs"
       mkdir "${DIR_TGT}"
       EFS_FILE_SYSTEM_ID="$(/usr/local/bin/aws efs describe-file-systems --region "${EC2_REGION}" | jq '.FileSystems[]' | jq "select(.Name==\"${EFS_FILE_SYSTEM_NAME}\")" | jq -r '.FileSystemId')"
       if [ -z "${EFS_FILE_SYSTEM_ID}" ]; then
         echo "ERROR: variable not set" 1> /etc/efssetup.log
         exit
       fi
       DIR_SRC="${EC2_AVAIL_ZONE}.${EFS_FILE_SYSTEM_ID}.efs.${EC2_REGION}.amazonaws.com"
       mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,soft,timeo=600,retrans=2 "${DIR_SRC}:/" "${DIR_TGT}"
       cp -p "/etc/fstab" "/etc/fstab.back-$(date +%F)"
       echo -e "${DIR_SRC}:/ \t\t ${DIR_TGT} \t\t nfs \t\t nfsvers=4.1,rsize=1048576,wsize=1048576,soft,timeo=600,retrans=2 \t\t 0 \t\t 0" | tee -a /etc/fstab
       docker stop ecs-agent
       /etc/init.d/docker restart
       docker start ecs-agent

Notes on YAML in this section:

• CloudFormation enables special functions in YAML which allows for some basic conditional logic and formatting.
• The !Join function joins an array of strings with the provided string to render a single string.
• The !Sub function substitutes variables in a string with values provided elsewhere in a CloudFormation template.
• The !If function returns a string based on the value of a CloudFormation variable.
• The !Base64 function base64 encodes a string.

Other notes on this section:

• The conditional logic on SetEndpointToECSAgent is part of standard ECS CloudFormation templates, and was not modified from defaults.
• The Bash script is provided as a single string which is simply joined with the default CloudFormation UserData section.
• The joined string is base64 encoded (saves space and prevents formatting issues) when it is provided to the ECS instance at time of provisioning.
• Only the “UserData” section of a typical CloudFormation template was provided; the whole template, with this included “UserData” section can be downloaded.

Why CloudFormation Template?

• It may not be clear why it’s worth bothering with the CloudFormation template — why not just mount the EFS filesystem directly on an ECS instance?
• There may be multiple ECS instances which are automatically scaled up (or down) based on the ECS cluster configuration and auto-scaling rules. The EFS filesystem needs to be mounted on all new ECS instances automatically or not all Docker containers will have access to it.

ECS Services & Tasks

Once an ECS host actually has the EFS filesystem mounted, it is easy to pass it on to an ECS service or task using the “Volumes” feature. Docker will make a part of an ECS instance’s local filesystem available to the Docker container at an arbitrary mount point.

This can be accomplished by specifying and naming the portion of the ECS instance’s filesystem in the “Volumes” (--volumes on CLI) section of a task definition, and then mapping that volume to a mount point within the Docker container in the “Containers” (--container-definitions on CLI) section of a task definition.

Example JSON input for the Volumes section:

[
  {
    "host": {
      "sourcePath": "/mnt/efs/dir1"
    },
    "name": "uploads"
  },
  {
    "host": {
      "sourcePath": "/mnt/efs/dir2"
    },
    "name": "processed"
  }
]

Example JSON input for the Containers section:

[
  {
    "command": [
      "bundle",
      "exec",
      "rackup"
    ],
    "cpu": 10,
    "environment": [
        "RACK_ENV": "production"
    ],
    "essential": true,
    "image": "accountnum.dkr.ecr.us-west-2.amazonaws.com/ecr-repo:latest",
    "memory": 10,
    "mountPoints": [
      {
        "sourceVolume": "uploads",
        "containerPath": "/chroot/home/aa6b7bdd/spin.atomicobject.com/html/uploads"
      },
      {
        "sourceVolume": "processed",
        "containerPath": "/chroot/home/aa6b7bdd/spin.atomicobject.com/html/procesed"
      },      
    ],
    "name": "webapplication",
    "portMappings": [
      {
        "protocol": "tcp",
        "containerPort": 80,
        "hostPort": 0
      }
    ],
    "volumesFrom": []
  }
]

To provide the JSON to the AWS CLI, it generally needs to be properly escaped on the command line, or referenced as a file with the --cli-input-json option (which requires a fully formed document, like this).

All that’s left to do is configure a service or run a task that uses this new task definition. The Docker container will have full access to the contents of the mounted EFS filesystem directories, allowing the data on the filesystem to be available to all containers across all ECS instances.

Troubleshooting & Recommendations

Getting all of this set up can be a little tricky the first time through. Here are some ideas and good points to start debugging:

• Make sure that an EFS filesystem can be manually mounted on an existing ECS instance before trying to automate it.
• Make sure that you scale down/up an ECS cluster (or just terminate existing ECS instances) to ensure that the new “UserData” section in the CloudFormation template is actually used (it does not retroactively apply to existing instances).
• Check the logs of the cloud init agent (/var/logs/cloud-init.log and /var/logs/cloud-init-output.log) after provisioning a new ECS instance if the EFS filesystem mount does not work as expected.
• EFS filesystems are not local EBS volumes — there will be different performance characteristics. There is better performance with fewer, larger files than many, smaller files.

Good luck!