Laurent Godet

A blog with too much yaml

EBS NVMe Block Device Mapping Using Volume IDs

Posted on Oct 20, 2019

Since the introduction of Nitro-based instances by AWS in 2017, it has always been notoriously difficult to mount EBS volumes in a reliable way. The reason behind it is that the device names generated for instances that support NVMe EBS volumes no longer conform to the traditional standard device path.

AWS Nitro

When EBS volumes get tied up to an NVMe running instance, devices follow the /dev/nvme[0-26]n1 naming convention (/dev/nvme0n1, /dev/nvme1n1, …), instead of the usual naming such as /dev/xvda or /dev/sdf.

In addition to the naming issue, the block device driver can attach these block devices in a different order than the one specified, because the created is based on the order in which the devices respond. For instance, if we attach 2 volumes, they won’t come up under the same device name, and will be attached in an arbitrary order.

# attach 2 volumes to a running ec2 instance
$ INSTANCE_ID=i-0c1fbcaa331a0b088
$ aws ec2 attach-volume --device /dev/sde --volume-id vol-1234567890abcdef0 --instance-id $INSTANCE_ID 
$ aws ec2 attach-volume --device /dev/sdf --volume-id vol-0558010cb55b095e7 --instance-id $INSTANCE_ID

# as we can see:
#   /dev/sde => /dev/nvme2n1
#   /dev/sdf => /dev/nvme0n1
ec2:~$ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0         7:0    0   89M  1 loop /snap/core/7713
loop1         7:1    0   18M  1 loop /snap/amazon-ssm-agent/1480
loop2         7:2    0 89.1M  1 loop /snap/core/7917
nvme0n1     259:0    0  100G  0 disk 
nvme1n1     259:1    0    8G  0 disk 
└─nvme1n1p1 259:2    0    8G  0 part /
nvme2n1     259:0    0  200G  0 disk

The following article will show you how to map original device paths to the NVME ones in an automated fashion.

N.B: Before diving into the details, the full code sample is available on https://github.com/lostick/nvme-mapping-launch-template, if you want to have a crack at it.

Prerequisite

The NVMe command line package is needed to interest with the NVMe driver and retrieve information about the NVMe devices.

We also install xfsprogs to mount xfs file systems, as well as jq to parse and sanitize command line outputs in a safe manner.

ec2:~$ apt update && apt install -y \
    nvme-cli \
    xfsprogs \
    jq

Why Mapping by Device Path is too clunky

Up until recently, the most common way to retrieve the original device path and map it to the NVMe device was to parse the device path using nvme-cli.

Albeit clunky, This was working fine in older distributions

# on ubuntu 16.04
ec2:~$ nvme id-ctrl -o binary /dev/nvme0n1 | cut -c3073-3104 | tr -s ' ' | sed 's/ $//g'
/dev/sdb

Unfortunately, this is no longer working with recent CentOS and Ubuntu releases, nvme does not output the path prefix anymore:

# on ubuntu 18.04
# vendor-specific output differs from one distribution to another
ec2:~$ nvme id-ctrl -v /dev/nvme0n1
NVME Identify Controller:
vid     : 0x1d0f
ssvid   : 0x1d0f
sn      : vol0d2fbec8ec2c1029e
mn      : Amazon Elastic Block Store              
fr      : 1.0  
rab     : 32
ieee    : dc02a0
cmic    : 0
mdts    : 6
cntlid  : 0
ver     : 0
rtd3r   : 0
rtd3e   : 0
oaes    : 0
ctratt  : 0
oacs    : 0
acl     : 4
aerl    : 0
frmw    : 0x3   
...
ps    0 : mp:0.01W operational enlat:1000000 exlat:1000000 rrt:0 rrl:0
          rwt:0 rwl:0 idle_power:- active_power:-
ps    1 : mp:0.00W operational enlat:0 exlat:0 rrt:0 rrl:0
          rwt:0 rwl:0 idle_power:- active_power:-
vs[]:
       0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
0000: 78 76 64 61 20 20 20 20 20 20 20 20 20 20 20 20 "xvda............"
0010: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 "................"
0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................"
0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................"
0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................"
...

Same goes with the binary dump option - /dev/ is dropped from the cli output

ec2:~$ nvme id-ctrl --output binary /dev/nvme0n1 
vol0d2fbec8ec2c1029eAmazon Elastic Block Store   1.0    ??fD@B@Bxvda              root:/home/ubuntu#

A safer option: Mapping Devices by Volume ID

Instead of trying to parse the device path from nvme cli’s binary output , we can work out another way to map the traditional device name with the NVME one by fetching the Volume ID.

First, make sure awscli is present as we will fetch data from aws API. No need to install it as a pip requirement on ubuntu 18.04, it’s available through apt as an official repository.

ec2:~$ apt install -y \
    awscli \
    curl

Next, create a policy to allow the instance read access to volumes metadata.

# create a policy document
$ cat ec2-vol-ro.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVolumeAttribute*",
                "ec2:DescribeVolumeStatus*",
                "ec2:DescribeVolumes*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

# create an IAM policy
$ aws iam create-policy \
    --policy-name ec2-vol-ro \
    --policy-document file://ec2-vol-ro.json
{
    "Policy": {
        "PolicyName": "ec2-vol-ro",
        "PolicyId": "POLICY-ID",
        "Arn": "arn:aws:iam::ACCOUNT-ID:policy/ec2-vol-ro",
        "Path": "/",
        "DefaultVersionId": "v1",
        "AttachmentCount": 0,
        "PermissionsBoundaryUsageCount": 0,
        "IsAttachable": true,
        "CreateDate": "2019-10-20T11:19:39Z",
        "UpdateDate": "2019-10-20T11:19:39Z"
    }
}

Get the policy ARN, and attach it to the instance role that you have created for the running EC2 instance.

# an instance role (here ec2-nvme-demo) needs to be created beforehand
$ aws iam attach-role-policy \
    --policy-arn arn:aws:iam::ACCOUNT-ID:policy/ec2-vol-ro \
    --role-name ec2-nvme-demo

We can now query EC2 API from the instance:

# set the region 
ec2:~$ export AWS_DEFAULT_REGION=us-west-1

# the device name was set previously when we attached the volume
ec2:~$ aws_block_device=/dev/xvda

# get instance id from ec2 metadata
ec2:~$ instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)

# retrieve the volume data for the device name
ec2:~$ aws ec2 describe-volumes --filters \
    Name=attachment.instance-id,Values=$instance_id \
    Name=attachment.device,Values=$aws_block_device
{
    "Volumes": [
        {
            "Attachments": [
                {
                    "AttachTime": "2019-10-20T07:00:43.000Z",
                    "Device": "/dev/xvda",
                    "InstanceId": "i-03efcaf4d1fd4a19f",
                    "State": "attached",
                    "VolumeId": "vol-0d2fbec8ec2c1029e",
                    "DeleteOnTermination": true
                }
            ],
            "AvailabilityZone": "us-west-1",
            "VolumeType": "gp2",
            ...
        }
    ]
}

Now, we just need to extract the raw ID, sanitize it and get the corresponding NVMe block device.

# rename the volume id to follow nvme convention naming
ec2:~$ volume_id=vol-0d2fbec8ec2c1029e
ec2:~$ nvme_volume_id=vol$(echo $volume_id | cut -c5-)

# get the nvme device that matches the volume id
ec2:~$ nvme list -o json | jq -r '.Devices | .[] | select(.SerialNumber | contains("$nvme_volume_id"))'
{
  "DevicePath": "/dev/nvme0n1",
  "Firmware": "1.0",
  "Index": 1,
  "ModelNumber": "Amazon Elastic Block Store",
  "ProductName": "Unknown Device",
  "SerialNumber": "vol0d2fbec8ec2c1029e",
  "UsedBytes": 0,
  "MaximiumLBA": 16777216,
  "PhysicalSize": 8589934592,
  "SectorSize": 512
}

Finally, symlink the original path to the nvme device path.

ec2:~$ ln -s /dev/nvme0n1 $aws_block_device

ec2:~$ file -s $aws_block_device
/dev/xvda: symbolic link to /dev/nvme0n1

Once the mapping is done, it is then trivial to automate the full process of creating a file system and mounting it.

The terraform implementation is available on github.

References

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes
https://github.com/oogali/ebs-automatic-nvme-mappingh
https://gist.github.com/jalaziz/bcfe2f71e3f7e8fe42a9c294c1e9279f