Openstack convert an instance with ephemeral root disk to persistent root disk

Recently a requirement from a client discovered that they didn’t like root disks in Openstack disappearing when an instance is terminated. I was asked if we could make the change on the fly. The short answer is no, you can’t do it on the fly, however what you can do is snapshot the current instance, create a volume from the snapshot image then boot a new instance using the volume as the instances root disk. The full requirements for this process are:

• Preserve the root disk, make it persistent.
• Move an volumes on the source instance to the new instance with persistent storage.
• Move the IP address from the source instance to the new instance.

I will first go through the manual steps, then provide a hacky python script I used to automate these steps. I perform all these tasks using the python based openstack client, sourcing the appropriate credential file before. These steps have been run on OSP 13. You will need to shutdown the source instance in order to remove the NIC and attached volumes from it.

1. The Manual way

First of all, gather some information from the source instance. Output that is not relevant has been omitted. Make a note of the displayed information, you will need this later!

$ openstack server show myserver1 -f json
{
 --- output omitted ---
  "addresses": "mynetwork=192.168.1.55",
  "flavor": "xlarge (ff35aa8a-819c-11e9-a889-005056893677)",
  "security_groups": "name='MY_SG'\nname='default'\nname='SQL_SG'",
  "volumes_attached": "id='0f71f246-819d-11e9-a889-005056893677'\nid='111b9dae-819d-11e9-a889-005056893677'\nid='111b9daf-819d-11e9-a889-005056893677'",
--- output omitted ---
}
 

Next find the network ID of the attached network.

$ openstack network show mynetwork
{
--- output omitted ---
  "id": "3245b26c-819d-11e9-a889-005056893677"
--- output omitted ---
}

Get some information about each of the attached volumes, I am only showing the output from the first volume, please run this for each attached volume, you will need the device value later.

openstack volume show 0f71f246-819d-11e9-a889-005056893677
{
  --- output omitted ---
  "status": "in-use",
  "id": "0f71f246-819d-11e9-a889-005056893677",
  "attachments": [
    {
      "volume_id": "0f71f246-819d-11e9-a889-005056893677",
      "device": "/dev/vdb",
      "id": "0f71f246-819d-11e9-a889-005056893677"
    }
  ]
  --- output omitted ---
}

Detach all attached volumes from the instance. Once again, I am only showing the first detach. Detach all additional volumes.

$ openstack server remove volume myserver1 0f71f246-819d-11e9-a889-005056893677

Remove attached network(s) interface(s).

$ openstack server remove network myserver1 3245b26c-819d-11e9-a889-005056893677

Create a snapshot of the shutdown source instance.

$ openstack server backup create --name  myserver1_snap myserver1
 

Wait for the snapshot to complete, keep an eye on the status, it is ready when the image in state active.

$ openstack image show myserver1_snap -f json
{
--- output omitted ---
   "status": "active",
--- output omitted ---
}

Create a bootable volume for the new instances root disk. Check status, wait until the new volume is in the state available. Ensure the size of the volume is the same size or greater than the image snapshot.

$ openstack volume create --image myserver1_snap --size 100 myserver1_root_vol
 
openstack volume show myserver1_root_vol -f json
{
   "status": "available",
}

Launch the instance using the created volume, attached the old IP and old volumes to the new instance. This is the point that you will want to know which volume maps to which device.

$ openstack server create --nic net-id=3245b26c-819d-11e9-a889-005056893677,v4-fixed-ip=192.168.1.55 --volume myserver1_root_vol --flavor xlarge --block-device-mapping 0f71f246-819d-11e9-a889-005056893677=/dev/vdb --block-device-mapping 111b9dae-819d-11e9-a889-005056893677=/dev/vdc 111b9daf-819d-11e9-a889-005056893677=/dev/vdd myserver1_new
 

After finishing, attach any security groups that are required.

2. The python script

Yes its hacky and ugly, but it does all the above steps :). Once again, must be run on a box with the python openstack client and a sourced creds file.

#!/usr/bin/python
import sys, os, json
 
if len(sys.argv) < 2:
  print "Error, invalid argument."
  print "Usage: %s src_instance" % sys.argv[0]
  sys.exit(1)
# set variables from args
SRC_INST = sys.argv[1]
#
# build up command to get information about instance
#
server_info = "openstack server show " + SRC_INST + " -f json"
#
# run command grab output in json
#
output = os.popen(server_info).read()
#
# parse the output, assign variables from json
#
output_json = json.loads(output)
flavor = output_json["flavor"].split(" ")[0]
security_groups = output_json["security_groups"]
#
# check if security group info exists and that volumes attached otherwise error out
#
if not security_groups:
  print "No nic attached, exiting"
  sys.exit(1)
volumes_attached = output_json["volumes_attached"].split("\n")
if not volumes_attached:
  print "No volumes found.. exiting"
  sys.exit(1)
addresses = output_json["addresses"].split("=")
ipaddr = addresses[1]
network = addresses[0]
netid_cmd = "openstack network show " + network + " -f json"
netid_output = os.popen(netid_cmd).read()
output_json = json.loads(netid_output)
netid = output_json['id']
#
# get network port id
#
port_info = "openstack port list -f json"
output = os.popen(port_info).read()
output_json = json.loads(output)
for nic in output_json:
  #
  # look for IP that matches instance, split the returned value
  #
  ip = nic["Fixed IP Addresses"].split("'")[1]
  if ip == ipaddr:
    portID = nic["ID"]
#
# create dictionary of attached volumes
#
volumes = {}
#print volumes_attached
# iterate through volumes attached list
for vol in volumes_attached:
  # strip out id and single quotes
  vol = vol.strip("id=").strip("'")
  # get device attached name for volume
  vol_info = "openstack volume show " + vol + " -f json"
  output = os.popen(vol_info).read()
  output_json = json.loads(output)
  # pull device name from json output
  device = output_json["attachments"][0]["device"]
  # add device and volume to list
  volumes[device] = vol
#
# get security groups, add to list
#
sec_group = []
for i in security_groups.split("\n"):
  name = i.split("=")[1].strip("'")
  #print name
  sec_group.append(name)
#
# remove network port
#
remove_port = "openstack server remove port " + SRC_INST + " " + portID
remove_output = os.popen(remove_port).read()
#
# detach volumes
#
for vol in volumes:
  cmd = "nova volume-detach " + SRC_INST + " " + volumes[vol]
  os.popen(cmd).read()
#
# build backup command, takes snapshot of source machine root disk, saves as an image
#
backup = "openstack server backup create --name " + SRC_INST + "_snap " + SRC_INST + " -f json"
# run backup
backup_output = os.popen(backup).read()
output_json = json.loads(backup_output)
#
# wait for snapshot image to be available
#
check_backup = "openstack image show " + SRC_INST + "_snap -f json"
check_output = os.popen(check_backup).read()
output_json = json.loads(check_output)
while output_json["status"] != "active":
  check_output = os.popen(check_backup).read()
  output_json = json.loads(check_output)
#
# share snapshot image
#
share_img = "openstack image set --shared " + SRC_INST + "_snap"
share_output = os.popen(share_img).read()
print share_img
#
# create volume from snap for new instance
#
create_vol = "openstack volume create --image " + SRC_INST + "_snap --size 100 " + SRC_INST + "_root_vol"
os.popen(create_vol).read()
print create_vol
check_vol = "openstack volume show " + SRC_INST + "_root_vol -f json"
check_output = os.popen(check_vol).read()
output_json = json.loads(check_output)
while output_json["status"] != "available":
  check_output = os.popen(check_vol).read()
  output_json = json.loads(check_output)
#
# create instance using newly created volume as the root disk
#
# build up create string
create_cmd = "openstack server create --nic net-id=" + netid + ",v4-fixed-ip=" + ipaddr + " --volume " + SRC_INST + "_root_vol --flavor " + flavor
for vol in volumes:
  create_cmd = create_cmd + " --block-device-mapping " + vol + "=" + volumes[vol]
create_cmd = create_cmd + " " + SRC_INST + "_new"
os.popen(create_cmd).read()
 
# update sec group
for x in sec_group:
  sec_group = "openstack server add security group " + SRC_INST + "_new " + x
  os.popen(sec_group).read()