A History Lesson
OpenStack Glance, the Image service, historically stored images used for creating Nova instances in filesystems, sometimes local to controller nodes, but more usually in a Ceph environment.
Over the years Glance has added different locations to store your images, the one of most interest to me being the use of a Cinder backend as the store, with this feature going GA back in the OpenStack Train release.
More recently the concept of multi-stores was developed where your images could be held in different locations, or all of them. Very useful for a core-edge type deployment.
(Factlet: the Glance project logo is a chipmunk and a group of chipmunks is called a scurry)
Why this blog?
Recently I have been approached by several people asking how they can remove Ceph from their OpenStack clusters and just use an external block device, such as a Pure Storage FlashArray.
It’s pretty easy to replace your Ceph volumes, boot and data, with volumes on an external device using an upstream Cinder driver, but invariably the Glance store is also on Ceph and therefore a little piece of Ceph is left just for Glance.
I decided to investigate the Glance Cinder store feature and specifically the multi-store feature to see if it was fit for purpose and how easy or difficult to configure.
Initial Configuration
Whilst not wanting to make my test environment too complex, it still needed to have multiple glance stores so that multi-store could be configured.
The easiest configuration was to create a simple, single-node OpenStack cluster using Devstack but provide three cinder backends on which Glance stores could be created.
Why three backends and not just two?
I also wanted to see if replicated volumes could be used as part of the Glance multi-store and if so what limitations may exist for these types of volumes.
I’m not going to go into the detail of how to configure multiple cinder backends and how you create different volume types to ensure volumes can be created on the appropriate backend, or how to set up a volume type for replicated volumes.
The final configuration has three cinder backends and four volume types; one for each backend and one for synchronously replicated volumes across backends two and three.
Changes to devstack
The default Devstack configuration is not designed to provide anything other than a simple file-based glance store and simple cinder backends.
Spinning up the default Devstack is the easiest way to move forward with this and then there are some modifications required to both the Cinder and Glance configuration files to enable Glance multi-store.
One recommendation I would give before you deploy Devstack is to ensure that you are not going to hit any Glance quota limits, so add the parameter GLANCE_LIMIT_IMAGE_SIZE_TOTAL to your local.conf before building and give it a big value. I went with 25000.
cinder.conf
Ensure that all three backends are correctly defined and that they are set in the enabled_backends parameter.
Luckily the default Devstack deployment does nearly everything you need. The only thing that needs to be done in cinder.conf is to add the following two lines to the [DEFAULT] stanza:
[DEFAULT]
allowed_direct_url_schemes = cinder
image_upload_use_cinder_backend = TrueYou will need to restart the cinder-volume service after this modification using
systemctl restart devstack@c-volEnsure that your backend services are running correctly:
stack@multi1:~$ openstack volume service list
+------------------+---------------------+------+---------+-------+----------------------------+
| Binary | Host | Zone | Status | State | Updated At |
+------------------+---------------------+------+---------+-------+----------------------------+
| cinder-scheduler | multi1 | nova | enabled | up | 2023-07-03T18:54:46.000000 |
| cinder-volume | multi1@puredriver-1 | nova | enabled | up | 2023-07-03T18:54:52.000000 |
| cinder-volume | multi1@puredriver-2 | nova | enabled | up | 2023-07-03T18:54:46.000000 |
| cinder-volume | multi1@puredriver-3 | nova | enabled | up | 2023-07-03T18:54:45.000000 |
+------------------+---------------------+------+---------+-------+----------------------------+and that you have your volume types set up:
stack@multi1:~$ openstack volume type list --long
+--------------------------------------+-------------+-----------+-----------------------------------+---------------------------------------------------------------+
| ID | Name | Is Public | Description | Properties |
+--------------------------------------+-------------+-----------+-----------------------------------+---------------------------------------------------------------+
| f32dbca6-ff96-4d78-b020-6c49c1414dcf | replicated | True | Replicated volumes on pure2 and 3 | replication_enabled='<is> True', replication_type='<in> sync' |
| e5eb0d60-d171-4896-a9cb-6354ea282ba2 | pure3 | True | Volumes on backend pure3 | volume_backend_name='puredriver-3' |
| e542399e-0c97-40fe-b7b0-84a67c67c514 | pure2 | True | Volumes on backend pure2 | volume_backend_name='puredriver-2' |
| b18b2e9f-5902-472c-b7c4-14a47fa4328b | pure | True | Volumes on backend pure1 | volume_backend_name='puredriver-1' |
| 0ed02615-4025-409c-8d80-a8ca7ac9b941 | __DEFAULT__ | True | Default Volume Type | |
+--------------------------------------+-------------+-----------+-----------------------------------+---------------------------------------------------------------+glance-api.conf
As we are going to be making major changes to the default Devstack Glance configuration, this is where things become a little more complicated.
Firstly, add the following lines into the [DEFAULT] stanza:
show_multiple_locations = True
show_image_direct_url = FalseWARNING: The parameter show_multiple_locations is enabled here purely for educational purposes to show how an image is located within multiple stores. In a Production environment it is recommended this parameter is disabled as it can expose secure access information depending on the type of image stores configured.
and, very importantly, a line to show which of the Glance stores we are going to be using:
enabled_backends = pure1:cinder, pure2:cinder, pure3:cinder, pure4:cinderEach of the backends listed; pure1, pure2, pure3 and pure4 must have their own definition stanza added to the file. I will show all four backends here as, although each stanza is very similar, there are important differences:
[pure1]
store_description = "Pure Array 1"
cinder_store_auth_address = http://controller/identity
cinder_store_user_name = glance
cinder_store_password = pureuser
cinder_store_project_name = service
cinder_volume_type = pure
[pure2]
store_description = "Pure Array 2"
cinder_store_auth_address = http://controller/identity
cinder_store_user_name = glance
cinder_store_password = pureuser
cinder_store_project_name = service
cinder_volume_type = pure2
[pure3]
store_description = "Pure Array 3"
cinder_store_auth_address = http://controller/identity
cinder_store_user_name = glance
cinder_store_password = pureuser
cinder_store_project_name = service
cinder_volume_type = pure3
[pure4]
store_description = "Pure Replicated"
cinder_store_auth_address = http://controller/identity
cinder_store_user_name = glance
cinder_store_password = pureuser
cinder_store_project_name = service
cinder_volume_type = replicatedThe important line to note in each stanza is cinder_volume_type. This is what controls which backend will be used for each image store, or in the case of the last stanza, the replicated volumes used in the store.
Some additional stanzas also need to be added to ensure multistore works correctly and that you can manipulate images in these stores.
Add the following stanzas:
[glance_store]
filesystem_store_datadir = /opt/stack/data/glance/images/
stores = file,cinder
default_store = cinder
default_backend = pure2
[os_glance_tasks_store]
filesystem_store_datadir = /opt/stack/data/glance/tasks_work_dir
[os_glance_staging_store]
filesystem_store_datadir = /opt/stack/data/glance/stagingYou must ensure that the directories referred to actually exist. You will need to create the tasks and staging directories.
Notice in the glance_store stanza that we are defining the default store as cinder and a default backend. Also, notice that the store value contains file. This is required as the staging store is a file store. The reason for the staging store will become apparent later.
Now you can restart the glance API service using
systemctl restart devstack@g-apiYou can check that all is looking good by running the following:
stack@multi1:~$ glance stores-info
+----------+----------------------------------------------------------------------------------+
| Property | Value |
+----------+----------------------------------------------------------------------------------+
| stores | [{"id": "pure1", "description": "Pure Array 1"}, {"id": "pure2", "description": |
| | "Pure Array 2", "default": "true"}, {"id": "pure3", "description": "Pure Array |
| | 3"}, {"id": "pure4", "description": "Pure Replicated"}] |
+----------+----------------------------------------------------------------------------------+(there isn’t, at the time of writing, an OSC command for this so you need to use the old glance CLI commands)
Creating an Image
Now we have the base setup, let’s put an image into the image stores.
First I’ll get an image that we will store for use in OpenStack and convert it to RAW, as this is the required format for images in cinder-backed stores. That is because each image in a cinder-backed glance store is an individual volume on the backend.
stack@multi1:~$ wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img
stack@multi1:~$ qemu-img convert -O raw focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.rawNow we upload the RAW image to glance and have a look at the image created:
stack@multi1:~$ openstack image create --public --container-format bare --disk-format raw --file ./focal-server-cloudimg-amd64.raw "Ubuntu 20.04"
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| container_format | bare |
| created_at | 2023-07-03T19:36:29Z |
| disk_format | raw |
| file | /v2/images/b2de3de1-5b6e-4b05-bace-7f1682f948a2/file |
| id | b2de3de1-5b6e-4b05-bace-7f1682f948a2 |
| min_disk | 0 |
| min_ram | 0 |
| name | Ubuntu 20.04 |
| owner | 6f9ddcf4d9f148789b03a20447a39f8c |
| properties | locations='[]', os_hidden='False', owner_specified.openstack.md5='', owner_specified.openstack.object='images/Ubuntu 20.04', owner_specified.openstack.sha256='' |
| protected | False |
| schema | /v2/schemas/image |
| status | queued |
| tags | |
| updated_at | 2023-07-03T19:36:29Z |
| visibility | public |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
stack@multi1:~$ openstack image show b2de3de1-5b6e-4b05-bace-7f1682f948a2
+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum | 891f576a5108c7f2803ba5105c295ccc |
| container_format | bare |
| created_at | 2023-07-03T19:36:29Z |
| disk_format | raw |
| file | /v2/images/b2de3de1-5b6e-4b05-bace-7f1682f948a2/file |
| id | b2de3de1-5b6e-4b05-bace-7f1682f948a2 |
| min_disk | 0 |
| min_ram | 0 |
| name | Ubuntu 20.04 |
| owner | 6f9ddcf4d9f148789b03a20447a39f8c |
| properties | locations='[{'url': 'cinder://pure2/877d053f-dcb5-47f2-a8e1-76c99c9f018b', 'metadata': {'store': 'pure2'}}]', os_hash_algo='sha512', os_hash_value='c4fc5c1f3088785ce0b9eaf82c614030c294f639011fbfbd75851cc2c23918991e24428152e325208cbe17111c4cc6e2c938d9802a49fd01f9d49871e2b83918', os_hidden='False', owner_specified.openstack.md5='', owner_specified.openstack.object='images/Ubuntu 20.04', owner_specified.openstack.sha256='', stores='pure2' |
| protected | False |
| schema | /v2/schemas/image |
| size | 2361393152 |
| status | active |
| tags | |
| updated_at | 2023-07-03T19:37:18Z |
| virtual_size | 2361393152 |
| visibility | public |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+Now there is a lot of information produced by the image show command, but the crucial piece is the properties section. To make this easier to see, I’ll use the old glance CLI command as it formats the information better IMHO:
stack@multi1:~$ glance image-show b2de3de1-5b6e-4b05-bace-7f1682f948a2
+----------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+----------------------------------+----------------------------------------------------------------------------------+
| checksum | 891f576a5108c7f2803ba5105c295ccc |
| container_format | bare |
| created_at | 2023-07-03T19:36:29Z |
| disk_format | raw |
| id | b2de3de1-5b6e-4b05-bace-7f1682f948a2 |
| locations | [{"url": "cinder://pure2/877d053f-dcb5-47f2-a8e1-76c99c9f018b", "metadata": |
| | {"store": "pure2"}}] |
| min_disk | 0 |
| min_ram | 0 |
| name | Ubuntu 20.04 |
| os_hash_algo | sha512 |
| os_hash_value | c4fc5c1f3088785ce0b9eaf82c614030c294f639011fbfbd75851cc2c23918991e24428152e32520 |
| | 8cbe17111c4cc6e2c938d9802a49fd01f9d49871e2b83918 |
| os_hidden | False |
| owner | 6f9ddcf4d9f148789b03a20447a39f8c |
| owner_specified.openstack.md5 | |
| owner_specified.openstack.object | images/Ubuntu 20.04 |
| owner_specified.openstack.sha256 | |
| protected | False |
| size | 2361393152 |
| status | active |
| stores | pure2 |
| tags | [] |
| updated_at | 2023-07-03T19:37:18Z |
| virtual_size | 2361393152 |
| visibility | public |
+----------------------------------+----------------------------------------------------------------------------------+Notice the stores field states that the image is in pure2. Remember that we set the default glance store to pure2.
If you look in the locations section you will also see "url": "cinder://pure2/877d053f-dcb5-47f2-a8e1-76c99c9f018b", again saying that the image is in the cinder store pure2, but also you get the ID of the actual cinder volume that was created and holds the image.
We can look at the actual cinder volume (if you have admin privileges, as the owner of the volume is the cinder internal tenant I mentioned right at the beginning). I will use the old cinder CLI command as it formats the information better than the new OSC command:
stack@multi1:~$ cinder show 877d053f-dcb5-47f2-a8e1-76c99c9f018b
+--------------------------------+------------------------------------------------------------------------+
| Property | Value |
+--------------------------------+------------------------------------------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2023-07-03T19:36:32.000000 |
| description | None |
| encrypted | False |
| id | 877d053f-dcb5-47f2-a8e1-76c99c9f018b |
| metadata | glance_image_id : b2de3de1-5b6e-4b05-bace-7f1682f948a2 |
| | image_owner : 6f9ddcf4d9f148789b03a20447a39f8c |
| | image_size : 2361393152 |
| | readonly : True |
| migration_status | None |
| multiattach | False |
| name | image-b2de3de1-5b6e-4b05-bace-7f1682f948a2 |
| os-vol-host-attr:host | multi1@puredriver-2#puredriver-2 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 60edd2c9fb0c4808b815037781599d4c |
| readonly | True |
| replication_status | disabled |
| size | 3 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2023-07-03T19:37:18.000000 |
| user_id | 5361f1afb7c8410399f75f0ca532c342 |
| volume_type | pure2 |
+--------------------------------+------------------------------------------------------------------------+There is a lot of information here, especially in the metadata section where you can see the glance_image_id, image_owner and the image_size.
What Have I Gained?
You may be asking; Why do I want to set up glance image stores on Cinder?
If you think about the OpenStack workflow when creating a boot volume for a Nova instance, it goes something like this:
- Request Nova instance to boot from image using a volume type backed by an external storage device
- Glance looks for the image requested and gets its location
- Cinder creates a volume on the backend device the size of the boot image requested
- Cinder mounts the new, empty, volume on the controller host running the cinder-volume service
- The glance image is copied over HTTP(S) from the glance store to the mounted volume, usually using Linux
ddto do this. - On copy completion, Cinder dismounts the volume (and resizes it to the requested size if necessary)
- Nova then boots the instance from the newly created volume.
Repeat these steps each time Nova wants to create an instance from the same boot image.
Now step 5, in particular, can take a very long time, especially if your image is many gigabytes in size, which some custom images can be. Imagine creating 20 instances all using the same image that has to be mounted, copied, and unmounted 20 times. This can add a lot of time to your workflows.
Using a Glance store on Cinder allows Glance to give your instance creation a helping hand.
If the volume type you have selected for your boot volume is on the same backend as the backend that your image is stored on, Glance will instantly clone the image volume to create the boot volume for Nova, thereby removing steps 3 through 5 detailed above. Depending on the backend, and this is true of Pure FlashArrays, the cloned volumes will consume practically zero space due to data reduction technologies available on the backend.
However, if your requested volume type is not the same as the image location, the same 7-step process is followed.
Multi-Store Advantages
The whole point of this blog was to set up multiple cinder stores for Glance. The advantage of multiple stores is that you can have the same image on multiple backends, thereby increasing your chances that the volume type you have requested for your Nova boot volume, matches one of the backends that holds a copy of the image.
So, therefore, I need to get the image I have just uploaded and which resides in the image store on the backend pure2 onto the other backend arrays I have configured.
You can do this using the image-import copy-image method and specify individual stores the image can be copied to (there is no OSC command for this that I am aware of):
glance image-import b2de3de1-5b6e-4b05-bace-7f1682f948a2 --stores pure1 --import-method copy-imageor you can use the all-stores option to force Glance to copy the image to every store it knows about:
stack@multi1:~$ glance image-import b2de3de1-5b6e-4b05-bace-7f1682f948a2 --all-stores true --import-method copy-imageThis command can take some time to complete, depending on the image size and also the number of stores you are copying to, but when complete if you look at the information about the image you will see that it is now referenced in all the images stores, both in the locations field, where the individual volumes are identified for each backend and the stores field.
I mentioned earlier that we added some additional stanzas to the Glance API configuration file. One of these os_glance_staging_store and its associated directory are required for the copy-image process to succeed, as it uses this area for staging the image as part of the copy procedure.
stack@multi1:~$ glance image-show b2de3de1-5b6e-4b05-bace-7f1682f948a2
+----------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+----------------------------------+----------------------------------------------------------------------------------+
| checksum | 891f576a5108c7f2803ba5105c295ccc |
| container_format | bare |
| created_at | 2023-07-03T19:36:29Z |
| disk_format | raw |
| id | b2de3de1-5b6e-4b05-bace-7f1682f948a2 |
| locations | [{"url": "cinder://pure1/1863128b-a1aa-434a-8f54-bc1bd5954c58", "metadata": |
| | {"store": "pure1"}}, {"url": |
| | "cinder://pure4/839599b4-1ae7-4d50-a436-399a8e71cb01", "metadata": {"store": |
| | "pure4"}}, {"url": "cinder://pure3/1dab6280-8b06-4959-8c2f-6294d4e56813", |
| | "metadata": {"store": "pure3"}}, {"url": |
| | "cinder://pure2/877d053f-dcb5-47f2-a8e1-76c99c9f018b", "metadata": {"store": |
| | "pure2"}}] |
| min_disk | 0 |
| min_ram | 0 |
| name | Ubuntu 22.04 |
| os_glance_failed_import | |
| os_glance_importing_to_stores | |
| os_hash_algo | sha512 |
| os_hash_value | c4fc5c1f3088785ce0b9eaf82c614030c294f639011fbfbd75851cc2c23918991e24428152e32520 |
| | 8cbe17111c4cc6e2c938d9802a49fd01f9d49871e2b83918 |
| os_hidden | False |
| owner | 6f9ddcf4d9f148789b03a20447a39f8c |
| owner_specified.openstack.md5 | |
| owner_specified.openstack.object | images/Ubuntu 20.04 |
| owner_specified.openstack.sha256 | |
| protected | False |
| size | 2361393152 |
| status | active |
| stores | pure1,pure4,pure3,pure2 |
| tags | [] |
| updated_at | 2023-07-03T19:37:18Z |
| virtual_size | 2361393152 |
| visibility | public |
+----------------------------------+----------------------------------------------------------------------------------+Notice that one of the store locations is pure4. If you remember back to the start of this, I set up store pure4 to use the volume type replicated.
Warnings…
As I mentioned at the beginning of this, as you start to use your Glance image stores, you may hit Keystone quota limits and receive an error message like this:
HTTP 413 Request Entity Too Large: The request returned a 413 Request Entity Too Large. This generally means that rate limiting or a quota threshold was breached.: The response body:: Project 6f9ddcf4d9f148789b03a20447a39f8c is over a limit for [Resource image_size_total is over limit of 25000 due to current usage 25784 and delta 0]You can check your quotas using the following command:
stack@multi1:~$ openstack registered limit list
+----------------------------------+----------------------------------+-----------------------+---------------+-------------+-----------+
| ID | Service ID | Resource Name | Default Limit | Description | Region ID |
+----------------------------------+----------------------------------+-----------------------+---------------+-------------+-----------+
| 56706fbba0654c1db3be68217552d531 | b4f2c55ef02e4ed5a9e9e63345f90220 | image_size_total | 25000 | None | RegionOne |
| 0ea7b0b812584c55b066ecdca7ce1cbf | b4f2c55ef02e4ed5a9e9e63345f90220 | image_stage_total | 25000 | None | RegionOne |
| d83cb84f658d4a06bc3c51feb430ea1b | b4f2c55ef02e4ed5a9e9e63345f90220 | image_count_total | 100 | None | RegionOne |
| d73fa5185c9a4c06b49a5550a84eda18 | b4f2c55ef02e4ed5a9e9e63345f90220 | image_count_uploading | 100 | None | RegionOne |
+----------------------------------+----------------------------------+-----------------------+---------------+-------------+-----------+
If you do hit your limit, then you can use the openstack image delete command to free up space, but not that this will delete the image across all the stores the image resides in.
I have noticed, and I suspect this is a function of the way Glance Image stores work, that if you have an image that has been created using a synchronously replicated volume type, the image will only be cloned on the backend array the creating volume type referenced.
So, in the case of my configuration, the replicated volume type was based on the one backend that had a replication target defined in the cinder configuration file stanza, therefore the underlying volumes os-vol-host-attr:host matched the backend pure3. Even though the volume was synchronously replicated to pure4 and is a full read-write volume on pure4, it would not be cloned for a new volume with a volume type backend by pure4 as the os-vol-host-attr:host value for the pure4 backend is different.
Given replicated target arrays do not have to be under the control of Cinder, this sort of makes sense, but it would be nice to be able to leverage a replicated volume on a target array that is under Cinder’s control. This would save having to have multiple copies of an image volume across multiple backend arrays. However, given the data reduction capabilities of Pure arrays, the point is probably moot anyway.
[UPDATED 5/24/24] – The above work was done with a single cinder controller. If you are running in an HA environment with multiple Active-Active cinder services, there is a known issue you should be aware of.
Notice in the cinder show output earlier that the os-vol-host-attr:host parameter starts with the actual name of the cinder controller host where the glance image was originally created from, in this case multi1. This is important as in an HA environment there is the concept of a cluster_name allowing volumes created on different controller nodes to be accessible by other controller nodes in the cluster. The internal filtering mechanism for the glance image DOES NOT use the cluster_name so this means that the glance image is selected it will only use rapid-clone if the volume request comes from the same controller node in the cluster as detailed in the image volume host attribute. Requests from other controller nodes will cause the glance image to be copied and a new image created, even if the new image is on the same backend as the glance image.
At the time of this update, this issue is being actively investigated by the core OpenStack Cinder team and when a resolution has been reached, I will add details here.
Conclusion
I hope that you found this information useful and interesting.
There are still a few more things I need to look into with regards to image import and copy image, but in general, I think that Glance multistore could be very useful once you can work out how to implement it in your deployment.
