[CONTACT]

[ABOUT]

[POLICY]

This is a only version

Found at: raymii.org:70/Fix_inconsistent_Openstack_volumes_and_instances_from_Cinder_and_Nova_via_the_database.txt

This is a text-only version of the following page on https://raymii.org:
---
Title       : 	Fix inconsistent Openstack volumes and instances from Cinder and Nova via the database
Author      : 	Remy van Elst
Date        : 	22-12-2014
URL         : 	https://raymii.org/s/articles/Fix_inconsistent_Openstack_volumes_and_instances_from_Cinder_and_Nova_via_the_database.html
Format      : 	Markdown/HTML
---
When running Openstack, sometimes the state of a volume or an instance can be
inconsistent on the cluster. Nova might find a volume attached while Cinder says
the volume is detached or otherwise. Sometimes a volume deletion hangs, or a
detach does not work. If you've found and fixed the underlying issue (lvm,
iscsi, ceph, nfs etc...) you need to bring the database up to date with the new
consistent state. Most of the time a reset-state works, sometimes you need to
manually edit the database to correct the state. These snippets show you how.

I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!

Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.

You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $100 credit for 60 days.

_Please note that it is important to find and fix the underlying issue._ If you
for example have a volume which hangs on detaching, resetting the database is a
quick hack and not a real fix. Make sure you first fix the underlying issue and
cause before you update the database.
These examples were tested with all components on Juno and on Icehouse with
MySQL as the backing database.
_Please be extermely carefull with these examples._
### Delete an instance
Your NFS backing storage might have crashed halfway during a VM delete. You've
manually deleted all the related files (disk, config etc) and removed the VM
domain from the backing hypervisor (virsh, esxi etc). However `nova show` still
sees the VM as active (or error). A `nova reset-state --active` doesn't fix the
delete part. The following query can be used to set an instance as deleted:
    
    
    $ mysql nova_db
    > update instances set deleted='1', vm_state='deleted', deleted_at='now()'' where uuid='$vm_uuid' and project_id='$project_uuid';
    
Normaly a `nova delete $uuid` is the correct way to delete a VM.
If you want to actually delete a from the database instead of marking it as
deleted, the following queries should do that:
    
    
    $ mysql nova_db
    > delete from instance_faults where instance_faults.instance_uuid = '$vm_uuid';
    > delete from instance_id_mappings where instance_id_mappings.uuid = '$vm_uuid';
    > delete from instance_info_caches where instance_info_caches.instance_uuid = '$vm_uuid';
    > delete from instance_system_metadata where instance_system_metadata.instance_uuid = '$vm_uuid';
    > delete from security_group_instance_association where security_group_instance_association.instance_uuid = '$vm_uuid';
    > delete from block_device_mapping where block_device_mapping.instance_uuid = '$vm_uuid';
    > delete from fixed_ips where fixed_ips.instance_uuid = '$vm_uuid';
    > delete from instance_actions_events where instance_actions_events.action_id in (select id from instance_actions where instance_actions.instance_uuid = '$vm_uuid');
    > delete from instance_actions where instance_actions.instance_uuid = '$vm_uuid';
    > delete from virtual_interfaces where virtual_interfaces.instance_uuid = '$vm_uuid';
    > delete from instances where instances.uuid = '$vm_uuid';
    
### Change the compute host of a VM
A `nova migrate` or `nova resize` might have failed. The disk could be already
migrated or still on your shared storage but nova is confused. Make sure the VM
domain is only one compute node (preferably the on it came from, use `nova
migration-list` to find that out) and the backing disk/config files are also
only on one hypervisor node (lsof and tgt-adm are your friends here). The
following query changes the VM hypervisor host for nova:
    
    
    $ mysql nova_db
    > update instances set host='compute-hostname.domain',node='compute-hostname.domain' where uuid='$vm_uuid' and project_id='$project_uuid';
    
Normally a `nova migrate $vm_uuid` or a `nova resize $vm_uuid $flavor` should be
enough.
### Set a volume as detached in Cinder
Your backing cinder storage might have issues or bugs which cause `nova volume-
detach $vm_uuid $volume_uuid` to fail sometimes. It might be detached in Nova
but still have the state `Detaching` in Cinder. Make sure the VM domain has the
actual disk removed. Also check our backing storage (ceph, lvm, iscsi etc..) to
make sure it is actually detached and not in use anymore.
Try a `cinder reset-state --state available $volume_uuid` first. If that fails,
the following `cinder` mysql query sets the Cinder state to available:
    
    
    $ mysql cinder_db
    > update cinder.volumes set attach_status='detached',status='available' where id ='$volume_uuid';
    
Absolutely make sure that there is no data being written from to the volume, it
might cause data loss otherwise.
Do note that the cinder python api (`import cinderclient.v2`) also has the
`cinder.volumes.detach(volume_id)` call. You do need to write some tooling
around that.
().
### Detach a volume from Nova
Sometimes the volume is detached from Cinder but Nova still shows it as
attached. Same caution warnings as above count, make sure you check your backing
storage first to see if the volume is actually detached and not in use, data
loss otherwise.
The followng query removes the nova block device mapping:
    
    
    $ mysql nova_db
    > delete from block_device_mapping where not deleted and volume_id='$volume_uuid' and project_id='$project_uuid';
    
The correct way is, of course, `nova volume_detach $vm_uuid $volume_uuid`.
If you use `virsh` make sure you also `nova reboot --hard $vm_uuid` to rebuild
the `virsh` domain. If you don't do that, the volume might fail to attach
because `virsh` can't attach it at the mount point (`/dev/vdX`) since it thinks
it is already in use.
### Delete a volume from Cinder
It might be that a volume has an error deleting. It ends up in the
`Error_deleting` state. Try a `cinder reset-state --state available
$volume_uuid` first. If all fails, check your backing storage to see what
happened and if the volume is actually removed or not. If not, remove it. Then
you can update the cinder database to set it as deleted:
    
    
    $ mysql cinder_db
    > update volumes set deleted=1,status='deleted',deleted_at=now(),updated_at=now() where deleted=0 and id='$volume_uuid';
    
The correct way is `cinder delete $volume_uuid`.
### Word of caution
If you have these inconsistencies you have bigger problems you need to fix
instead of manually setting state and updating components. Openstack should make
that part easier, remember?
If you execute these queries wrong you can cause serious data loss!
Check your logging, set it to debug everywhere and get a reproducable scenario.
Then find a solution, report a bug, test the fix and deploy it in your test,
accept and then production environment.
   [1]: https://www.digitalocean.com/?refcode=7435ae6b8212
---
License:
All the text on this website is free as in freedom unless stated otherwise. 
This means you can use it in any way you want, you can copy it, change it 
the way you like and republish it, as long as you release the (modified) 
content under the same license to give others the same freedoms you've got 
and place my name and a link to this site with the article as source.
This site uses Google Analytics for statistics and Google Adwords for 
advertisements. You are tracked and Google knows everything about you. 
Use an adblocker like ublock-origin if you don't want it.
All the code on this website is licensed under the GNU GPL v3 license 
unless already licensed under a license which does not allows this form 
of licensing or if another license is stated on that page / in that software:
    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.
    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.
    You should have received a copy of the GNU General Public License
    along with this program.  If not, see .
Just to be clear, the information on this website is for meant for educational 
purposes and you use it at your own risk. I do not take responsibility if you 
screw something up. Use common sense, do not 'rm -rf /' as root for example. 
If you have any questions then do not hesitate to contact me.
See https://raymii.org/s/static/About.html for details.


AD: