Datastore

Snapshot consolidation failed with error

/ johannstander / Leave a comment

Virtual machine was showing following alert:

Virtual machine disk consolidation is needed.  Virtual machine consolidation needed status

After trying to consolidate the virtual machine snapshots I received the following error:

Status: An error occurred while consolidating disks: Could not open/create change tracking file

Cannot complete the operation because the file or folder ds:///vmfs/volumes/*.vmdk already exists

Troubleshooting:
SSH to ESXi host and browse the volume where the VM resides on.
cd into the VM folder and found flat files for each of the vmdk disks but the VM does not show that any snapshots exists.
Tried to vMotion the VM and received same error
Tried to storage vMotion the VM and received same error
Tried creating a new snapshot and delete it again but still could not consolidate afterwards
Found the following KB 2013520 which outlines the same problems but just for committing a snapshot that already exists.
The issue seems to exist due to corrupted CTK files.  These files are associated with each delta disk and flat file for the virtual machine.
Resolution:
  • Power off the VM
  • Create a temp folder in the VM folder on the datastore
  • Move all the CTK files into the temporary folder.  The files names will look like this “*-ctk.vmdk”
  • Right click the VM and select snapshot -> Consolidate
On a side note:
After doing some further investigation I found that the there was a very large snapshots on this VM (> 150Gb) and after committing the snapshot it seems to have corrupted the CTK files.  
CTK file description
Always nice learning something new each day 🙂

The CTK files describe the VMDK characteristics/attributes of which the actual ‘data’ is stored in the in the *-flat.vmdk file.

Like in our case if you have multiple disks attached to a VM, then you’ll have multiple descriptor files that link to their actual disks.  You can actually view the contents of the file  which provides information such as chain IDs, the type of VMDK, it’s data disk, hw version, etc.
These files are also used for change block tracking (CBT) and VMware has a good KB to read on this:
Links:

Datastore unmount error: Cannot unmount volume because file system is busy

/ johannstander / Leave a comment

Just this week ran into this issue while trying to unmount some stale datastore from vCenter server.

Troubleshooting:

SSH into the ESXi host and browse the datastore.
On the datastore I found the following files listed with following names:
vsantraces*

Found KB 2069171 specifying this problem where the VSAN modules stores the traces needed to debug VSAN related problems which is keeping a lock on the datastore.

Resolution:

Since we are not using VSAN I had to disabled the VSAN services on the ESXi host to remove the datastore.

  • To verify open files for vsantrace:
    • lsof | grep vsantraced | grep volumes
  • Run command to stop services:
    • /etc/init.d/vsantraced stop
  • Perform a Refresh for Storage
  • Unmount the datastore
  • Run command to start the services: 
    • /etc/init.d/vsantraced start

If you are however using VSAN then you need to change the trace location with the following command:
esxcli vsan trace set -p

To verify open files for vsantrace
lsof | grep vsantraced | grep volumes

On a side note:

Initially the unmount of the datastore still did not exceed and this was due to my SSH sessions which has the datastore open 🙂 Close the SSH session before trying to unmount.

You could potentially still have problem with unmount when following scenario exists: 
  • Scratch for host is configured on the problem datastore. Modify the ScratchConfig value and change it to another datastore. For more information, see Creating a persistent scratch location for ESXi 4.x and 5.x (1033696).
  • Coredump is configured to write to a file in the problem datastore in an ESXi 5.5 host. For more information see, Move the coredump to a File from the VMware vSphere 5.5 documentation Center.
    • Run following command to check for paths:
      • esxcli system coredump file list
    • Verify the UUID and compare to datastore name.
    • Run following command to remove the coredump files. 
      • esxcli system coredump file remove –force
Links:

vCloud Director: Error on provider VDC "No datastores accessible to VDC "

/ johannstander / Leave a comment

This normally happens when a new datastore is added to the storage cluster which is associate to a storage policy for the vCloud Director Organization.  The rule-set (storage capability) was not properly set to the datastore before it was added to the datastore cluster.

All the datastores, datastore clusters and policies disappear and you get the following errors:

Error on provider VDC:
No datastores accessible to VDC

On Provider VDC storage policies tab:
This storage policy is not backed by any usable datastores accessible from this VDC.
There is no usable datastore accessible from this VDC that supports this storage policy. Ensure that this storage policy is backed by at least one datastore in vCenter Server, this datastore is enabled in vCloud Director, and the datastore(s) are accessible by the hosts in the clusters used by this VDC.

Solution:

Verified that all the datastores in the datastore cluster have the storage capability (Tag) assigned which is associated to the provider VDC.

Check Compliance of the VM Storage profile in vSphere Web client

  • On Home page, select Vm Storage Policies.
  • Right click the storage policy and select “Check compliance”
  • Select the VM Storage Policy
  • Verify everything compliant
  • Select Monitor tab
  • Select Matching Resources
  • Verify that the datastore cluster is showing up as well as all the datastores
Refresh storage policies for vCenter on vCloud Director System.

  • In vCloud Director, under vSphere Resources select vCenters.
  • Right click on the vCenter name and select “Refresh Storage Policies”

vSphere web client 5.5: Add new datastore to storage cluster with associate storage profile.

/ johannstander / Leave a comment

Adding a new datastore to vCloud storage profile has change just a little bit with vSphere Web client.
refer to my previous post to see the difference:
http://virtualrealization.blogspot.com/2014/01/assigning-storage-profilepolicy.html

Information to know up front:
Storage policy/profile name
Storage capability (Rule-Set 1 Tag) – vSphere Web migrated the capabilities to Tags

Steps:

Backend storage:

  1. This is normally handled by storage team which will provide the LUN.
  2. Since I am also the storage engineer I created the LUN from my Dynamic Pool on the Hitachi AMS 2100.

Within vCenter server web client:

  1. Right click the ESXi host from vCloud datastore
  2. Select “all vCenter actions”  from menu -> Rescan storage
  3. Only need to check the Scan for new Storage devices.
  4. Rick click the ESXi host again
  5. Select “New datastore”.
    1. Type – VFMS or NFS depending on block or file level storage.
    2. Datastore name
    3. Select version
    4. Partition size
  6. Go to vSphere web client home page.
  7. Select VM storage policies which will be associated to this datastore.
  8. Edit the VM storage policy, select Rule-set and write down the Tag name
  9. Right click the newly added datastore 
  10. Select “Assign Tag”
  11. Add the Tag to datastore for association to VM Storage policy.
  12. Now you can add the new datastore to the Datastore cluster.

Assigning Storage Profile/Policy: Difference between vSphere client and web client

/ johannstander / Leave a comment

The vSphere client makes use of the following terminology:
Storage profile
Storage capabilities

The vSphere web client makes use of the following terminology:
Storage policy
Rule-Set through TAGS

The main thing to take note of here is that Storage capabilities created in vSphere client is migrated over to TAGS within vSphere Web client.  These tags are then assigned to the VM Storage Policy through the rule-set.


.