Today I worked with Avamar support to work out some snapshot issues with our VM’s being backed up with Avamar Image level backup.

We would run reports looking at our outstanding snapshot in VMWare (from VMWare VI Tool Kit Power shell script “get-vm | get-snapshot | ft -Property vm,created,name”) we began to notice more and more Consolidated Snapshots hanging out, also Avmar was not cleaning up after it’s self very well.  We would attempt to manually remove some of the Avamar snapshots but we would get a error stating unable to find the file.. but the snapshot would clear out, unless we took another snapshot and the consolidated snapshot would show back up. After looking at the files we notice a lot of delta.vmdk files stacking up.

Ok enough of the problem how do we fix it.. What was happening was our Avamar Proxies were not releasing the disk that it mounted to copy the files off ( what the Avamar Proxy does it takes a snapshot of a VM, takes the original vmdk file and mounts it as another disk drive to the proxy and copies the files off, and when it is done it removed the drive, and removes the snapshot) our problem was the drive was not being removed from the proxy, so after removing the drive from proxy we were then able to clean up the snapshots without error. ( If after you remove the drive from the proxy and you had already tried to remove the snapshot and it no-longer shows in snapshot manager just take start a manual snapshot, wait a few and then delete then delete all snapshots.)

Two notes:

After you manually remove the drive from the proxy, you need to go onto the proxy and remove the folders associated with the drives that were mounted the folders reside in /tmp/vmware-root/ (they will be a long formatted string of numbers)

Avamar has acknowledged this issue and the support tech said they have a script that can be installed and run from the Avamar Utilty node, if your are having chronic issues, call support so they can work with you.

here are some commands you can use to help find which Proxy has the disk locked

get-vm * | Get-HardDisk | Where {$_.Persistence -match “IndependentNonPersistent”}|ft -Property Parent,FileName

(you can speed up your seach if you replace the * with the starting name of your proxy servers. ex “avmarproxy*” to limit the seach to just your avamar proxies)