Avamar VM Backup snapshot removal error

Today I worked with Avamar support to work out some snapshot issues with our VM’s being backed up with Avamar Image level backup.

We would run reports looking at our outstanding snapshot in VMWare (from VMWare VI Tool Kit Power shell script “get-vm | get-snapshot | ft -Property vm,created,name”) we began to notice more and more Consolidated Snapshots hanging out, also Avmar was not cleaning up after it’s self very well.  We would attempt to manually remove some of the Avamar snapshots but we would get a error stating unable to find the file.. but the snapshot would clear out, unless we took another snapshot and the consolidated snapshot would show back up. After looking at the files we notice a lot of delta.vmdk files stacking up.

Ok enough of the problem how do we fix it.. What was happening was our Avamar Proxies were not releasing the disk that it mounted to copy the files off ( what the Avamar Proxy does it takes a snapshot of a VM, takes the original vmdk file and mounts it as another disk drive to the proxy and copies the files off, and when it is done it removed the drive, and removes the snapshot) our problem was the drive was not being removed from the proxy, so after removing the drive from proxy we were then able to clean up the snapshots without error. ( If after you remove the drive from the proxy and you had already tried to remove the snapshot and it no-longer shows in snapshot manager just take start a manual snapshot, wait a few and then delete then delete all snapshots.)

Two notes:

After you manually remove the drive from the proxy, you need to go onto the proxy and remove the folders associated with the drives that were mounted the folders reside in /tmp/vmware-root/ (they will be a long formatted string of numbers)

Avamar has acknowledged this issue and the support tech said they have a script that can be installed and run from the Avamar Utilty node, if your are having chronic issues, call support so they can work with you.

here are some commands you can use to help find which Proxy has the disk locked

get-vm * | Get-HardDisk | Where {$_.Persistence -match “IndependentNonPersistent”}|ft -Property Parent,FileName

(you can speed up your seach if you replace the * with the starting name of your proxy servers. ex “avmarproxy*” to limit the seach to just your avamar proxies)


Posted in Avamar, Backups, VMWare | 1 Comment

Fear Not

Fear not the power of CLI!!

Posted in Uncategorized | Leave a comment

Robocopy issues to EMC VNX and Celerra

Robocopy issues to EMC VNX and Celerra (possibly other NAS devices as well)

While using Robocopy

“robocopy %source% %dest% /mir /dcopy:T /copyall /secfix /timfix /r:0 /w:0 /np /FFT /log:”C:\LOGS\All.log” /tee”* to copy to a EMC VNX I would get occasional “ERROR 1307 (0x0000051B) and a few other (0x00000xxx) codes Copying NTFS Security to Destination Directory”  and the normal “are you logged in as an account with permissions” etc did not apply. After digging I found a few issues.

All of the issues are security and permission related, but at three different levels..

First issue Permissions

If there are permissions assigned to an account that is no longer resolvable (you only see a ssid and not a resolved user name) you need  to remove these permissions before copying to the Celerra.

Second issue Ownership

If the owner of theFile/Folder is no longer resolvable you will need to correct this issue before you are able to copy to Celerra.

Third issue Security Group

This issue is harder to see, as I had to bust out powershell command line to see the issue and correct. If you run in powershell against the file/folder “get-acl %fileorfolder% | Format-List”* see example..

Path   : Microsoft.PowerShell.Core\FileSystem::D:\Share\doc1.doc
Owner  : BUILTIN\Administrators
Group  : SERVER_1\None
Access : BUILTIN\Administrators Allow  FullControl
         EXAMPLE\Share-Users Allow  Modify, Synchronize
         NT AUTHORITY\SYSTEM Allow  FullControl
Audit  :
Sddl   : O:BAG:DUD:AI(A;ID;FA;;;BA)(A;ID;0x1301bf;;;S-1-5-21-1417001000-1284227000-725345000-22260)(A;ID;FA;;;SY)

Now if under Group: you get a SSID or as in this example SERVER_1\None your copy to the Celerra will fail.  Now the way that I fixed this for myself was to find a folder that did not list the ssid or None and did a “get-acl %goodfileorfolder% | set-acl -Path %Badfileorfolder%”* now please be careful with these commands as depending on your file / folder inheritance, and file folder structure you may destroy your permission with one fail swoop, so please move forward if you know what impact you are going to have.

Once that I was able to find, and resolve the issues as stated I was able to copy all of my files up to the VNX NAS datamover without issue.


*Please note that some of the command line references used variables, the variables are noted by the surrounding %% sign, for when you run the command yourself please either declare your variable properly or substitute your own path / file / folder name in its place.  No %% is not a proper variable call in powershell..

Posted in Celerra, Data Migration, EMC, RoboCopy, VNX File | Leave a comment

VNX trespassed luns command line

EMC Command line find and remedy trespassed luns (this is being run on a EMC VNX 7500)

To list all of the luns that are trespassed on your array run (example output below)

NaviSECCli.exe -h emcvnx7500spa.example.com -user sysadmin -password sysadmin -scope 0 getlun -trespass
Default Owner:              SP B
Current owner:              SP A

Ok so now you want to fix the trespass how do you do that?

NaviSECCli.exe -h emcvnx7500spa.example.com -user sysadmin -password sysadmin -scope 0 trespass mine
NaviSECCli.exe -h emcvnx7500spb.example.com -user sysadmin -password sysadmin -scope 0 trespass mine

So what is that last command doing and why am I running it twice?

The trespass command “Trespasses one or more LUNs to the control of an SP” and the “mine” option “Performs whatever trespassing is necessary to take all of the LUNs that have this SP as
a default owner.” Since you might have luns failed over from both SP’s you run it on each sp to have that SP grab control back of its own lun.

Additional note you can run this command from your Celerra or VNX File /nas/sbin/navicli -h emcvnx7500spb.example.com getlun -trespass

Posted in Clariion, EMC, Storage Arrays, VNX Block | 3 Comments

VNX and Brocade Loop Port

I have been working with two new EMC VNX’s and ran into a odd problem and could not find anything about the issue, so I am going to post what I know and see if anyone has any more in-depth knowledge to share about the problem.

After wiring up the fiber ports of the VNX to the Brocade switches I notice that not all of the ports “Logged in” looking a bit farther I noticed that some of the ports were labled as Loop ports.. well that was odd, as the VNX should show as a (F) port and not a Loop (L) port.  I dug around EMC’s power link, Brocades forums and could not find anyone else mentioning this problem.. Long story short I ended up setting the prefrences on the individual ports on the switch and forced their speed, and port type to (F) port only.  After doing that and the typical warning from Brocade that changing the settings will disrupt traffic (how can you disrupt no traffic??) anyway after forcing the port settings on the switch all worked as expected.

Just in case you are thinking it is a model issue I experienced the same issues with:

EMC VNX 7500 connecting to two Brocade DCX’s 8gb cards and

EMC VNX 57oo connecting to two Brocade SilkWorm 4100’s

If you experience similar issues please post and and maybe get a answer to this issue.

Posted in Brocade, EMC, FIber Switches, Storage Arrays | 8 Comments

VMWare ALUA path preference

Updating some path types the other day in VMware for a ALUA compliant storage array, came across


Duncan wrote the post very well so as I was running some of the commands I noticed that one of them appeared to be EMC CX specific, so I was wondering what is the proper setting for my EVA that I was running the commands against and for that matter what is the proper setting for other arrays.  Here is the command that sets the default Path Selection Policy (PSP) for the array type / luns that you will be adding in the future. You will want to run this command to have all of the hard work you just completed be carried through when you add additional luns, anyway here is the command that Duncan ran: 

esxcli nmp satp setdefaultpsp –satp VMW_SATP_ALUA_CX –psp VMW_PSP_RR

This command works great if you are running an ALUA compliant EMC Clariion CX storage array.

but if you are running a different storage array other than an EMC Clariion CX what do you substitute, “VMW_SATP_ALUA_CX”, with?

Well to se a full list of predefined plugins you can run:

esxcli nmp satp listrules

that will give you a list and in my case I should use “VMW_SATP_ALUA” and for that matter it looks like this is the setting you use for all ALUA compliant systems other than the Clariion CX.. So that brings up the question what is special about Clariion ALUA? Maybe a different post….

Posted in Storage Arrays, VMWare | Leave a comment

Identify storage performance issues with esxtop

Taken from http://kb.vmware.com/kb/1008205

  • Start esxtop by typing ‘esxtop’ at the command line
  • Press ‘d’ to switch to disk view (HBA mode.
  • Press s, then 2 to alter the update time to every 2 seconds and press Enter.

Now I have used esxtop from time to time and unless you have a good baseline the numbers can be a bit overwhelming.. Now there is a golden nugget of info in this KB article..

“All arrays perform differently, however DAVG/cmd, KAVG/cmd, and GAVG/cmd should not exceed more than 10 milliseconds (ms) for sustained periods of time.”


Posted in Perfomance, VMWare | Leave a comment