Celerra / VNX File / VG Gateway and adjusting the lacp hello timers

Posted on April 29, 2013 by loopbackconnector

**Please be aware that following the steps below will require you to reboot your datamover and will disrupt traffic!!

Ok you are minding your own business and the networking guy calls you in the middle of the night during the change window and starts panicking, they had an update scheduled, it was supposed to be non-disruptive, but they are saying the ports to your datamover are causing it to not be disruptive they are sending you stuff you have never seen before:

cisco-nexus7k-01# show lacp issu-impact
For ISSU to Proceed, Check the following:
1. All port-channel member port should be in a steady state.
2. LACP rate fast should not be enabled on member ports.

The following ports are not ISSU ready
Eth6/13 , Eth4/14 , Eth5/14 , Eth6/14 , Eth4/23 ,
Eth4/24 , Eth3/13 , Eth4/13 , Eth5/13 , Eth3/14 ,

Funny a few of those ports are your datamovers LACP ports..

So what can you do to help the networking team not have a disruptive update of their expensive cisco switch?

You start digging around you pull up the “command reference for file” pdf and heck all you can see is how to adjust the failover type for the LACP trunk you have.
server_sysconfig

[protocol=lacp]
Initializes link aggregation control protocol for Ethernet devices
that allows a non-exponential number of devices. For example,
one, three, or five entries.

[lb=mac|ip|tcp]
Specifies the statistical load balancing method that applies to the
aggregator/trunk. The mac option indicates that the output link
on the trunk/aggregator is chosen based on source and
destination MAC addresses. The ip (default) option indicates that
the output link on the trunk/aggregator is chosen based on
source and destination IP addresses. This is useful in the situation
that the Data Mover is connected to a router or to a layer 3 switch.
The tcp option indicates that the output link on the
trunk/aggregator is chosen based on source and destination IP
addresses as well as source and destination TCP ports.

Great that has nothing to do with some silly hello timer or whatever they are calling it (and why would that cause their supper expensive switch to not do a non-disruptive update? (we will leave those issues to networking guys)).

Ah ha there is another document you need to reference the Parameters guide.. in there you find exactly what you need you can fix the issue and send the network guys on their way and get some sleep:

server_param server_2 -f lacp -i timeout -v

But the command fails:

server_2 :
Error 2111: server_2 : Unsupported argument: lacp

Fear not, there is a “Documentation Bug” and as per emc305565

Here is how you can update the setting and make your changes:

.server_config server_2 -v “param” |grep -i lacp

Name      Location      Current      Default
—-      ———-      ———-      ———-
lacp.timeout 0x0003f88840 ‘short’ ‘short’

To modify this parameter follow these steps:

1. Log in as nasadmin and su to root.

2. Edit file /nas/server/slot_2/param.

3. Add this entry to the file and save:

param lacp timeout=long

4. Reboot the Data Mover:

(This will disrupt traffic!!! and reboot your datamover) /nas/bin/server_cpu server_x -m -reboot now (This will disrupt traffic!!! and reboot your datamover)

After it comes back up you should be able to run:

.server_config server_2 -v “param” |grep -i lacp

and now see

Name Location Current Default
—- ———- ———- ———-
lacp.timeout 0x0003f88840 ‘long’ ‘short’

I hope some one out there finds this helpful, as when I was presented with this issue I was going ??what are you talking about..!

Posted in EMC, Storage Arrays, VNX File | 2 Comments

Brocade 8 gb how to talk when idle.. portCfgFillWord

Posted on February 14, 2013 by loopbackconnector

Are you having weird lun loss, lun connectivity issues, is your hba connecting but not logging in to the fabric? Your Brocade FC 8 gb switch may be having issues figuring out what to do when idle..

What am I talking about?? There are some known (but not overly discussed) issues with Brocade 8 gb switches.  For some reason there was a change in how hba’s and switches started dealing with idle time when they started speeding everything up to 8 gb.  Before I go to much further let me point out how to change the setting.

From the CLI you use

portCfgFillWord (portCfgFillWord <Port#> <Mode>)

Mode Link Init/Fill Word

Mode 0 IDLE/IDLE

Mode 1 ARBF/ARBF

Mode 2 IDLE/ARBF

Mode 3 If ARBF/ARBF fails, use IDLE/ARBF

Modes 2 and 3 are compliant with FC-FS-3 specifications (standards specify the IDLE/ARBF behavior of

Mode 2, which is used by Mode 3 if ARBF/ARBF fails after 3 attempts). For most environments, Brocade recommends using Mode 3, as it provides more flexibility and compatibility with a wide range of devices. In the event that the default setting or Mode 3 does not work with a particular device, contact your switch vendor for further assistance.

 Examples:

portCfgFillWord 0 3  – this changes the fill word of port 0 to mode 3

portCfgFillWord 1/19 3 – this changes the fill word of slot 1 port 19 to mode 3

How can I figure out if this applies to me? Well here are some questions to ask:

Do I have a Brocade 8 Gb switch? If yes more than likely you are affected see below how to identify what ports..

Are 16 Gb switches affected? Apparently not, the Condor3 ASIC auto detects the proper fill word.

All of my devices connected to my switch are only 4 gb am I affected? Nope, if that is a true statement.. Remember you may have 4gb HBA’s in your blades but the link aggregator is 8gb so please check.

Ok I have an 8 Gb switch and I have 8 Gb devices can I blindly change all of my ports to mode 3, I would not. Not all devices, and 4Gb support “ARBF/ARBF fails, use IDLE/ARBF” so you can look at each port and see if there are issue. Here is a quote from Brocade “However, some 8G devices are not capable of properly establishing links with Brocade 8G Fibre Channel switches when ARB/ARB or IDLE/ARB primitives are used. These 8G devices require the legacy IDLE/IDLE sequence to achieve successful link initialization.”

I look at all of my errors and I don’t see any errors are you sure that I am having issues? Well Brocade likes to spread the info across a few places.. be sure to check “portstatsshow” if you see something like this :

>portstatsshow

.  – truncated–

er_toolong              0           Frames longer than maximum

er_bad_eof              0           Frames with bad end-of-frame

er_enc_out              655692      Encoding error outside of frames

er_bad_os               1102115214  Invalid ordered set

er_rx_c3_timeout        0           Class 3 receive frames discarded due to timeout

er_tx_c3_timeout        0           Class 3 transmit frames discarded due to timeout

er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable

er_other_discard        0           Other discards

. – truncated–

You probably have issues and you just haven’t noticed. . or you are having issues any you found this post because you are having issues.

The er_bad_os should be 0, ok it can increment a few hundred  and you are fine.. these errors happen if you have a few hundred er_enc_out (to a few thousand) due to a few reboots, cable unplugs etc.

But if you are having issues er_bad_os issues it will be actively incrementing and in the 10’s of thousands+..

Ok what are all of my ports currently set to? You can run

>portcfgshow

Ports of Slot 0         0   1   2   3     4   5   6   7     8   9  10  11    12  13  14  15

———————-+—+—+—+—+—–+—+—+—+—–+—+—+—+—–+—+—+—

Speed                  AN  AN  8G  8G    8G  8G  8G  8G    AN  AN  8G  AN    AN  AN  8G  AN

Fill Word(On Active)    0   0   3   3     3   3   3   3     0   0   3   0     0   0   3   0

Fill Word(Current)      0   0   3   3     3   3   3   3     0   0   3   0     0   0   3   0

AL_PA Offset 13        ..  ..  ..  ..    ..  ..  ..  ..    ..  ..  ..  ..    ..  ..  ..  ..

Trunk Port             ON  ON  ON  ON    ON  ON  ON  ON    ON  ON  ON  ON    ON  ON  ON  ON

Long Distance          ..  ..  ..  ..    ..  ..  ..  ..    ..  ..  ..  ..    ..  ..  ..  ..

VC Link Init           ..  ..  ..  ..    ..  ..  ..  ..    ..  ..  ..  ..    ..  ..  ..  ..

Locked L_Port          ..  ..  ..  ..    ..  ..  ..  ..    ..  ..  ..  ..    ..  ..  ..  ..

. – truncated-

Ok I have found a port and made the change how can I see easily if the errors have stopped. Well run “statsclear” and check the counts again..

Can I make the changes without interruption? Sorry the “portCfgFillWord” command does cause a brief interruption in the flow of traffic.

Now there are some specific issues known out there.

IBM storage arrays: SVC, Storwize V7000 and Storwize V7000 Unified.

http://www-01.ibm.com/support/docview.wss?uid=ssg1S1003699

HP Virtual Connect

When VC 8Gb 20-port FC Module and VC FlexFabric 10Gb/24-port Module Fibre Channel uplink ports are configured to operate at 8Gb speed and connect to HP B-series (Brocade) Fibre Channel SAN switches, the minimum supported version of the Brocade Fabric OS (FOS) is v6.4.x. In addition, the Fill Word on those switch ports must be configured with option Mode 3 to prevent connectivity issues at 8Gb speed.

On HP B-series (Brocade) FC switches, use the

portCfgFillWord (portCfgFillWord <Port#> <Mode>)

command to configure this setting.

Mode Link Init/Fill Word

Mode 0 IDLE/IDLE

Mode 1 ARBF/ARBF

Mode 2 IDLE/ARBF

Mode 3 If ARBF/ARBF fails, use IDLE/ARBF

Modes 2 and 3 are compliant with FC-FS-3 specifications (standards specify the IDLE/ARBF behavior of

Mode 2, which is used by Mode 3 if ARBF/ARBF fails after 3 attempts). For most environments, Brocade recommends using Mode 3, as it provides more flexibility and compatibility with a wide range of devices. In the event that the default setting or Mode 3 does not work with a particular device, contact your switch vendor for further assistance.

Click to access c03359139.pdf

EMC       “8 GB link does not initialize to FA adapter when using ARB(FF) as fill word on Brocade switches”

EMC Primus emc223491

Here is what an official Brocade Document states in regards to 8gb best practices:

BROCADE FABRIC OS:

LINK INITIALIZATION AND FILL WORD PRIMITIVE CONFIGURATION ON 8G FC PLATFORMS FC PLATFORMS

Revision 1.0

July 26, 2010

                Best practice guideline for 8G FC platforms

For Brocade 8G platforms running FOS v6.3.1 or higher, configuring the fill word to mode 3 typically enables the port to use the right combination of Fibre Channel primitives to fully interoperate with the attached end device. Once set to mode 3, user intervention is no longer needed to configure the right primitives to enable the link between the Brocade 8G switch port and the end device.

If a device is known to require mode 2 (IDLE/ARB) then configuring mode 2 on the 8G switch port will activate the link sooner. If mode 3 is used in this environment, then the port will first attempt mode 1 then eventually settle to mode 2 which will cause a delay in initializing the port.

If a device requires IDLE/IDLE combination for link initialization then one needs to set the needed port to mode 0 using portcfgfillword CLI.

If you are changing the fill word primitives on FC long distance ports using portcfglongdistance CLI, please make sure that there is no conflict between the portcfglongdistance setting and portcfgfillword mode setting.

Please refer to the Procedure to Change Long Distance Fill Word Mode to Change Long Distance Fill Word Mode Settings section of this document for additional details.

http://h30499.www3.hp.com/hpeb/attachments/hpeb/bladescategory04/859/1/FOS%208G%20Link%20Init%20Fillword%20Behavior%20v1.pdf

Root Cause Details

The T11 fiber committee changed the FC-PI-3 standard for 8 Gb attachment on February 4, 2008. This change uses the ARB(FF) primitive signal in place of IDLE as a fill word to maintain link synchronization. During link initialization, 6 IDLE frames are sent before sending ARB(FF) at port state LR3. Slower link speeds only use the IDLE fill word and do not have this issue.

Here are some commands that can help:

General commands that are helpful in this process:

Portcfgshow – show what the current fillword is set to right now

Portstatsshow – look for high er_bad_os counts

portCfgFillWord – change the fill word to be used

statsclear – set the counters back to 0 and see if you are still having issues

##A command that can show all of the ports specified and there er_bad_os count

>portstatsshow -i 0-23 | grep “port:\|\er_bad_os”

##Script to disruptively change fillword to mode 3 for all ports on card. Repeat command for each port card slot in switch.  *note I have not tried this and copied it from the EMC primus listed above.

>for ((i=0;i<32;i++)); do (echo <slot>/$i;portcfgfillword <slot>/$i 3); done

or

>or ((slot=1;slot<13;slot++)); do (for ((port=0;port<48;port++)); do (echo $slot/$port; portcfgfillword $slot/$port 3 passive; portcfggport $slot/$port 1; portcfgeport $slot/$port 0); done); done

Note: there has been issues reporting that running a script like this and the port is not disabled first that it could cause a panic and a reload of the ports..  DEFECT000316106 see  https://community.emc.com/message/640245 for this thread.

This is one of my longer posts with lots of info from different locations. While I believe all of the info to be true please if you find errors please let me know so I can fix.  I hope people find this useful.

Posted in Brocade, FIber Switches, Storage Arrays | 4 Comments

VMware Storage Path settings for vSphere5

Posted on July 4, 2012 by loopbackconnector

A while back I wrote on VMware and ALUA path preferences.. well the info is good, it is just that with vSphere 5 they decided to change the commands around a bit.  Here are the updated commands:

Here is the command to set the default path preference (once again change VMW_SATP_ALUA_CX to your type of storage (this example is for EMC Clariion / VNX with failover type 4 (alua) enabled):

esxcli storage nmp satp set –default-psp VMW_PSP_RR –satp VMW_SATP_ALUA_CX

Ok so the preference is set, how do you change existing paths that were added before the change to the default type?

for i in `ls /vmfs/devices/disks | grep naa.6006`;

do esxcli storage nmp device set –device $i –psp VMW_PSP_RR;done

In this example you see the “grep naa.6006” 6006 is specific to EMC type storage so as this is written it will change all EMC luna presented and exclude other lun types (for example LSI storage which is naa.600a) now to get all of the luns just change the 6006 to 600 and you should be done.

Posted in Storage Arrays, VMWare | Tagged , , | Leave a comment

VMWare lun size / how many VM’s Per lun / Thin Provisioning Primer

Posted on May 1, 2012 by loopbackconnector

Being a storage admin in the age of VMWare two of the most common questions are:

  • How large should my LUNs be that I present to VMWare?
  • How many VMs should I run per lun?

These two questions tie into each other and you cannot answer one without the other. With that being said VM’s per lun is more of a rule of thumb with major influences based on Storage Array performance and VM IO considerations.

I am going to outline what has worked well for me in several different situation.  It is a flow chart because there are some things that should be accounted for.  Bottom line is there is no magical answer that will work for everyone.  What I am recommending is based off of best practices and experience.  If you know your environment by all means adjust some of the thresholds appropriately.

There is going to be two rough, good rules of thumb we are going to define (none of the numbers are concrete and you will not tear a hole in the universe if you violate these number these are just a good starting numbers):

No more than 10 VM’s per Lun.

Now lets talk about why: (the more you know the better you can fine tune your environment). The biggest reason for this is SCSI reservations, there is already a lot of info on this (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1005009), in a nut shell there are some situation where VMWare locks a lun for exclusive use and the more VM’s you have on a lun the more of the locks are potentially going to attempt to be placed on a lun, and a lot of cycles are spent on trying to acquire locks on a lun, causing delays etc.  Another is IO (now this is a highly volatile number and is 100% dependent on what your VM is used for (if your VM IO is too high this will reduce the number of VM’s you want on that LUN) but for average VM usage with typical VM’s 10 VM’s on a typical SAN LUN is a good number to start the conversation.)

No more that 2TB Lun size

This use to be a hard number but with vSphere 5 the 2TB limit has been shattered (64TB).  Even with this ability I would try not to exceed this limit unless necessary. For one unless you are running the latest and greatest storage / storage firmware and OS your storage array may have issues dealing with larger than 2TB, and advanced features (replication, snapshots, etc) may be affected (also who wants to run disk  repairs on a lun that is larger that 2TB, really??)

First there are some variables we need to define:

avmS = How much storage does your average system take? “(Size of virtual machine’s hard disk(s)) + (size of RAM for virtual machine) + (100MB for log files per virtual machine) is the minimum space needed for each virtual machine. ” from http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003755

avmPL = How many VM’s per LUN, you can use 10 as a starting number, or lower it if you have a slow SAN, or really big VM’s etc. This is going to be average (don’t let one or two really big (or small) VM’s fully define your average)

lunBS% = Buffer Space, how much do you want free on your lun after your VM’s (this space is used for snapshots, temp move space etc) if you are using pre vSphere 4 update 2, I would not do anything less than 20% (if you have ever run a lun out of space because of a snapshot you know why) and because of how snapshots collapse see http://www.yellow-bricks.com/2010/07/05/changes-to-snapshot-mechanism-delete-all/ for more info. Now with the changes to how snapshots collapse the 20% is still a good rule, but can be fudged a bit more (aka not require as much buffer space). Other factors should include how long do you hold snapshots, and how big do your snapshots get?

Here is the math:

Lun size = avmS * avmPL + lunBS%

example:

2TB Lun (rounded up 1920 gb) = 160gb avmS * 10 avmPL + 320gb 20%

1TB Lun (rounded up 960 gb ) = 80gb avmS * 10 avmPL + 160gb 20%

Ok so are we done?  well in a simple world yes.. But our world No..

Now come the questions like what about thin provisioning?

Ok thin provisioning where? At the VM level or at the Storage array level?

I will go into details on a later post but here is my stance on it. If it is thin provisioning at the VM level we are still done (you would potentially have more free space, but what would you do with it? I would not add more VM’s to that lun as that would violate the 10 VM limit)

If you have enabled Storage array thin provisioning I personally would set your Luns to 2TB, why you might ask? Un-used space does not consume much of anything (a very small bit, and you are already taking the SAN performance hit by thin provisioning) and it actually gives more flexibility in the long run. One other note if this is pre vSphere 5, I personally have had the best flexibility when I formatted all of my luns with the 8mb block size this let me grow my systems and move systems around without worry or extra complication.

Posted in Lun Size, Thin Provisioning, VMs per Lun, VMWare | Leave a comment

VMAXe and expanding a meta

Posted on April 12, 2012 by loopbackconnector

I ran into an issue the other day while trying to extend a meta of VMAXe, I was trying to do it through the Symmetrix Management Console, I had pre-created the BCV, but I the device was not listed as a selectable BCV for protected expansion. After a bit of digging and actually getting a error message when I tried to expand the meta from the CLI I realized (and in hind site is makes sense), but you need to fully preallocate your BCV’s (and have it be presistant through reclaim or copy, that little check box below the preallocate options) now you can do this at BCV creation, or post creation the choice is yours.

Posted in SMC, VMAXe | 3 Comments

Storage Scripting

Posted on October 10, 2011 by loopbackconnector

I thought I would start a new segment and see how well it does.  The premise will be around scripting (or at least automating) some common storage commands through the CLI of different storage arrays. 

Right now the focus will be on and around the VNX, but over time it should cover HP EVA and Netapp. I am sure there will be some Brocade peppered in there as well.

So what do I use for allot of my scripting with these different arrays? Excel

I use excel to build the syntax of the scripts, use some if statements to put data in the right place and copy and paste. Now for some of my more complex commands I will filter them through word and notepad (notepad to strip off cell settings etc, and word to search and replace special characters I use to put place holders in my excel script and replace with paragraph marks).

What I am going to be presenting is how I have done it for the last several years not to say this is the best way and by all means if you have better experience doing it other ways please share.  Excel, Word and notepad are what I typically use because I don’t have to ask for any special software, it is on almost every system I have used on a daily basis and at least with the Excel workbooks I have a history of what I did, and in a worst case scenario I can rebuild in a flash (or stand up a DR site with the exact same settings very quickly as well which is a better scenario than losing all of your data and having to rebuild). Also depending on how well you keep your workbook up to date you can do storage reports based off of it..

Let’s start with creating a lot of luns on the VNX.

Why would you need to manage a lot of luns that are not one off’s in your storage array VMWare, Exchange, Database servers.  This example will be around creating a lot of luns for VMWare.

Anytime I am creating a lot of luns that I feel are going to be evenly used I personally like to specify a few static things on the lun creation.

In this example I really utilize the fill down feature of excel (start with a number and then excel auto increments the next cells going down.  In this example I am wanting to create 10 luns for VMWare and I want my even numbered luns to be associated with pool 0 and SP B and conversely odd numbered luns with pool 1 and SP A.  First I create a table with the headings (if you are going to use this table start with Name in column B (we will be adding additional headings in the future):

Name Type Capacity(gb) Pool ALU SP Tiering initialTier
VNX-VMFS-001 Thin

2040

 pool 1 1001 A autoTier optimizePool

After I have my heading and some sample data in there I can begin to use excel to build my commands.  As a formula further across in (column P) I have the excel formula of:

=”lun -create -type “&C2&” -capacity “&D2&” -sq gb -poolName “””&E2&””” -sp “&G2&” -l “&F2&” -name “””&LOWER(B2)&””” -tieringPolicy “&H2&” -initialTier ”

Now I just copy all of the settings down the rows and let the name cell auto increment down and I now have the beginning of my script to send to NaviCLI. 

Now you might be asking, I thought you wanted to load balance between the two SP’s and the two pools that you have… I do and here is the formula I have for that. 

Here lets replace E2 with: =IF(ISEVEN(RIGHT(B2,3)),”Pool 0″,”Pool 1″)

Please replace the pool names with your own if you are not using Pool 0 and Pool 1 (case matters)

Also lets replace G2 with the formula: =IF(ISEVEN(RIGHT(B2,3)),”B”,”A”)

Now you should have your luns load balanced between the two pools (if you are doing so) and if nothing else the two SP’s.

Now you might be asking can I copy and paste this as is? Not yet we need to call the navicli command which will understand all of this.

In O2 lets put our navisphere commands: =”NaviSECCli.exe -user sysadmin -Password sysadmin -Scope 0 -h VNX01SPA”

Now lets tie it all together in S2 with: =O2&” ”

And now we have a command we can copy and paste into a command prompt (of a system that has navicli installed and under a directory that can call the navicli command (typically it is something like “C:\Program Files (x86)\EMC\Navisphere CLI” and here is our complete command when we are done: “NaviSECCli.exe -user sysadmin -Password sysadmin -Scope 0 -h VNX01SPA lun -create -type Thin -capacity 2040 -sq gb -poolName “Pool 1” -sp A -l 1001 -name “vnx-vmfs-001″ -tieringPolicy autoTier -initialTier optimizePool”

Just fill the cells down in excel (have the Name cell auto increment the name) and have fun.

Later installment we will talk about adding the luns to a storage group, and dealing with multiple sites that we are replicating storage to and wanting the storage to match.

Posted in Automating, Clariion, EMC, Excel, Storage Arrays, Utilities, VNX Block | 1 Comment

Profile issues with VNX File / Celerra

Posted on September 27, 2011 by loopbackconnector

Have you set your users to use technologies like user profiles / directory synchronization, file replication etc.. and have recently moved to a EMC based NAS and virtually everyone is now taking forever to load / log on / log off / synchronize.  Conversely are you having issues using robocopy copying data to your NAS on subsequent copies (a lot of *Older* files that have not changed)

Here is what is going on:  EMC nas arrays (VNX File and Celerra) do not support NTFS time stamping, it in fact uses FAT time stamps (with a two second granularity) and not NTFS granularity which is down to +- 100 nano seconds.

Fix: if your problem is only robocopy and nothing else, you can add the /fft switch your list of switches used with robocopy (FFT is Fat File Times).

If your problem is more on the profile issue there is one setting you can change on the NAS: you can have the NAS round up the time stamp placed on the file as it rests on the NAS.  This will alleviate the *Older* files problem but it can cause a issues on the first updated sync as some of the files will be *Newer* and copied back down from the array to your client, but your client will now have the rounded time stamp on their file and all subsequent syn’s should not have an issue.

You can enable the setting of NanoRoundoff

$ server_param server_2 -f cifs -m nanoroundoff -v 1

You will have to restart CIFS on your NAS after setting the parameter to enable it.

For more information please see EMC Primus emc169068

 

Posted in Celerra, Data Migration, EMC, RoboCopy, Storage Arrays, VNX File | Leave a comment

Avamar VM Backup snapshot removal error

Posted on September 20, 2011 by loopbackconnector

Today I worked with Avamar support to work out some snapshot issues with our VM’s being backed up with Avamar Image level backup.

We would run reports looking at our outstanding snapshot in VMWare (from VMWare VI Tool Kit Power shell script “get-vm | get-snapshot | ft -Property vm,created,name”) we began to notice more and more Consolidated Snapshots hanging out, also Avmar was not cleaning up after it’s self very well.  We would attempt to manually remove some of the Avamar snapshots but we would get a error stating unable to find the file.. but the snapshot would clear out, unless we took another snapshot and the consolidated snapshot would show back up. After looking at the files we notice a lot of delta.vmdk files stacking up.

Ok enough of the problem how do we fix it.. What was happening was our Avamar Proxies were not releasing the disk that it mounted to copy the files off ( what the Avamar Proxy does it takes a snapshot of a VM, takes the original vmdk file and mounts it as another disk drive to the proxy and copies the files off, and when it is done it removed the drive, and removes the snapshot) our problem was the drive was not being removed from the proxy, so after removing the drive from proxy we were then able to clean up the snapshots without error. ( If after you remove the drive from the proxy and you had already tried to remove the snapshot and it no-longer shows in snapshot manager just take start a manual snapshot, wait a few and then delete then delete all snapshots.)

Two notes:

After you manually remove the drive from the proxy, you need to go onto the proxy and remove the folders associated with the drives that were mounted the folders reside in /tmp/vmware-root/ (they will be a long formatted string of numbers)

Avamar has acknowledged this issue and the support tech said they have a script that can be installed and run from the Avamar Utilty node, if your are having chronic issues, call support so they can work with you.

here are some commands you can use to help find which Proxy has the disk locked

get-vm * | Get-HardDisk | Where {$_.Persistence -match “IndependentNonPersistent”}|ft -Property Parent,FileName

(you can speed up your seach if you replace the * with the starting name of your proxy servers. ex “avmarproxy*” to limit the seach to just your avamar proxies)

 

Posted in Avamar, Backups, VMWare | 1 Comment

Fear Not

Posted on September 1, 2011 by loopbackconnector

Fear not the power of CLI!!

Posted in Uncategorized | Leave a comment

Robocopy issues to EMC VNX and Celerra

Posted on August 31, 2011 by loopbackconnector

Robocopy issues to EMC VNX and Celerra (possibly other NAS devices as well)

While using Robocopy

“robocopy %source% %dest% /mir /dcopy:T /copyall /secfix /timfix /r:0 /w:0 /np /FFT /log:”C:\LOGS\All.log” /tee”* to copy to a EMC VNX I would get occasional “ERROR 1307 (0x0000051B) and a few other (0x00000xxx) codes Copying NTFS Security to Destination Directory”  and the normal “are you logged in as an account with permissions” etc did not apply. After digging I found a few issues.

All of the issues are security and permission related, but at three different levels..

First issue Permissions

If there are permissions assigned to an account that is no longer resolvable (you only see a ssid and not a resolved user name) you need  to remove these permissions before copying to the Celerra.

Second issue Ownership

If the owner of theFile/Folder is no longer resolvable you will need to correct this issue before you are able to copy to Celerra.

Third issue Security Group

This issue is harder to see, as I had to bust out powershell command line to see the issue and correct. If you run in powershell against the file/folder “get-acl %fileorfolder% | Format-List”* see example..

Path   : Microsoft.PowerShell.Core\FileSystem::D:\Share\doc1.doc
Owner  : BUILTIN\Administrators
Group  : SERVER_1\None
Access : BUILTIN\Administrators Allow  FullControl
         EXAMPLE\Share-Users Allow  Modify, Synchronize
         NT AUTHORITY\SYSTEM Allow  FullControl
Audit  :
Sddl   : O:BAG:DUD:AI(A;ID;FA;;;BA)(A;ID;0x1301bf;;;S-1-5-21-1417001000-1284227000-725345000-22260)(A;ID;FA;;;SY)

Now if under Group: you get a SSID or as in this example SERVER_1\None your copy to the Celerra will fail.  Now the way that I fixed this for myself was to find a folder that did not list the ssid or None and did a “get-acl %goodfileorfolder% | set-acl -Path %Badfileorfolder%”* now please be careful with these commands as depending on your file / folder inheritance, and file folder structure you may destroy your permission with one fail swoop, so please move forward if you know what impact you are going to have.

Once that I was able to find, and resolve the issues as stated I was able to copy all of my files up to the VNX NAS datamover without issue.

 

*Please note that some of the command line references used variables, the variables are noted by the surrounding %% sign, for when you run the command yourself please either declare your variable properly or substitute your own path / file / folder name in its place.  No %% is not a proper variable call in powershell..

Posted in Celerra, Data Migration, EMC, RoboCopy, VNX File | Leave a comment