Are you having weird lun loss, lun connectivity issues, is your hba connecting but not logging in to the fabric? Your Brocade FC 8 gb switch may be having issues figuring out what to do when idle..

What am I talking about?? There are some known (but not overly discussed) issues with Brocade 8 gb switches.  For some reason there was a change in how hba’s and switches started dealing with idle time when they started speeding everything up to 8 gb.  Before I go to much further let me point out how to change the setting.

From the CLI you use

portCfgFillWord (portCfgFillWord <Port#> <Mode>)

Mode Link Init/Fill Word

Mode 0 IDLE/IDLE

Mode 1 ARBF/ARBF

Mode 2 IDLE/ARBF

Mode 3 If ARBF/ARBF fails, use IDLE/ARBF

Modes 2 and 3 are compliant with FC-FS-3 specifications (standards specify the IDLE/ARBF behavior of

Mode 2, which is used by Mode 3 if ARBF/ARBF fails after 3 attempts). For most environments, Brocade recommends using Mode 3, as it provides more flexibility and compatibility with a wide range of devices. In the event that the default setting or Mode 3 does not work with a particular device, contact your switch vendor for further assistance.

 Examples:

portCfgFillWord 0 3  – this changes the fill word of port 0 to mode 3

portCfgFillWord 1/19 3 – this changes the fill word of slot 1 port 19 to mode 3

How can I figure out if this applies to me? Well here are some questions to ask:

Do I have a Brocade 8 Gb switch? If yes more than likely you are affected see below how to identify what ports..

Are 16 Gb switches affected? Apparently not, the Condor3 ASIC auto detects the proper fill word.

All of my devices connected to my switch are only 4 gb am I affected? Nope, if that is a true statement.. Remember you may have 4gb HBA’s in your blades but the link aggregator is 8gb so please check.

Ok I have an 8 Gb switch and I have 8 Gb devices can I blindly change all of my ports to mode 3, I would not. Not all devices, and 4Gb support “ARBF/ARBF fails, use IDLE/ARBF” so you can look at each port and see if there are issue. Here is a quote from Brocade “However, some 8G devices are not capable of properly establishing links with Brocade 8G Fibre Channel switches when ARB/ARB or IDLE/ARB primitives are used. These 8G devices require the legacy IDLE/IDLE sequence to achieve successful link initialization.”

I look at all of my errors and I don’t see any errors are you sure that I am having issues? Well Brocade likes to spread the info across a few places.. be sure to check “portstatsshow” if you see something like this :

>portstatsshow

.  – truncated–

er_toolong              0           Frames longer than maximum

er_bad_eof              0           Frames with bad end-of-frame

er_enc_out              655692      Encoding error outside of frames

er_bad_os               1102115214  Invalid ordered set

er_rx_c3_timeout        0           Class 3 receive frames discarded due to timeout

er_tx_c3_timeout        0           Class 3 transmit frames discarded due to timeout

er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable

er_other_discard        0           Other discards

. – truncated–

You probably have issues and you just haven’t noticed. . or you are having issues any you found this post because you are having issues.

The er_bad_os should be 0, ok it can increment a few hundred  and you are fine.. these errors happen if you have a few hundred er_enc_out (to a few thousand) due to a few reboots, cable unplugs etc.

But if you are having issues er_bad_os issues it will be actively incrementing and in the 10’s of thousands+..

Ok what are all of my ports currently set to? You can run

>portcfgshow

Ports of Slot 0         0   1   2   3     4   5   6   7     8   9  10  11    12  13  14  15

———————-+—+—+—+—+—–+—+—+—+—–+—+—+—+—–+—+—+—

Speed                  AN  AN  8G  8G    8G  8G  8G  8G    AN  AN  8G  AN    AN  AN  8G  AN

Fill Word(On Active)    0   0   3   3     3   3   3   3     0   0   3   0     0   0   3   0

Fill Word(Current)      0   0   3   3     3   3   3   3     0   0   3   0     0   0   3   0

AL_PA Offset 13        ..  ..  ..  ..    ..  ..  ..  ..    ..  ..  ..  ..    ..  ..  ..  ..

Trunk Port             ON  ON  ON  ON    ON  ON  ON  ON    ON  ON  ON  ON    ON  ON  ON  ON

Long Distance          ..  ..  ..  ..    ..  ..  ..  ..    ..  ..  ..  ..    ..  ..  ..  ..

VC Link Init           ..  ..  ..  ..    ..  ..  ..  ..    ..  ..  ..  ..    ..  ..  ..  ..

Locked L_Port          ..  ..  ..  ..    ..  ..  ..  ..    ..  ..  ..  ..    ..  ..  ..  ..

. – truncated-

Ok I have found a port and made the change how can I see easily if the errors have stopped. Well run “statsclear” and check the counts again..

Can I make the changes without interruption? Sorry the “portCfgFillWord” command does cause a brief interruption in the flow of traffic.

Now there are some specific issues known out there.

IBM storage arrays: SVC, Storwize V7000 and Storwize V7000 Unified.

http://www-01.ibm.com/support/docview.wss?uid=ssg1S1003699

HP Virtual Connect

When VC 8Gb 20-port FC Module and VC FlexFabric 10Gb/24-port Module Fibre Channel uplink ports are configured to operate at 8Gb speed and connect to HP B-series (Brocade) Fibre Channel SAN switches, the minimum supported version of the Brocade Fabric OS (FOS) is v6.4.x. In addition, the Fill Word on those switch ports must be configured with option Mode 3 to prevent connectivity issues at 8Gb speed.

On HP B-series (Brocade) FC switches, use the

portCfgFillWord (portCfgFillWord <Port#> <Mode>)

command to configure this setting.

Mode Link Init/Fill Word

Mode 0 IDLE/IDLE

Mode 1 ARBF/ARBF

Mode 2 IDLE/ARBF

Mode 3 If ARBF/ARBF fails, use IDLE/ARBF

Modes 2 and 3 are compliant with FC-FS-3 specifications (standards specify the IDLE/ARBF behavior of

Mode 2, which is used by Mode 3 if ARBF/ARBF fails after 3 attempts). For most environments, Brocade recommends using Mode 3, as it provides more flexibility and compatibility with a wide range of devices. In the event that the default setting or Mode 3 does not work with a particular device, contact your switch vendor for further assistance.

Click to access c03359139.pdf

EMC       “8 GB link does not initialize to FA adapter when using ARB(FF) as fill word on Brocade switches”

EMC Primus emc223491

Here is what an official Brocade Document states in regards to 8gb best practices:

BROCADE FABRIC OS:

LINK INITIALIZATION AND FILL WORD PRIMITIVE CONFIGURATION ON 8G FC PLATFORMS FC PLATFORMS

Revision 1.0

July 26, 2010

                Best practice guideline for 8G FC platforms

For Brocade 8G platforms running FOS v6.3.1 or higher, configuring the fill word to mode 3 typically enables the port to use the right combination of Fibre Channel primitives to fully interoperate with the attached end device. Once set to mode 3, user intervention is no longer needed to configure the right primitives to enable the link between the Brocade 8G switch port and the end device.

If a device is known to require mode 2 (IDLE/ARB) then configuring mode 2 on the 8G switch port will activate the link sooner. If mode 3 is used in this environment, then the port will first attempt mode 1 then eventually settle to mode 2 which will cause a delay in initializing the port.

If a device requires IDLE/IDLE combination for link initialization then one needs to set the needed port to mode 0 using portcfgfillword CLI.

If you are changing the fill word primitives on FC long distance ports using portcfglongdistance CLI, please make sure that there is no conflict between the portcfglongdistance setting and portcfgfillword mode setting.

Please refer to the Procedure to Change Long Distance Fill Word Mode to Change Long Distance Fill Word Mode Settings section of this document for additional details.

http://h30499.www3.hp.com/hpeb/attachments/hpeb/bladescategory04/859/1/FOS%208G%20Link%20Init%20Fillword%20Behavior%20v1.pdf

Root Cause Details

The T11 fiber committee changed the FC-PI-3 standard for 8 Gb attachment on February 4, 2008. This change uses the ARB(FF) primitive signal in place of IDLE as a fill word to maintain link synchronization. During link initialization, 6 IDLE frames are sent before sending ARB(FF) at port state LR3. Slower link speeds only use the IDLE fill word and do not have this issue.

Here are some commands that can help:

General commands that are helpful in this process:

Portcfgshow – show what the current fillword is set to right now

Portstatsshow – look for high er_bad_os counts

portCfgFillWord – change the fill word to be used

statsclear – set the counters back to 0 and see if you are still having issues

##A command that can show all of the ports specified and there er_bad_os count

>portstatsshow -i 0-23 | grep “port:\|\er_bad_os”

##Script to disruptively change fillword to mode 3 for all ports on card. Repeat command for each port card slot in switch.  *note I have not tried this and copied it from the EMC primus listed above.

>for ((i=0;i<32;i++)); do (echo <slot>/$i;portcfgfillword <slot>/$i 3); done

or

>or ((slot=1;slot<13;slot++)); do (for ((port=0;port<48;port++)); do (echo $slot/$port; portcfgfillword $slot/$port 3 passive; portcfggport $slot/$port 1; portcfgeport $slot/$port 0); done); done

Note: there has been issues reporting that running a script like this and the port is not disabled first that it could cause a panic and a reload of the ports..  DEFECT000316106 see  https://community.emc.com/message/640245 for this thread.

This is one of my longer posts with lots of info from different locations. While I believe all of the info to be true please if you find errors please let me know so I can fix.  I hope people find this useful.