Enabling Broadcom FCoE

Posted on Leave a commentPosted in FCoE, Linux, RHEL 6, Storage

Following on from my previous post on the topic on BL460c Gen8 and the missing HBA, I thought I’d provide a quick script to automate the required installation of rpms and associated config changes, so here you go;

</p>
<p>echo &quot;Installing Pre-Reqs for FCoE on Broadcom NICs...&quot;</p>
<p>yum install -y lldpad lldpad-libs fcoe-utils</p>
<p>echo &quot;Configuring FCoE...&quot;</p>
<p>cd /etc/fcoe</p>
<p>DEVICES=&quot;eth2 eth3&quot;<br />
COUNT=0</p>
<p>for i in $DEVICES; do<br />
cp cfg-ethx cfg-$i<br />
sed -i 's/DCB_REQUIRED=&quot;yes&quot;/DCB_REQUIRED=&quot;no&quot;/g' cfg-$i<br />
done</p>
<p>echo &quot;Starting lldpad service...&quot;</p>
<p>/sbin/chkconfig lldpad on<br />
/sbin/service lldpad start</p>
<p>for i in $DEVICES; do<br />
lldptool set-lldp -i $i adminStatus=disabled<br />
COUNT=$[COUNT+1]<br />
done</p>
<p>ADMINSETCOUNT=`grep adminStatus /var/lib/lldpad/lldpad.conf | wc -l`</p>
<p>if [ $ADMINSETCOUNT == $COUNT ]; then<br />
echo &quot;adminStatus appears to have been set correctly&quot;<br />
else<br />
echo adminStatus has not been set correctly, you should investigate this further!&quot;<br />
echo &quot;Useful URL - http://h20564.www2.hp.com/hpsc/doc/public/display?docId=mmr_kc-0115755-3&quot;<br />
fi</p>
<p>echo &quot;Restarting services after changes made&quot;<br />
/sbin/service lldpad restart<br />
/sbin/service fcoe restart</p>
<p>echo &quot;Setting FCoE service to start on boot...&quot;<br />
/sbin/chkconfig fcoe on</p>
<p>echo &quot;Enable Ethernet interfaces to start on boot&quot;</p>
<p>for i in $DEVICES; do<br />
sed -i 's/ONBOOT=no/ONBOOT=yes/g' ifcfg-$i<br />
done</p>
<p>

The above does make the assumption that you are using eth2 and eth3 for FCoE.

HP BL460c Gen8 and the missing HBA

Posted on Leave a commentPosted in FCoE, Linux, RHEL 6, Storage, System Administration

Let me begin my tale of woe by setting the scene appropriately;

  • 4x BL460cGen8
    • 2x Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz (10 Cores)
    • 128GB RAM
    • HPFlexFabric10Gb2-Port534FLB Adapter
      • in iLO (integrated Lights Out)
  • 4x c7000 enclosure (some platinum, some not)
    • OA (Onboard Administrator) firmware version: 4.30
    • VC (Virtual Connect) firmware version: 4.30
      • A server profile has been created for each server with the Ethernet connectivity and FCoE)
      • The server profile has also been assigned to the required bay (just putting this in so there is no doubt, it is set-up as it should be)
  • RHEL (Red Hat Enterprise Linux) 6.5 and 6.6 were tested
  • HP SPP (Service Pack for ProLiant) 2014.09.0(B) installed across the board and the RHEL 6.6 Supplemental for the SPP have been applied where required
    UPDATE – HP have release SPP 2015.04.0.

The tale of woe!

I had a new requirement to set-up 4 new servers which would run RHEL 6.6 and would very much rely upon SAN based storage.

Installing RHEL was the easy part, getting the server to see it had access to FCoE (Fibre Channel of Ethernet) was more tricky, but only because there is a marked difference between the implementation of FCoE when using a Broadcom CNA (Converged Network Adapter) vs Emulex or Qlogic.

Things I checked

  • lspci – Didn’t show anything Fibre related.
  • dmesg – Equally didn’t show anything HBA or SAN related.
  • looking in the /sys/class directory structure showed no signs of the fc_host I am accustomed too.
  • A quick reboot a trawl through the BIOS (in HP speak RBSU) also showed no signs of anything SAN related.
  • Googled the issue to death
  • Confirmed that the CNA definitely was SAN compatible (you never know)
  • Logged onto the SAN switches to confirm there was connectivity between the Virtual Connect modules and the switches (it was but it showed that no blades had attempted to log into the fabric)
  • In the end I logged a ticket with HP.

The solution

So… After much head banging, and three hours describing the issue to HP first line technical bods, plus submitting (what felt like in triplicate) numerous logs from OA, VC and the server, I was finally pointed to the following URLs;

Now the above guides are all well and good assuming you haven’t followed the advice by HP tech support to upgrade to the latest firmware/drivers.  If you have done exactly as they have asked (not the you usually have much choice in the matter), then the following guide proved more useful;

One word of warning.  In the first article (Broadcom-Based CNAs…) above, when you get to step 6, It states you should run the following command;

# lldptool set-lldp-i ethY adminStatus=disabled

There is a typo and the line should be entered at the command line (as root) as follows;

# lldptool set-lldp -i ethY adminStatus=disabled[/bash

With the above you will need to replace ethY with the actual interface name.

Also:  Its not made crystal clear but you will need to make sure that you update the ifcfg-ethY files for the NICs that are using the FCoE personality so that they are brought online at boot.

Going forward

For me, I have created a script to recreate this, so should we purchase servers with this card installed again, the set-up remains consistent and documented.  That said it is still very much a manual step to enable the FCoE personality on the card.