Let me begin my tale of woe by setting the scene appropriately;
- 4x BL460cGen8
- 2x Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz (10 Cores)
- 128GB RAM
- HPFlexFabric10Gb2-Port534FLB Adapter
- in iLO (integrated Lights Out)
- 4x c7000 enclosure (some platinum, some not)
- OA (Onboard Administrator) firmware version: 4.30
- VC (Virtual Connect) firmware version: 4.30
- A server profile has been created for each server with the Ethernet connectivity and FCoE)
- The server profile has also been assigned to the required bay (just putting this in so there is no doubt, it is set-up as it should be)
- RHEL (Red Hat Enterprise Linux) 6.5 and 6.6 were tested
- HP SPP (Service Pack for ProLiant) 2014.09.0(B) installed across the board and the RHEL 6.6 Supplemental for the SPP have been applied where required
UPDATE – HP have release SPP 2015.04.0.
The tale of woe!
I had a new requirement to set-up 4 new servers which would run RHEL 6.6 and would very much rely upon SAN based storage.
Installing RHEL was the easy part, getting the server to see it had access to FCoE (Fibre Channel of Ethernet) was more tricky, but only because there is a marked difference between the implementation of FCoE when using a Broadcom CNA (Converged Network Adapter) vs Emulex or Qlogic.
Things I checked
- lspci – Didn’t show anything Fibre related.
- dmesg – Equally didn’t show anything HBA or SAN related.
- looking in the /sys/class directory structure showed no signs of the fc_host I am accustomed too.
- A quick reboot a trawl through the BIOS (in HP speak RBSU) also showed no signs of anything SAN related.
- Googled the issue to death
- Confirmed that the CNA definitely was SAN compatible (you never know)
- Logged onto the SAN switches to confirm there was connectivity between the Virtual Connect modules and the switches (it was but it showed that no blades had attempted to log into the fabric)
- In the end I logged a ticket with HP.
So… After much head banging, and three hours describing the issue to HP first line technical bods, plus submitting (what felt like in triplicate) numerous logs from OA, VC and the server, I was finally pointed to the following URLs;
- Broadcom-Based CNAs: How to Configure FCoE Support in RHEL 6.4 and Later with HP Virtual Connect –
The above gives you a good overview of the manual steps which must be performed.
- HP FCoE Configuration for Broadcom-Based Adapters User Guide –http://h10032.www1.hp.com/ctg/Manual/c04236477
Now the above guides are all well and good assuming you haven’t followed the advice by HP tech support to upgrade to the latest firmware/drivers. If you have done exactly as they have asked (not the you usually have much choice in the matter), then the following guide proved more useful;
- NPARs Configuration for HP Network Adapters User Guide – http://h20565.www2.hp.com/hpsc/doc/public/display?sp4ts.oid=5404529&docId=emr_na-c04408042&docLocale=en_US
One word of warning. In the first article (Broadcom-Based CNAs…) above, when you get to step 6, It states you should run the following command;
[bash]# lldptool set-lldp-i ethY adminStatus=disabled[/bash]
There is a typo and the line should be entered at the command line (as root) as follows;
[bash][/bash]# lldptool set-lldp -i ethY adminStatus=disabled[/bash
With the above you will need to replace ethY with the actual interface name.
Also: Its not made crystal clear but you will need to make sure that you update the ifcfg-ethY files for the NICs that are using the FCoE personality so that they are brought online at boot.
For me, I have created a script to recreate this, so should we purchase servers with this card installed again, the set-up remains consistent and documented. That said it is still very much a manual step to enable the FCoE personality on the card.