Virtual Connect Flexfabric Common Cabling Diagram

One of commonly asked questions is how to cable up Virtual Connect Flexfabric modules to LAN and SAN switches. It’s quite simple after you understand the module but a little confusing when you are first learning it. I created a short powerpoint deck trying to show users very common used VC cabling in real-world scenario.

You can download the deck here.

 

image

About these ads

23 thoughts on “Virtual Connect Flexfabric Common Cabling Diagram

  1. Hi, I am curious about why not choose to connect x1&x2 ports to the different SAN switches, just like Lan switches’ connection(vPC-like switches)? Is there any restriction?
    Thanks.

    • If you look at VC 3.70 user guide page 112, it has some diagrams and descriptions.

      To create a proper Virtual Connect fabric, all VC-FC or FlexFabric module uplinks that are included in the fabric must be connected to the same physical SAN fabric as shown in the following figure.
      If the VC SAN fabric uplinks are not connected to the same SAN fabric as shown in the following figure, then the fabric becomes degraded and a log message indicates that the uplinks are connected to different SAN switches.

      The WWN of the principal fabric switch indicates connectivity to the same SAN fabric in Virtual Connect.

  2. Thanks Hongjun, I noticed the section in VC 3.70 user guide which stated what you said, but the further question is what cause this happen? N-port used in VC-FC or anything else? Because if I change VC-FC module with normal FC SAN switch, it can definitely work. Thanks again.

    • Hi Dao-Ming,

      I just checked with VC team and it’s ok for you to connect same-vc-fabric uplinks to different SAN switches as long as the switched are in the same SAN fabric.

      Sorry for misleading info. The VC doc of “physical” word has led me thinking it’s the same physical switch. VC team agrees that the doc wording of “physical” is misleading and they’ll request the doc to be clarified.

  3. Your diagram is very helpful. Thanks for sharing it. Your diagram shows cabling for a single enclosure. What would the cabling look like if you were to stack 4 enclosures that utilize the FlexFabric modules?

    • If you look at my VC tech overview presentation under bookmark page, at the end of the deck, you’ll see the recommended diagram. For more stacking info, please check VC stacking guide, which you can find under bookmark page as well.

  4. Hongjung,

    Great post(s). I also watched your video and am appreciative that you have posted it.

    I’m having trouble finding detailed info on stacking. When I connect two Flex10/D together in the chassis – does something proprietary happen similar to the 3750s/3130s/3030s or is this just like connecting two normal ethernet switches together?

    I’m a network guy and still trying to figure out how to connect a chassis with 4 Flex10 switches. If this was legacy – ie using Cisco 3130 or the like- I’d stack two each and create a LAG to each stack. Is this still a supported technology? The links in your book marks mostly just show designs with two flex switches.

    The server guys do have on-site services inbound but I’m trying to get as much info out front as I can.

    • thanks for the comments.

      if you want to 4 VC modules in one enclosure, the cabling should look like Page 107 of my VC techoverview deck.

      For general stacking information, the best doc is under my VC bookmark tab “Stacking design” section.

      it depends on VC configuration/design, stacking link can carry user data traffic or internal VC control packets(proprietary).

      For port channel config, very typical config is to run 2 10G LACP links from module 1(one bundle) and another 2 10G LACP links from module 4(another seperate bundle) if you have 4 modules like below.

      1 2
      3 4

      • Hongjung,

        Thanks a lot for answering this question. Again, this site is great. Thank you very much.

        /r
        Will

  5. Hi all,

    I am currenty working on a single c7000 configuration which includes 4 VC-FF modules, I would like to know if the vertical stacking link configuration is a requirement or not.

    Plus in case the stacking link would not be strictly required I ask to myself if I may set up two different VC Domains (each on an horizontal couple of VC-FF modules) or otherwise I must end up with a single VC Domain. What are the advantages (if any) of a stacking link configuration if that should not be mandatory?

    Thank you very much for your help and best regards,
    Salvatore

    • you have to vertical stack VC ethernet modules(including VC-FF) at the moment. maybe in the future HP doesn’t require you to do that but not with current release. things might work if you don’t do vertical stacking but you’ll get a red alarm in VC manager saying your stacking link is in error and it’s not HP supported config if stacking link is not healthy.

      inside one enclosure, you can only have one VC domain.

      • Thank you very much for your reply and for your great blog, Hongjun.

        I have another question on this matter, I guess I should know the answer since you confirmed me already how the stacking link is a requirement for HP’s supportability (and for correct enclosure’s operations I may add since in your presentation you mention it’s required for VC internal control/synching), but I would like to know if that’s so even in case one’d implement network connectivity through VC-FF in bays 1-2 only and say san connectivity through VC-FF in bays 5-6 only, so that there’s no really need for any network routing inside the two VC-FF couples. I’m asking this since we are figuring out if it would be a good idea to save a port on each of the four VC-FF in our c7000 by not connecting the stacking link cable or not.

        thank you again and best regards,
        Salvatore

      • technically speaking it will work without the problem but you will still get big red VCM alarm when you login and I think sometimes when you call in HP support, they may not be willing to do further troubleshooting.

        you can use X7 port on each module for vertical stacking and leave X8 for internal horizontal stacking and that should keep VC manager happy and you still have 6 ports free to use. will it be enough ports to you?

        On Sun, Nov 24, 2013 at 11:18 PM, Blog on HP Virtual Connect and BladeSyst

  6. Hi Hongjun,

    I have another question regarding one of our c7000 enclosures hosting ESXi 5.1 blade servers in which its VC-FF are connected via 10GbE uplinks to a NetApp 3220 NAS storage serving the Vmware cluster datastores.

    A couple of NetApp best practices documents (http://www.netapp.com/us/media/tr-3749.pdf, http://www.netapp.com/nl/media/tr-4068.pdf) suggests for 10GbE ‘to disable flow control on ESXi, NetApp FAS and the switches in between’ (I confirm you the flow control setting is disabled on all of our 10GbE Cisco LAN switches).

    We have discovered that it’s not possible to disable the flow control setting at the NIC level as we usually do on other HP rackmount servers (DL series) by using the ethtool command as per the VMware KBA http://kb.vmware.com/kb/1013413:

    ethtool –pause tx off rx off

    In fact the command fails with the following error:

    ~ # ethtool –pause vmnic3 tx off rx off
    Cannot set device pause parameters: Operation not permitted

    We know we can manage the flow control setting from the VC-FF, currently our setting is Auto as follows:

    ->show advanced-networking
    PacketBufferOverallocationRatio : 1
    FlowControl : auto

    but we also know this is a domain-level setting, and we are afraid this may impact over the other non ESXi blade servers installed in the same enclosure (RHEL Oracle RAC nodes, RHEL MySQL cluster nodes, ecc…).

    This way the flow control is OFF on the uplinks (VC-FF to LAN switches) and ON on the downlinks (VC-FF to Blades NICs), I would like to know if this is a correct configuration or otherwise how to deal best with such matter.

    Best regards and happy new year,
    Salvatore

    • Hi Salvatore,

      VC default setting is “auto” for flow control and that actually means that we will NOT do flow control for uplink(Cisco Nexus default flow control is off as well) and do “auto” will server downlinks. This should be the right setting for VC unless you have particular reason to change it. Most of VC customers don’t change this setting.

      Before VC 4.01, some customers need to turn off VC flow control because they got faulty NIC firmware/driver and they are flooding false flow control packets to VC and cause VC to have performance issue. There was an advisory for it

      http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c02623029

      VC4.01/4.10 introduces a feature to disable the ports to these bad server NICs if they try to flood VC with massive pause frames(due to NIC bug) and it’s on by default so now it’s safe to keep VC default flow control setting on.

      Thanks and Happy New Year.
      Hongjun

  7. Thanks for the article! I have been looking for confirmation that I cannot connect to different San Fabrics from each bay. I did notice the degradation warning came up as I’ve been trying to solve a problem that I hope maybe you have a quick answer to.

    I have an ESXi cluster built on three bl460c blades with 2 flex fabric virtual connect modules bay 1 and 2. I’ve connected 4 FC connections from each bay to two different Fabrics. Connected to both fabrics apart from the C7000 is a Net app FAS 4020 and an hp 8100 eva which I plan to move off of shortly. the problem is that despite all the zoning looking correct, and all the WWN’s showing up as expected I cannot get VMware to see both fabric paths through. All of my Luns show up on one adapter as expected but not the other, VMware knows the HBA WWN but nothing is presented through that HBA.

    I’ve got the same issue on the C7000 outside of the ESXi hosts where I have 9 tapes I want to discover, the one path shows, the other doesn’t

    I know it sound like a brocade issue but I also have a C3000 on the same brocade which can see both fabrics.

    My question is although absolutely all of the VCM config checks out, even with HP is it possible that the second card is acting in a Fail over state? Can I change this to an Active/Active state?

    I’ve really tried to narrow this down by presenting luns from both the eva and netapp to verify that they are not at fault, but no dice, only one path working.

    Hope you have advice, thanks again for the article.

    Regards,

    Jason

    • Hi Jason,

      Both VC Modules will be Active/Active forwarding for SAN Fabrics. You should be able to see LUN through both paths. I’m assuming that you defined two SAN fabrics within VC Manager and assigned 4FC ports on VC-1 to fabric-A and 4FC ports on VC-2 to fabric-B. that should be the only configuration you need to do.

      Besides checking brocade and storage array, one thing you can check also is Emulex CNA firmware/driver, sometimes it’s CNA firmware/driver causing issues. the best place to check VMware firmware/driver recommendation is

      http://vibsdepot.hp.com/

      2nd link is current support recipe.

      Hope it helps.
      Hongjun

      • Thought Id say thanks for the link, I applied the latest patches to everything I could and also changed out the brocade switch. In this case I think despite the brocade switch looking as though all was fine, it wasn’t. Im up and running as I should be now. Thanks again!
        Jason

  8. Hi,

    link in this article, is not working anymore.
    Virtual Connect Flexfabric Common Cabling Diagram (Posted by hongjunma on July 13, 2012)

    You can download the deck here.
    Error (404) – We can’t find the page you’re looking for. Check out our Help Center and forums for help, or head back to home.

    May i ask You to repair it?

    Tnx

    Mark

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s