I'm off to a patchy start on this blogging thing. There's been a lot of work (as usual) and my free time has been occupied with woodworking and organizing the garage. Today was an interesting day for me, though. Today I got to work on prepping some Avaya ERS 4800 series switches for an SPBM interoperability test with the HP 5900 switches we've requested for testing.
Our initial SPBM distribution network design relied on Avaya VSP 7000 switches, but in order to do switch clustering (a must for the level of redundancy we're going for), they require additional licensing. On top of that, the HP switches have more 10gig ports (48 vs 24), more 40gig ports (4 vs 0), OpenFlow support, software upgrades without reboot (ISSU), and much simpler cluster configuration and management.
For edge switches, though, the Avaya 4800 series switches are hard to beat. They come at a decent price point, have lifetime warranties and software updates, dual power supplies, Layer 2 SPBM, and PoE+ support. Since we'll be retaining Avaya for the edge switches, I'd like to see if there are any hangups between their SPBM implementation and HP's.
One of the big advantages of SPBM is that adding new links is essentially a plug-and-play affair. Since the switches rely on a routing protocol (ISIS) for loop prevention, adding new switches forms new adjacancies and triggers the SPF algorithm. VLANs aren't trunked, they're carried using I-SIDs that point directly to the other end edge switch. No additional configuration is needed on SPBM NNI links to carry additional VLANs or I-SIDs. This means that, in the event of a link failure between two distro switches, traffic could potentially re-route through a mesh of edge switches, further improving our redundancy and reducing downtime.
Unfortunately, either due to my ineptitude (most likely) or equipment failure, my SPBM demo is not working right. I have four switches in a series, all running SPB, all with nearly identical configurations, all running the same software version. I have a single management VLAN shared among all four switches, a laptop that's a member of that VLAN, and the VLAN is mapped to the same I-SID on all four switches. Each switch can see the neighboring switches (ISIS adjacency) and each switch can see the more distant switches (ISIS LSDB), but I can't get any connectivity from end to end. Avaya supports CFM (Continuity Fault Messages) over SPBM, so you can conduct traceroutes and pings at layer 2. I can successfully l2ping and l2traceroute from one switch to its neighbors, but I can't make it passed one hop; I have no end-to-end continuity. Similarly, my SPBM unicast FIB is empty! I have no earthly idea where my data is going, but I'm guessing it's somewhere near Bermuda.
UPDATE:
We ended up going with the Avaya VSP 7254 series switches for our backbone (48x10gb, 6x40gb, switch clustering at the base license level). They overcame a lot of the limitations I found in the VSP 7024 switches.
The issues I had with the Avaya ERS 4800 series switches was a license/feature issue, not a configuration one. While operating in ERS mode (vs VSP mode which required a different boot image), they forcibly (non-configurably) flipped a bit in the SPBM handshake that was the equivalent of a stub flag. Since the same switch with the same hardware could operate either in VSP or ERS mode, I'm not entirely sure why this flag was necessary, but I was told repeatedly that it wasn't solely a licensing thing. Regardless, the fix was to swap the switch over to VSP mode and let that flag return to its now-again-normal off mode.
Friday, October 3, 2014
Sunday, August 31, 2014
Testing Cisco REP with Avaya SPBM
We have a fairly large Metropolitan Area Network (MAN) that's shared by many customers. In its current state, the network uses an expanded star topology. We use two Avaya 8800 switches in a cluster for core switches which, unfortunately, limits our applications of Spanning Tree Protocol (STP). Since we can't run any flavor of STP over any dual-homed links from the core, we can't connect any two points of our star topology together as there wouldn't be any common loop-blocking technology to detect redundant links. Avaya offers a technology called Simple Loop Protection Protocol (SLPP), but it is distinctly lacking in features, tunability, or reconvergence. We also don't have much in the way of a distribution layer, so most of the links from our remote sites come all the way back to the core, devouring physical infrastructure along the way.
The network was designed and built long before I started working here and has grown significantly from its roots. The administrator at the time of conception was not well versed in best business practices and made some serious design mistakes which have only gotten worse as the network has grown. The whole MAN is a flat layer 2 network and the network core is one of the only places that does routing. This sin could be forgiven since routing at the distribution layer is a fairly modern idea, but there is no physical segregation or logical assignment of VLANs. There are at least 3 VLANs that are common to nearly every switch in the MAN, so a broadcast frame originating in the NE corner of town will be distributed across literally hundreds of links on its way to the SW, NW, and SE corners. One of these VLANs in particular uses a class B RFC 1918 address and has all manner of devices in it, ranging from printers to infrastructure devices to workstations to servers, both physical and virtual. It's a network admin's nightmare. To make matters worse, most device mappings, including printer mappings, are done with no regards to central management and without the aid of DNS or DHCP. This means that each printer's IP address is statically configured, then a tech reaches out to each computer that needs to print to that printer and adds the printer via IP address. This makes any significant VLAN restructuring incredibly painful and effectively means I have no option to implement redundancy using a layer 3 distribution setup.
One of our customers is in the process of converting mission-critical equipment from analog communications to digital. In the early phases of implementation, the analog communications lines are supplanted by analog-digital converters which are plugged into an ethernet network. Because of the mission critical nature of these devices, their ethernet network is kept separate from our main network, both logically and physically. They have their own switches (Cisco Industrial Ethernet (IE) 3000), their own fiber strands, their own routers, etc. This network is configured in a large ring using Multiple Spanning Tree Protocol (MSTP) for convergence and loop avoidance. This works well assuming there's never more than a single point of failure along the loop and that the loop doesn't grow too large for MSTP to converge.
One of the most interesting projects I have been tasked with has been building redundancy into the network. Due to the limitations of our network expressed above, this redundancy has to be built in at layer 2 and without using ol' faithful - Spanning Tree. This has led me down a long road of research and discovery whicih has introduced me to all sorts of cool technologies like MPLS, VPLS, MPBGP, SPBM, TRILL,, PBB, QinQ, VSS, IRF, SDN, LISP, OpenFlow, and OpenStack. I can't even begin to guess at how many hours I've spent between technical documents, Wikipedia entries, demo videos, CBT Nuggets, and e-mails with vendors. VPLS would have been a good fit for what we need except it requires a full mesh of VPLS routers which complicates expansion and makes large-scale growth nearly impossible. Trill, still in its infancy, has already been fractured by non-standard implementations (Fabric Path, QFabric, VCS, etc) and has very limited support outside of the data center. SDN is a good possibility, but without a solid standard like OpenFlow, it can be a mess, and our current switch vendor (Avaya) runs a proprietary SDN implementation that isn't OpenFlow-compatible. Even OpenFlow is still new enough that it's hard to find good training and documentation for it and most controllers have a steep learning curve. This left SPBM.
Shortest Path Bridging MAC uses a combination of MAC-in-MAC encapsulation from Provider Backbone Bridging, or PBB) and ISIS to create, in essence, a routed layer 2 network. It capitalizes on ISIS topology learning behavior so each SPBM switch learns a shortest path topology to every other SPBM switch in the network. It then translates VLANs into I-SIDs (Instance Service IDentifier) for transport. Because the source and destination backbone MAC (B-MAC, the SPBM switch's MAC as opposed to a C-MAC, a Customer MAC, the MAC of the end device) are the only ones used after the customer frame has been encapsulated into an I-SID, none of the intermediate SPBM switches (Backbone Core Bridges, BCB) need to learn the MAC addresses of customer devices, allowing the network to scale. Also, since VLAN-to-I-SID mapping is done on the SPBM switch that touches the customer equipment (Backbone Edge Bridge, BEB, a PE in MPLS terms), VLANs don't need to be trunked along all the links going back to the core. This becomes especially handy when you start thinking about reconvergence and alternate paths. If there were 100 redundant paths to the core, for full redundancy in an 802.1q network, I'd need to make sure that the customer VLAN was trunked across all 100 paths. Even though there are technologies that make this fairly simple (VTP), the reconvergence of ISIS SPF will always be quicker than waiting for STP convergence and then MAC address table updates, which might only be periodic.
I could spend pages talking about SPBM at this point, but it would detract from the original topic of this blog post, which was to focus on the interactions between Cisco's Resilient Ethernet Protocol and SPBM. Remember that customer MSTP ring network I talked about earlier? Part of the goal of this distribution network was to trim the size of the customer's ring and provide some increased resiliency. Imagine a clock. The two routers (running VRRP) are at the 5 and 6 o'clock position. There are switches at all the other numbers. If I cut the fiber at the 3 and 8 o'clock positions, I lose everything from 9 to 2. If, instead, I segment (purposeful word choice) the clock into 5 sections of 2 or 3 switches and string both ends of each segment back into an SPBM cloud, I have significantly less reliance on any part of the infrastructure, and even if I lose a segment, there's far less affected by the outage.
The problem we're left with is that STP and SPBM don't do a good job of communicating. Whereas SPBM is designed to have all active links, STP is not, and will fail spectacularly if the BPDUs aren't passed to the appropriate places. Avaya's SPBM implementation intentionally blocks STP BPDUs which means STP isn't the answer here. For a visual, let's go back to the clock. I have STP switch at the 1 and 2 o'clock position directly connected via fiber. We'll call them switches 1 and 2 respectively. 1 is connected to SPBM switch A and 2 is connected to SPBM switch B. We'll use VLAN 100 tie it to I-SID 1000. When a broadcast originates on VLAN 100 on switch 1, it's forwarded out to switch 2 and switch A. Switch A puts the broadcast on I-SID 1000 and sends it to switch B. Meanwhile, switch 2 has received the broadcast from switch 1 and also forwards it out to switch B. Switch B has no way of knowing that the broadcast it received from switch A and switch 2 are the same, so it passes both along, forwarding the traffic from switch 2 to switch A and vice versa. This process continues to repeat and you're left with a switching loop.
The fix would be a unified technology across both the SPBM cloud and the IE 3000 ring. Unfortunately, no such option currently exists that meets the customer's regulatory requirements, so we instead need something smart that can monitor either the Cisco side or the SPBM side and break the loops on its own. The best solution I came across is Cisco's REP. Using REP, you identify the boundaries of the Cisco segment as the links extending from switch 1 to switch A and from switch 2 to switch B. The metro ethernet switches then communicate across the segment, learn the endpoints, and implement their own block partway through the segment. The only thing left traversing the blocked port are the REP messages: hellos and Topology Change Notifications. Now when that same broadcast originates on VLAN 100 on switch 1, it's only sent to switch A. Switch A forwards to switch B, switch B forwards to switch 2, and that's as far as it goes.
To test this in the lab, I rigged up 3 SPBM switches and 6 REP switches. The SPBM switches (A-C) were configured in a triangle (A-B-C-A) and the REP switches (1-6) were broken up into 2 segments and dual-homed into the SPBM network, so segment 1 went A-1-2-3-B and segment 2 went B-4-5-6-C. To test the reconvergence, I used fping to send a series of pings from switch 2 to switch 5 on a 50ms interval with a 50ms timeout. I could swap around the SPBM links all day long and never got a reconvergence longer than about 250ms, which is phenomenal. Playing around with the REP links produced some interesting results. I could plug and unplug links in segment 1 without ever seeing more than 3 or 4 pings drop. When I started swapping around links in segment 2, however, I could lose as much as 7 seconds!
When REP switches receive a TCN, they flush their MAC address tables and relearn everything. That behavior doesn't carry over to the SPBM switches, leaving them thinking that a MAC address still exists on a link off of switch B when it's really only reachable via switch C now. The reason we didn't see the same failure when toying around with segment 1 is because we were constantly advertising our source MAC with every outgoing ping, so the SPBM switches learned the new location quickly.
The end result is that we met our redundancy requirements, but the reconvergence with the two technologies married together is not as great as it could be.
The network was designed and built long before I started working here and has grown significantly from its roots. The administrator at the time of conception was not well versed in best business practices and made some serious design mistakes which have only gotten worse as the network has grown. The whole MAN is a flat layer 2 network and the network core is one of the only places that does routing. This sin could be forgiven since routing at the distribution layer is a fairly modern idea, but there is no physical segregation or logical assignment of VLANs. There are at least 3 VLANs that are common to nearly every switch in the MAN, so a broadcast frame originating in the NE corner of town will be distributed across literally hundreds of links on its way to the SW, NW, and SE corners. One of these VLANs in particular uses a class B RFC 1918 address and has all manner of devices in it, ranging from printers to infrastructure devices to workstations to servers, both physical and virtual. It's a network admin's nightmare. To make matters worse, most device mappings, including printer mappings, are done with no regards to central management and without the aid of DNS or DHCP. This means that each printer's IP address is statically configured, then a tech reaches out to each computer that needs to print to that printer and adds the printer via IP address. This makes any significant VLAN restructuring incredibly painful and effectively means I have no option to implement redundancy using a layer 3 distribution setup.
One of our customers is in the process of converting mission-critical equipment from analog communications to digital. In the early phases of implementation, the analog communications lines are supplanted by analog-digital converters which are plugged into an ethernet network. Because of the mission critical nature of these devices, their ethernet network is kept separate from our main network, both logically and physically. They have their own switches (Cisco Industrial Ethernet (IE) 3000), their own fiber strands, their own routers, etc. This network is configured in a large ring using Multiple Spanning Tree Protocol (MSTP) for convergence and loop avoidance. This works well assuming there's never more than a single point of failure along the loop and that the loop doesn't grow too large for MSTP to converge.
One of the most interesting projects I have been tasked with has been building redundancy into the network. Due to the limitations of our network expressed above, this redundancy has to be built in at layer 2 and without using ol' faithful - Spanning Tree. This has led me down a long road of research and discovery whicih has introduced me to all sorts of cool technologies like MPLS, VPLS, MPBGP, SPBM, TRILL,, PBB, QinQ, VSS, IRF, SDN, LISP, OpenFlow, and OpenStack. I can't even begin to guess at how many hours I've spent between technical documents, Wikipedia entries, demo videos, CBT Nuggets, and e-mails with vendors. VPLS would have been a good fit for what we need except it requires a full mesh of VPLS routers which complicates expansion and makes large-scale growth nearly impossible. Trill, still in its infancy, has already been fractured by non-standard implementations (Fabric Path, QFabric, VCS, etc) and has very limited support outside of the data center. SDN is a good possibility, but without a solid standard like OpenFlow, it can be a mess, and our current switch vendor (Avaya) runs a proprietary SDN implementation that isn't OpenFlow-compatible. Even OpenFlow is still new enough that it's hard to find good training and documentation for it and most controllers have a steep learning curve. This left SPBM.
Shortest Path Bridging MAC uses a combination of MAC-in-MAC encapsulation from Provider Backbone Bridging, or PBB) and ISIS to create, in essence, a routed layer 2 network. It capitalizes on ISIS topology learning behavior so each SPBM switch learns a shortest path topology to every other SPBM switch in the network. It then translates VLANs into I-SIDs (Instance Service IDentifier) for transport. Because the source and destination backbone MAC (B-MAC, the SPBM switch's MAC as opposed to a C-MAC, a Customer MAC, the MAC of the end device) are the only ones used after the customer frame has been encapsulated into an I-SID, none of the intermediate SPBM switches (Backbone Core Bridges, BCB) need to learn the MAC addresses of customer devices, allowing the network to scale. Also, since VLAN-to-I-SID mapping is done on the SPBM switch that touches the customer equipment (Backbone Edge Bridge, BEB, a PE in MPLS terms), VLANs don't need to be trunked along all the links going back to the core. This becomes especially handy when you start thinking about reconvergence and alternate paths. If there were 100 redundant paths to the core, for full redundancy in an 802.1q network, I'd need to make sure that the customer VLAN was trunked across all 100 paths. Even though there are technologies that make this fairly simple (VTP), the reconvergence of ISIS SPF will always be quicker than waiting for STP convergence and then MAC address table updates, which might only be periodic.
I could spend pages talking about SPBM at this point, but it would detract from the original topic of this blog post, which was to focus on the interactions between Cisco's Resilient Ethernet Protocol and SPBM. Remember that customer MSTP ring network I talked about earlier? Part of the goal of this distribution network was to trim the size of the customer's ring and provide some increased resiliency. Imagine a clock. The two routers (running VRRP) are at the 5 and 6 o'clock position. There are switches at all the other numbers. If I cut the fiber at the 3 and 8 o'clock positions, I lose everything from 9 to 2. If, instead, I segment (purposeful word choice) the clock into 5 sections of 2 or 3 switches and string both ends of each segment back into an SPBM cloud, I have significantly less reliance on any part of the infrastructure, and even if I lose a segment, there's far less affected by the outage.
The problem we're left with is that STP and SPBM don't do a good job of communicating. Whereas SPBM is designed to have all active links, STP is not, and will fail spectacularly if the BPDUs aren't passed to the appropriate places. Avaya's SPBM implementation intentionally blocks STP BPDUs which means STP isn't the answer here. For a visual, let's go back to the clock. I have STP switch at the 1 and 2 o'clock position directly connected via fiber. We'll call them switches 1 and 2 respectively. 1 is connected to SPBM switch A and 2 is connected to SPBM switch B. We'll use VLAN 100 tie it to I-SID 1000. When a broadcast originates on VLAN 100 on switch 1, it's forwarded out to switch 2 and switch A. Switch A puts the broadcast on I-SID 1000 and sends it to switch B. Meanwhile, switch 2 has received the broadcast from switch 1 and also forwards it out to switch B. Switch B has no way of knowing that the broadcast it received from switch A and switch 2 are the same, so it passes both along, forwarding the traffic from switch 2 to switch A and vice versa. This process continues to repeat and you're left with a switching loop.
The fix would be a unified technology across both the SPBM cloud and the IE 3000 ring. Unfortunately, no such option currently exists that meets the customer's regulatory requirements, so we instead need something smart that can monitor either the Cisco side or the SPBM side and break the loops on its own. The best solution I came across is Cisco's REP. Using REP, you identify the boundaries of the Cisco segment as the links extending from switch 1 to switch A and from switch 2 to switch B. The metro ethernet switches then communicate across the segment, learn the endpoints, and implement their own block partway through the segment. The only thing left traversing the blocked port are the REP messages: hellos and Topology Change Notifications. Now when that same broadcast originates on VLAN 100 on switch 1, it's only sent to switch A. Switch A forwards to switch B, switch B forwards to switch 2, and that's as far as it goes.
To test this in the lab, I rigged up 3 SPBM switches and 6 REP switches. The SPBM switches (A-C) were configured in a triangle (A-B-C-A) and the REP switches (1-6) were broken up into 2 segments and dual-homed into the SPBM network, so segment 1 went A-1-2-3-B and segment 2 went B-4-5-6-C. To test the reconvergence, I used fping to send a series of pings from switch 2 to switch 5 on a 50ms interval with a 50ms timeout. I could swap around the SPBM links all day long and never got a reconvergence longer than about 250ms, which is phenomenal. Playing around with the REP links produced some interesting results. I could plug and unplug links in segment 1 without ever seeing more than 3 or 4 pings drop. When I started swapping around links in segment 2, however, I could lose as much as 7 seconds!
When REP switches receive a TCN, they flush their MAC address tables and relearn everything. That behavior doesn't carry over to the SPBM switches, leaving them thinking that a MAC address still exists on a link off of switch B when it's really only reachable via switch C now. The reason we didn't see the same failure when toying around with segment 1 is because we were constantly advertising our source MAC with every outgoing ping, so the SPBM switches learned the new location quickly.
The end result is that we met our redundancy requirements, but the reconvergence with the two technologies married together is not as great as it could be.
Tuesday, July 22, 2014
First!
Hey, folks! It's going to take me a little bit to get this thing up and running the way I want it to be. I'd like to be able to use this platform to discuss networking in general and will guide the blog with projects I'm currently working on and things I read about. A lot of my musings will be based on questions posted on networkengineering.stackexchange.com which is another excellent source of network engineering information.
Subscribe to:
Posts (Atom)