New Core Routers in Chicago

Note: As of July 2013 we are no longer utilizing Brocade MLXe-16 routers and have upgraded to Juniper MX480 routers due to their increased routing table capacity.

I'm sure at least some of you noticed the announcement this morning that we're going to be replacing our core routers. I figured it was worth a blog post to detail these changes and what they mean for you.

First of all, we've been using Cisco 6500 routers with the SUP720-3BXL management cards at our core, which handles all BGP with our carriers and peers as well as some customer VLANs, for many years now. These routers have worked out great and have served us well, but we're reaching the end of their useful lives for us, at least here in Chicago. There have been some incidents causing higher CPU usage levels than we'd like to see and some bugs on the BGP side, with the large number of sessions we have setup for peering and the like, and that had led us on the initial search for a replacement. It was clear the 6500 platform would not last us much longer as core routers in Chicago.

Our primary concern in looking for a new router was the processing and throughput capacity along with the density of 10 GigE ports and "future proofing." In the end, we selected the Brocade (formerly Foundry) MLXe-16, the newest member of the MLX family that isn't planned for official release until mid-September. It takes less of a footprint and less power than the CIsco 6500's, but gives us a significantly improved platform. We had also considered Juniper, and we love the Juniper OS, but we simply couldn't get the density and capacity we wanted.

Here is a comparison:

Cisco 6500

Processing for up to 400 Million Packets per Second
640 Gbit/sec of throughput
40 Gigabit backplane capacity per module
Up to 32 full line speed 10 Gigabit circuits
No support for 40 GigE or 100 GigE

Brocade MLXe

Processing for up to 4 Billion Packets per Second
Over 6 Tbit/sec of throughput
240 Gbit/sec backplane capacity per module
Up to 128 line speed Gigabit circuits with current line cards, higher density line cards are in the works
Support for 40 and 100 Gigabit Ethernet are in the works and the backplane capacity exists to support it

As you can see, the Brocade gives us roughly 10 times the capacity as well as the ability to continue growing with 40 GigE and 100 GigE modules. In addition, the line cards for the Brocade all have their own forwarding processors. Having this distributed processing means that we are much less likely to face similar CPU issues as we did with the Cisco 6500's, and that we have significantly more processing power at the router level.

The pair of MLXe-16's we're starting with each come with 8x 10 GigE ports (SFP+) and 20 GigE ports (1000BaseT) and leave us with 14 additional slots for future expansion, including the possibility of 100 GigE ports. The units also each have redundant management cards, which allow us to do software upgrades and maintenance without downtime, redundant switch fabric modules, and redundant power supplies. These are extremely redundant boxes on their own, but we believe strongly in our 100% network uptime SLA and still choose to have two of these routers in place.

Note: The picture attached is from right after we got them and were doing the initial software upgrades and base configurations. Don't worry, they won't be sitting on the floor when we turn them live!