Global Internet Suffers Slowdown Blip After Cisco Kit Hits Routing Limit

Most of you probably won’t have noticed but a sizeable chunk of the global Internet suffered a brief period of instability and slowdown today (around 9am BST), which occurred after an error with Verizon’s autonomous systems and Cisco’s older routers caused a critical Internet routing limit to be reached. It will probably happen again too.

As Renesys says, “There was minor consternation in Internet engineering circles today, as the number of IPv4 networks worldwide briefly touched another magic “power of 2″ size limit. As it turns out, 512K (524,288 to be exact, or 2-to-the-19th power) is the maximum number of routes supported by the default TCAM configuration on certain aging hardware platforms.”

Put another way, the size of the global routing table is about 500K and this number marginally varies per provider (e.g. Level 3 in Amsterdam has a total of 497,869 routes and NTT in Chicago is sitting at 499,523). The routing table has been growing organically alongside the Internet for years, as per the chart below.

Crucially some older routers (e.g. Cisco’s CAT 6500/7600 Series), which are responsible for managing some of the routing tables, often have a default limit set at 512K entries in their memory (TCAM). So when Verizon’s system, for whatever reason, briefly chucked a few too many entries into the mix, taking this total above 515K, then that in turn caused the Internet to wobble on those older routers as they struggled to cope.

Andree Toonk of BGPmon said:

“Luckily Verizon quickly solved the de-aggregation problem, so we’re good for now. However the Internet routing table will continue to grow organically and we will reach the 512,000 limit soon again.”

The good news is that it is possible to raise the limit on Cisco’s older kit (details), although it’s rather frustrating that a known problem like this could not be tackled before it caused a problem. Admittedly a few minutes of disruption might not seem like much to you or me but, on an Internet that has to handle trillions of pounds worth of transactions and trade, even a small blip can have serious consequences. Just ask eBay, which spent a big chunk of the day offline and many suspect that the 512K fail was to blame.

The bad news is that, over the next couple of months, we’re likely to see more and more of these issues crop up as the routing table grows to expose similar failings in existing hardware. One of the problems here is that in raising the default limit for IPv4 connections you can end up sacrificing those assigned to IPv6, although in a few cases some providers have assigned more entries to IPv6 addresses than really necessary.

As renesys notes, “There’s far too much TCAM alloted to IPv6. In at least one case, 256K routes, when the current IPv6 routing table still requires fewer than 20K.” Indeed anyone who failed to future-proof their deployment and is still running this older gear probably has very little IPv6 traffic on their network anyway. So should you be worried about all this? Probably not, but it is a big cost and admin headache for the relevant network engineers.