Today’s BGP leak in Brazil

Posted on by

Earlier today several people noticed network reachability problems for networks such as Twitter, Google and others. The root cause turned out to be another BGP mishap.

Between 11:09 and 11:27 UTC traffic for many large CDN was rerouted through Brazil. Below an example for the Internet’s most famous prefix 8.8.8.0/24 (Google DNS)

At 2017-10-21 11:09:59 UTC, AS33362, an US based ISP saw the path towards Google’s 8.8.8.0/24 like this:

33362 6939 16735 263361 15169

This shows the US based network AS33362, would have sent traffic to Google via 6939 (HE) to 16735 (Algar Telecom, Brazil), to 263361 infovale telecom which would have tried to delivered it to Google. The successful delivery of packets would have been unlikely, typically due to congestion which would have been the result of the increase in attracted traffic or an ACL blocking the unexpected traffic.

Below an example from a network connected to the Toronto Internet Exchange, trying to reach 8.8.8.8 at the time of the incident. The packet is routed over the Toronto Internet Exchange towards HE, which then hands it off to Algar telecom where it dies (also note the crazy high RTT of over 4secs, another sign indicating the congestion)

traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 40 byte packets
 1  torix-core1-10G (67.43.129.248)  0.031 ms  0.040 ms  0.022 ms
 2  he.ip4.torontointernetxchange.net (206.108.34.112)  27.823 ms 11.690 ms  0.403 ms
 3  100ge12-2.core1.ash1.he.net (184.105.64.45)  18.467 ms  18.546 ms 20.114 ms
 4  100ge8-2.core1.atl1.he.net (184.105.213.69)  30.181 ms  38.585 ms 30.422 ms
 5  100ge4-1.core1.mia1.he.net (184.105.213.26)  44.769 ms  44.352 ms 44.341 ms
 6  * * *
 7  * * *
 8  * * ae0-0.ptx-b.spo511.algartelecom.com.br (170.84.35.78)  4661.357 ms
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *

In this case 263361 should have never sent this BGP announcement to 16735 and 16735 should have not passed this on to 6939. It’s a clear example of a BGP leak.

Let’s look at another example for Netflix’s prefix 45.57.8.0/24 which appears to be hosting nflxso.net DNS services. This example AS path was observed by a Canadian network AS14442:
14442 174 16735 263361 2906 40027
Or another example as seen by AS1126 based out of the Netherlands:
1126 6939 16735 263361 2906 40027
In both cases the path unexpectedly traverses 263361 infovale telecom and 16735 (Algar Telecom, Brazil). The Canadian network learned this Brazilian path via Cogent, where the Dutch network saw it via HE (6939).

These are just two examples, there are plenty of other examples involving many other prefixes and ASns. In most cases the root cause appears to be related to the ‘16735 263361’ relation where 263361 announced peer routes to Algar AS16735, which then passed that along to its peers and even transit providers such as Cogent.

So if you are wondering why your tweets didn’t load, or had issues resolving DNS names, or other reachability issues between 11:09 and 11:27 UTC, this could very well have been caused by this accidental reroute through Brazil.