T-Mobile NL routed all internet traffic through Germany and broke the Internet for small firms.

T-Mobile NL has had an exceptionally bad week. It should have been a jubilant week. It should have been the week of €50 for unlimited mobile and 1Gbps fixed combined (!). An excellent idea that launched Tuesday and for which the press started calling on Monday. It was supposed to show of the CEO’s Søren Abildgaard’s skills. Instead it highlighted how T-Mobile had broken much of the Dutch Internet, for its customers only, by routing all its traffic both fixed and mobile through Germany and ending all interconnections in Amsterdam (direct or on AMS-IX). This didn’t affect Netflix and Google much. It did break everything else for T-Mobile customers. That else is all that runs Dutch society and pays the bills for thousands of small firms. (BTW here is a full explanation on how Peering and Transit works)

In theory there shouldn’t have been a problem

Now in theory routing all the IP-traffic of fixed and mobile users shouldn’t have been a problem. Despite the size of our ego and the consistent positioning of the Netherlands in the top 10 of the world’s countries, and at the top of peering, AS-numbers, IP-addresses and Internet use per capita, the country is very small and insignificant when it comes to real world (telecom use). It is much smaller than most German Bundesländer (states). Much closer to Bonn and Frankfurt than most of Germany is. A few milliseconds is at most the effect this rerouting of traffic should have had and the amount of traffic should have spiked Deutsche Telekom’s German traffic stats only mildly. Why was it such a big problem and did it piss off so many customers? (one of the reasons is Germans just don’t use the Internet that much, because it is expensive, crappy and not available everywhere)

What happened?

A quick recap of what happened. Around October 24, 2019 T-Mobile NL stopped exchanging Internet traffic in the Netherlands and routing all traffic to Germany. For mobile traffic they likely had done this already since July or August and though some of its customers had noticed a degradation in performance it didn’t look too serious, just annoying. This all changed after Friday. All of a sudden for T-Mobile fixed customers in NL (then offering up to 750Mbps symmetric FTTH) performance to many sites dropped. Packet loss spiked. Some DNS services, such as 1.1.1.1, showed losses of 30%! One of my colleagues has a son at Leiden University and that son complained to his father that he couldn’t access university servers anymore from home. It went the same for other people who studied from home and who couldn’t access online video, because the stream kept bufffering.

Pulling out of AMS-IX (and Dutch interconnects)

It appears that what T-Mobile had done was the equivalent of a no-deal Brexit. It had told its peering partners in the Netherlands that it wanted out of The Netherlands and out of AMS-IX. It was going to keep a token presence of 20Gbps instead of 200Gbps it had for T-Mobile Thuis (former Glasoperator) or something as most of it bypasses the IX to direct interconnects. From now on the Dutch would have to deal with Germany. Unfortunately it had forgotten to renegotiate a deal with the Dutch Internet firms so that they were still accessible from Germany. Last Friday several 100gbps of traffic hit the Germany border routers of Deutsche Telekom AG in Germany…. And there was nothing there to deliver it to. Whatever there was, was saturated. And all of a sudden the (mobile) Internet stopped.

It likely didn’t affect Google and Netflix. Despite these two making up roughly 66% of peak traffic in the Netherlands at 21:00 they have direct interconnects and private peerings. To some extent the same likely goes for Akamai. The majority of traffic may have looked alright. But why then, with the bulk of traffic facing no problems, did the Dutch still complain? I think this has to do with the structure of the Dutch Internet compared to the German Internet.

The Dutch Internet vs the German Internet

In Germany everybody knows you need to pay Deutsche Telekom one way or another for Internet traffic you send or receive from their network. You can be stubborn and principled and not try to do it, but the only ones who complain are your customers. Deutsche Telekom is running saturated links to most of the rest of the world, unless you become their customer or a customer of their customers and then they provide adequate connectivity. This may or may not be true for Google and Netflix, but it is certainly true for smaller firms and ISPs. You could call this extortion, but a mammoth like Deutsche Telekom still lives in the ice ages and you can’t block or turn them. Bow down, know your place and pay.

The Dutch Internet scene resembles much more its hippie and cooperative history. The Dutch have always cooperated to keep the water out and were quick to start common Internet Exchanges, such as AMS-IX (and NDIX where I worked for 18 months). And one of the things the Dutch did was hook up a whole lot of non-telecom entities to the Internet and Internet Exchanges. Where the Germany DE-CIX is focused on large telecom and hosting firms, AMS-IX has a whole lot of smaller organisations as members too. What to think of municipalities Heerlen and Den Haag, Here (SatNav), ICT consulting firm CGI, the Central government shared service center Logius, networking consultancy and gin distillery Meanie, Coloclue association, Dutch Rail, the Dutch public broadcasters, RTL and the Dutch lawful intercept service platform association (NBIP), the Surfnet academic network and a whole bunch of hosting and VoIP providers. In the Netherlands just anyone would get an AS, an IP-range and hook up to AMS-IX. All these networks very irresponsibly failed to call their favorite Deutsche Telekom Sales person in Germany to purchase massive amounts of traffic and interconnection capacity. Stunning irresponsible (the Germans would think).

All of a sudden, all the traffic that had magically flowed over or near AMS-IX ceased to flow. The focus of academics and others is always on how this affects Google and Netflix and forces them to pay up… the reality was in all the smaller networks. And unlike Google and Netflix that were hardly affected because DT actually has connections to these networks, these smaller networks provide essential services that really, really need to work for their customers. If you can’t access certain services at the central government. If Dutch Rail is hurt people can’t access their timetable. No Surfnet means no more access to homework. All those hosting providers provide access to intranets, shared drives, accounting applications and private applications for thousands of small and medium enterprises. There is certain stuff that pays the bill and has to happen and that stuff isn’t on Netflix’s servers.

The fall-out

Quite quickly after stuff broke down, the Dutch Internet started to complain in the forums of T-Mobile and online community Tweakers.net. Erik Bais of A2B Internet wrote a very informative post on what had happened with great graphics explaining it. T-Mobile had been trying to get journalists to write about them on Tuesday and journalists did… about the crappy quality that users were experiencing. The Dutch equivalent of the BBC (NOS) and other press such as the financial newspaper wrote about it.

The CEO was cornered on almost any forum and even deleted some of his posts, such as this one, where he said they would increase capacity but stick with the policy

Later on Tuesday the whole policy was reversed and T-Mobile connected to AMS-IX again. At least.. the fixed part.. customers are still complaining about mobile.

IP-Interconnection isn’t about big networks, it’s about small ones

Too often the discussion about interconnection and net neutrality is about the big firms, Google, Amazon, Facebook and Akamai. In reality it is much more about the small ones. The issue isn’t even the upstarts, the wanna be competitors of GAFA. The problems are faced by the firms and organisations that support every day life and commerce in the Netherlands. It is because the Netherlands has these firms hooked up to AMS-IX that our networks and services are more competitive than in many neighbouring countries.

What can the regulator do?

If we don’t guarantee good, low latency access from large networks to these small firms, the Dutch Internet will suffer. I am well aware that both KPN and VodafoneZiggo have moved their traffic away to private interconnects and/or NL-IX. However they did it in ways that considered not to break everything massively. It isn’t good behaviour either and we really should consider as a nation whether we want to regulator to intervene.

Vodafone Germany recently committed to having three transits unsaturated at all times. This commitment was necessary to purchase Liberty Global’s network in Germany and cover the whole country. We need to look into whether we can do the same for major networks in the Netherlands. Ideally we would have these networks hook up to AMS-IX route server voluntarily. Maybe it could be a proportional remedy to impose on market players with joint dominance; hook up to a number of route servers and keep your connections to the IXP and 3 transit providers at a maximum of 60%?? saturation. It is not that the peers that would get access this way could dump traffic on the dominant networks. The Internet doesn’t allow you to send traffic without the consent of the recipient (unless if it ‘s a Denial of Service attack) so the networks would still receive the same traffic. They would still have to purchase the same amount of interfaces, but they couldn’t fiddle with the traffic anymore.