CDN Strategy Blog

It started last year in New York and San Francisco. AT&T’s network was degrading. Phone calls dropped. At some peak times a third of all calls dropped. More and more AT&T customers throughout the U.S. started to complain about slow data, lost data connectivity and dropped calls.

When I attended Content Delivery Summit in New York this month I experienced exactly the same. Calls dropped, or I suddenly got voicemail messages without getting calls. 3G was slower than GPRS. The phone constantly switched between 3G, GPRS and Edge, and lost connections many times. The signal strength kept changing from max to a single bar. In-session switching between 3G, GPRS and Edge is a drama, because your IP address gets lost so your stream / email / surfing session gets lost too.

I’ve had the same issues in the past with KPN when they just introduced 3G. In the first 2 years, I had to switch 3G off to be sure I could be reached. It has been fixed, but in recent months I occasionally miss phone calls and get voicemail messages much later.

Last week, T-Mobile (another provider with exclusive iPhone contracts) publicly admitted having similar problems in the Netherlands.

In online forums (Dutch link, but English via Google translate), subscribers from all over the country are complaining about T-Mobile. Phone calls are dropping at peak times. Data traffic is slow, or there is no data connection at all. Phones are not ringing. Text messages don’t get through. According to T-Mobile this is due to the unexpected increase of mobile traffic usage. iPhone owners use 40 times more traffic than other subscribers.

T-Mobile has invested over 10 billion Euros (±12,5 billion dollars) in their 3G network. They face 2x as much traffic on their network as a year ago (235TB per month at this moment). T-Mobile announced that they are planning 30% additional radio towers before the end of the year. But that may not help. And it’s not just AT&T and T-Mobile…

Mobile traffic, especially video is going through the roof

Mobile operators face extreme increases in mobile traffic usage. They sell flat-fee broadband at very low rates. Which is good because we’re not paying through the nose for traffic anymore (we still do for roaming). Mobile operators sell 3G dongles for laptops. Especially those operators who exclusively sell iPhone contracts face disruptive traffic usage. Mobile operators say that over 50% of the traffic is online video and radio. They expect that this is going to be over 90% in the next years.

And these are just iPhones and other smart phones. Wait until the iPad and similar devices get into volumes. The iPhone has a limited screen. The video experience is better than with regular 3G phones, but still limited. But the iPad gives a near-HD experience that will drive mobile video consumption. This is going to kill mobile networks. They can’t shape traffic, that will kill the video performance. But I assume phone calls have to be prioritized: many public services depend on them.

The demand for mobile video content will explode in the next years. But when the networks can’t cope with the demand, there won’t be a content business case: no volume = no ads = no revenues. No QoS = no pay-per-view = no revenues.

Mobile networks extremely limited

235TB per month for a single mobile provider with millions subscribers. That’s really nothing. 235TB / month is just 750Mbps flat. A single streaming server can generate more traffic. We’ve built a 15Gbps RTSP 3GPP streaming platform many years ago, but even if there would be demand in this volume, we wouldn’t be able to distribute the content into the mobile networks. With todays iPhone HTTP streaming technologies, CDNs can push hundreds Gbps to mobile clients. Theoretically.

Mobile operator networks are very limited in capacity. First of all, they are connected via just a few 10, 100 or 1000Mbps to the internet via public and private peering links. (Ref: AMS-IX connections). Second, their backbone capacity is limited. The backhaul is the worst. Every radio tower has just ±8Mbps of backhaul capacity through the air to another tower. In some dense countries like the Netherlands radio towers are also connected via DSL or fiber, giving them a slightly better backhaul capacity. But not in Italy, Spain or other countries: 8Mbps from tower to tower, shared for all the connected subscribers including voice… A single HSDPA device can do 7.1Mbps… upgrading backhaul capacity and adding radio towers is going to cost billions. And you can’t just place radio towers every 75 meters.

Test your operators capacity

Your mobile provider may not like you doing this, but you can test the nearby radio towers capacity pretty easily. We recently did a very interesting test which you can easily replicate. All you need are 4 or more iPhones with a 3G connection (turn off wifi). We had 2 connected to KPN and 2 connected to T-Mobile. All at maximum signal strength.

The first iPhone (KPN) ran the Speedtest.net app (iTunes link). It got to ±2Mbps. Then we launched a YouTube video on the second iPhone (T-mobile). We launched the Speedtest.net app on the first iPhone again and it reported ±1Mbps. Then we launched another YouTube video on a KPN connected iPhone. The Speedtest.net app went down to ±200Kbps. Then we launched another YouTube video on the third (T-Mobile) iPhone. All videos had buffer underrun problems and the Speedtest.net app couldn’t report traffic at all…

With just a few phones we completely congested the area’s mobile networks. It wasn’t a crowded area and there were plenty radio towers nearby. Apparently KPN and T-mobile shared radio tower or backhaul capacity. It could also have been a frequencies issue, I’m not an expert on that. But it was a live situation that demonstrated actual limitations. Test this yourself! Your mileage may vary, I’m very interested to hear your results!

Shaping, Caching, CDN

Many mobile operators are already shaping traffic. They don’t do this to frustrate customers or services but to maintain the integrity of their network. Recently we see more activities focussing on caching content, to offload peers and backbones. But there is a lot of content that cannot (technically) or should not (legally) be cached. We also see more interest in getting CDNs deployed in their networks. I can’t share anything about this (NDA’s) but I am (still) working on our Telco CDN strategy white paper that covers a lot of the underlying strategical, business and technical choices. Stay tuned!