Someone asked me yesterday about another old article I wrote around the network problems seen during the 2010 World Cup. They are rightly worried about the performance problems we should expect during next year's London Olympic Games.
I actually wrote two articles - one before the event and one afterwards. The one before the event is currently available online:
http://www.info4security.com/story.asp?storycode=4123557
... the one after the event was over is reprinted below.
How well did your network play during the World Cup?
As predicted, the 2010 FIFA World Cup was the biggest global event in web history. More people watched the matches, more tweets were posted and more pages viewed and updated than any previous event. Of course, this huge spike of traffic caused a number of business problems; reminding us of the inexorable rise in traffic, the blurring between work and personal usage, and most importantly that IT management need to plan for the next “big thing” to ensure that business can continue during popular events.
Akamai claimed a peak of 1.6million simultaneous streams, many in HD, and many broadcasters around the world delivered twice the previously-highest peak traffic. Just in the UK, the BBC delivered 800,000 streams during the England vs. Slovenia match alone and total UK Internet usage increased by over 30%. People watched on their PCs, through their mobile phones and iPads, at home, while travelling and in the workplace. The BBC statistics for June 2010 showed that there were 9.7million requests for live simulcast content in June 2010, an increase of 26% over the previous month and around a 500% increase on a year ago [source BBC iStats]. This reminds us of the growing expectation that live TV can be watched online and that if something is considered newsworthy that users will do this, even during the working day.
As with car traffic to popular events, the traffic problems are in the last few miles. It wasn’t the main Internet highways that suffered problems, but the physical bottlenecks along country roads and into car parks were replaced by the final connection to the businesses or sometimes the initial connection to the broadcaster. Some businesses found that their own Internet connections and links to branch or remote offices using their Wide Area Network (WAN) were overloaded by World Cup traffic, meaning that other data couldn’t be sent or received and business-critical applications came to a halt.
During the initial matches, a number of broadcasters had underestimated the demand – as their servers failed to deliver the quantity of requests made, leading to outages and poor quality video – Twitter and blogs were full of complaints. It’s clear that a significant demand came during business hour when employees were presumably at work. By the end of the tournament it seemed most of the complaints had subsided or perhaps viewers went back to watching on their TVs – though the later-timed matches perhaps helped the broadcasters out too.
Any popular event will entice the scamers and malware writers out from their dark corners, and the World Cup was no exception. Eight out of the top ten spam messages during June were related to the World Cup, including countless phishing ploys, and there were many Web pages trying to entice people to pay for promised online coverage (often in HD). Scam news articles promised behind the scenes footage that led to the common ploy of “update your codec here” attempts to surreptitiously download malware.
The main issue was the clearly the impact on business networks – there were stories of network traffic failures, followed by rapid emails asking everyone in an organisation to stop watching the World Cup and people saying “if you can’t beat them, join them” as they downed tools for an hour and a half during a particular match.
Just during a 30 minute period I picked up the following tweets, some about business traffic and some just complaining that watching a match was impossible:
AgentOwen: Our internet usage at the office has gone way up during World Cup. We just got a spanking as it’s slowed down our network.
Dave: Received email from IT “Don’t watch the World Cup – we’ve got business to do” – fine, I’ll go down the pub.
Speedvegan: Note to self: Do not schedule any releases while the US is playing in the World Cup. Network slows to a halt.
Flokemon: Ingerland playing, USA playing, corporate network slows down to a crawl, can I get a stream working eeeekk
Epheramaldog: Ha! The entire wi-fi network went down at precisely 3pm, funny that!
Monchote: On no!! Too many people streaming England’s game at work and the company’s network is about to go down.
PixelMagazine: everyeone in the office watching the match on their PC tends to slow the network down – give up calling tech support, it ain’t gonna happen
Mrmahoosive: If the BBC site buffers one more time I will take down our core network so only I can stream!
TvMiller: Americans turn to iPhone for understanding “offsides” call during World Cup match, AT&T network down leads to “bad call” row.
It’s no surprise that different matches caused problems in different places around the world depending on the teams playing at the time. The countries reporting the highest difficulties usually had the following characteristics:
- The country was actually in the tournament and playing at the time
- The local rights-holder delivered the matches online
- Football is a sport with a large following
- The match was taking place during the typical working day
- Typical bandwidth at work of less than 1Mbps per user
- Online broadcasting is popular, promoted by the local TV broadcaster
- Employers didn’t bring in large screens and encourage shared viewing
Various quality of broadcast were available, typically taking between 800Kbps and 3.5Mbps (HD) per screen. Of course, in a business, this fights with normal business traffic, hence the negative impact on other applications and poor quality streaming for many users who did attempt to watch. As most organisations do not have the means to minimize the effect of live video streams, such as with stream splitting, each new user accessing video adds another stream with the same network demands. In large organisations, where Internet traffic is commonly backhauled to and from the data centre across the WAN, remote offices often suffered from poor Internet access and, worse, even to centralised internal applications.
It wasn’t just internal business networks that buckled under the weight; public wireless networks were under strain causing problems for travelling users. Mobile data bandwidth usage in the USA increased by 24% and post-match mornings saw 32% incease in YouTube traffic, see this email from a colleague:
Due to recent bad weather I spent more time than I cared for at several large airports (Chicago, Washington DC). In the past few days, the wireless service was bogged down by users watching the World Cup. If anything exciting happened in the match, you could hear the shouts and fan reaction through the terminal as well. Many times I had no signal, or very limited web access to email with virtually no ability to surf the web…the impact was real and noticeable.
Perhaps, for you, the World Cup didn’t impact your network. However, it has lessons for us all – Are you ready for the next explosion in demand for popular content? These demands are happening with increasing frequency—take for example Tiger Woods press conference and President Obama’s State of the Union address in the USA for example. Local news or political announcements often create significant peaks too. Some content can be predicted, but a global live newsflash can appear at any time.
IT managers need to look at their own statistics for the World Cup and plan for the next flood of content, look at each office and every country – watching the growing popularity of streaming to predict and put in place the solutions to ensure business traffic can be delivered during popular “stream-storms.”
There are a number of different approaches that IT managers can take. If you can predict the next set of content demands, some organisations may take a strict approach by attempting to block web access to all known sites that stream the content. For example, by using web filtering systems, IT management can block access to global sports sites, though users are likely to be unhappy and may still spend time attempting to circumvent the blocking.
A second option would be to block the protocols used for streaming, however this may include all Real, Microsoft and Flash streams – and in doing so, block internal streams, streaming news and standard parts of web sites, interfering with work-related web information. This approach will not work with on-demand video clips that are generally delivered as an integral part of the Internet flow.
Instead of either of the above approaches, organisations should look to adopt a more flexible attitude. IT management can improve their network infrastructure to reduce stream usage through real-time stream-splitting, optimise streaming data or allow users to time-shift the content to be during normal breaks in the working day, as follows.
Firstly, bandwidth management devices at the Internet egress point can be set to define one stream provider as “approved” and given a high priority (management then encourage employees to use that stream), other streams are lower priority or blocked.
Secondly, appliances can be installed within the organisation’s network to split the streams – meaning that one external stream request can be sent to multiple users simultaneously. This greatly reduces the upstream bandwidth required.
Thirdly, WAN optimisation appliances that support stream splitting can be deployed between offices and at Internet connection points to take a single stream and divided as needed to serve user demand.
Fourthly, many of the stream splitting appliances can also cache the streams, allowing users to time-shift and watch the game later.
Happily, this doesn’t mean installing four different kinds of network appliances, as there are some devices can deliver multiple benefits in one.
In this way, management can allow (or even encourage) video content whilst minimizing the load on the Internet gateway or branch office by caching locally through a proxy appliance and splitting a single video stream into as many are needed to meet the demands.
There are further benefits to installing solutions to optimise streaming content. As streaming is embedded in so many business sites, the general load on the network will fall and quality of web content delivered improve. Internal streams (such as CIO broadcasts) are optimised in the same manner as external streaming, and web video-conferencing between users and customers can be enhanced.
