Skip Navigation

Deja vu as BT core network appears to suffer two days in a row
Thursday 21 July 2016 09:07:27 by Andrew Ferguson

We had hoped that the problems with the BT core network were a one off but looking at social media and other data sources there appears to be packet loss somewhere in the core BT network meaning people using broadband at home and work will have another morning of frustration.

Problems started at around 8.30am on Wednesday 20th July when a power blip at a Telecity data centre in London triggered problems on the BT network, evidenced by lots of packet loss or total loss of access to anything useful on the internet. By around 11am things had started to improve and generally by midday everyone was back up and running. When things were really bad around 9.30am there was a time when almost every speed test we saw from a BT user was under 1 Mbps.

Alas the problems for users are back and it does not matter if you are paying £10.99/m or £1999/m since when the core network has issues it is likely to affect everyone with packet loss.

For business users even if you have a strong SLA the compensation for downtime may not compensate for the loss of business, so if broadband is critical to a business having two or more fully diverse connections is important. For a small SME this might just mean having a 4G dongle that can be connected to your router for emergency use, or if in a rural area a satellite broadband connection. One common solution where FTTC and ADSL2+ are available is to have FTTC as the primary connection and then an ADSL2+ provider who use a different backhaul network, e.g. if you had BT FTTC then a TalkTalk ADSL2+ service.

With the attention on Openreach two days of problems is not going to help, though it must be pointed out that the IP core network is not part of the Openreach remit and even if Openreach had rolled out nationwide FTTP by now issues like this could still happen.

The fact that the BT Home Hub for some years has had ICMP/ping responses blocked means that one diagnostic option open to people to remotely spot issues is not available, perhaps it is time for the BT Home Hub development team to consider allowing advanced users the ability to turn on ping responses to make monitoring remote works and friends and family connections easier. This is made more important since with the latest problems the 'speed test' built into the hub seemed to return the normal results and the mobile app for some was saying no problems in their area.

For clarity the problem on Wednesday was a power issue in Telecity Hex 8/9 and the problem on Thursday is a different data centre Telehouse North with reports saying there was a power failure on part of the 3rd floor.


Posted by Chrismb 9 months ago
I'm not sure how allowing WAN ICMP response on the Home Hubs would help, since the TBBQM doesn't work with dynamic IPs.
Unless the TBBQM was upgraded to use a Dynamic DNS entry?
Posted by andrew (Favicon staff member) 9 months ago
Was not referring specifically to the BQM system, but just the simple ability for someone away from home able to just run a quick ping test see if a connection is good for packet loss or not.
Posted by TheEulerID 9 months ago
I'm not sure what you mean. The hubs don't block pings (or at least the HH5A & HH5B I have access to don't), they just don't respond to pings. But they do let them through (outbound that is). Unfortunately, many of the routers on the BT Wholesale network don't respond to pings either, which can be a bit frustrating if you do a tracert.
Posted by rorrocks 9 months ago
BT users getting a speed of less than 1Mbps ?

If you found the internet unusable at that speed then that gives you some idea what its like for many people in rural areas who still have to put with that kind of performance every single day from BT.

The only difference is that the urban service will be quickly restored whereas BT have no intention of upgrading rural lines unless they are compelled to do so, despite the massive subsidies they have been given.
Posted by comnut 9 months ago
and all those who have said in other placers...
"BT is the best, I am changing to them"
"why attack BT? I have never had problems"

etc, etc will now understand...
Posted by andrew (Favicon staff member) 9 months ago
@rorrocks This was under 1 Mbps speeds with 10 to 40% packet loss too. So different to a 1 Mbps connection that is running normally.
Posted by Northwind 9 months ago
I haven't much sympathy for anyone affected and who has been flapping about 'lack of communication'; you've chosen to be an interchangeable one-of-eight-million revenue-generating-unit instead of picking an ISP that has an incentive to provide good customer service.

Posted by BREWERDAVE 9 months ago
Wasn't just BT - I'm with Plusnet and had problems both days whilst my daughter is with Sky and couldn't log on to any banking sites on Wednesday either!
Posted by WWWombat 9 months ago
Neither outage was caused by BT.
The first was a UPS issue at Telecity, and the second a circuit breaker at Telehouse North.

Both are major data centres in London, but play significant roles in the routing of UK internet traffic as it passes between ISPs and service providers (ie website servers, etc).

The first outage, for example, affected 20% of traffic carried by LINX - the London Internet Exchange that allows for free interchange of traffic between ISPs.
Posted by comnut 9 months ago
er, plusnet is owned by BT.. and most of the lines are owned by them too...

guess who doesn't use BT lines AT ALL, and not even telecity??
Posted by ggremlin 9 months ago
is it coincidence that both happened on the hottest days of the year. (lots of heat takes more cooling, more cooling takes more power, and is less efficient since that radiators don't work as well in hot weather.)
Posted by blfamily 9 months ago
Coincidence or not but my IP Profile dropped on one of these days. Not happy.
Posted by WWWombat 9 months ago
IP Profile - Absolutely coincidence.
Hottest day - probably not a coincidence.
Posted by chris6273 9 months ago
@northwind Doesn't seem like you have a clue about the issue. It wasn't caused by BT and other providers were also affected. Many of our customers at work suffered on various ISPs. Although I feel it could have been dealt with much better by ISPs.
Posted by michaels_perry 9 months ago
We've been told for years that the internet is 'robust' and can withstand loss of several elements without affecting users connectivity.

These incidents prove that to be incorrect. No matter how critical one element is, there should be automatic alternatives if the robustness is to be believed.

You cannot blame BT nor Openreach for equipment failures that are not within their remit or control. You can look to all internet providers world-wide to improve their robustness and provide a safe and secure system that is not affected by some system/node failures anywhere.
Posted by WWWombat 9 months ago
I don't recall being told "the internet" is robust. I keep seeing it described as "best efforts".

I'm pretty sure that our residential services can expect no better.

There is, of course, a wide variety of intent behind the term "best efforts", and some will be better than others.
You must be logged in to post comments. Click here to login.