We had hoped that the problems with the BT core network were a one off but looking at social media and other data sources there appears to be packet loss somewhere in the core BT network meaning people using broadband at home and work will have another morning of frustration.
Problems started at around 8.30am on Wednesday 20th July when a power blip at a Telecity data centre in London triggered problems on the BT network, evidenced by lots of packet loss or total loss of access to anything useful on the internet. By around 11am things had started to improve and generally by midday everyone was back up and running. When things were really bad around 9.30am there was a time when almost every speed test we saw from a BT user was under 1 Mbps.
Alas the problems for users are back and it does not matter if you are paying £10.99/m or £1999/m since when the core network has issues it is likely to affect everyone with packet loss.
For business users even if you have a strong SLA the compensation for downtime may not compensate for the loss of business, so if broadband is critical to a business having two or more fully diverse connections is important. For a small SME this might just mean having a 4G dongle that can be connected to your router for emergency use, or if in a rural area a satellite broadband connection. One common solution where FTTC and ADSL2+ are available is to have FTTC as the primary connection and then an ADSL2+ provider who use a different backhaul network, e.g. if you had BT FTTC then a TalkTalk ADSL2+ service.
With the attention on Openreach two days of problems is not going to help, though it must be pointed out that the IP core network is not part of the Openreach remit and even if Openreach had rolled out nationwide FTTP by now issues like this could still happen.
The fact that the BT Home Hub for some years has had ICMP/ping responses blocked means that one diagnostic option open to people to remotely spot issues is not available, perhaps it is time for the BT Home Hub development team to consider allowing advanced users the ability to turn on ping responses to make monitoring remote works and friends and family connections easier. This is made more important since with the latest problems the 'speed test' built into the hub seemed to return the normal results and the mobile app for some was saying no problems in their area.
For clarity the problem on Wednesday was a power issue in Telecity Hex 8/9 and the problem on Thursday is a different data centre Telehouse North with reports saying there was a power failure on part of the 3rd floor.