BT Release Information about last Tuesdays OutageBT have released information about what happened last Tuesday, this was included with an apology for the problems that it caused to their broadband, BTnet direct and Dial IP customers.
The problem was a software bug that caused the core network interface cards to reset preventing data traffic flowing smoothly or in some cases no traffic at all. Part of the work to get the network back up involved an upgrade on the network cards, which while fully tested previously didnt do what was expected hence the later outage at 12:25 on Tuesday, i.e. when some people who'd made it back online got knocked off again. The network was restored at 18:35 with a stable network been reached by about 22:00. BT considers that the network was functioning again as soon as reasonably practical.
BT say that they think they understand what caused the network to fall over, it seems to have happened again for a short while on Friday. The important part is that they say they are working with their supplier to get a fully tested solution to the problem which will be implemented via normal upgrade channels, i.e. we should get notification of the upgrade date.
The North American Network Operators Group have been discussing this outage with concern as to whether it affects the US DSL rollout, since a lot of kit is similar. They identify the supplier in question as Cisco, if interested the discussion they've been having can be found here.