Ofcom have released the full ISP breakdown (PDF) of the mystery shopping they carried out to check compliance with the Broadband Speeds Voluntary Code of Practice (COPBB). The main finding from the initial report was that more accurate information of broadband speed estimates were needed and work toward this would be carried out by trying to standardise how service providers report this.
Ofcom have identified within the full data where statistically significant differences have occurred between ISP's. This is where the results are significantly higher or lower than other results and are deemed to be measured with high confidence, not down to chance. The breakdown covers the ISPs with enough samples which could be deemed accurate. These were BT, O2, Orange, Plusnet, Sky, TalkTalk, Virgin DSL and 'Other' providers.
Access line speeds provided via telephone contact with ISP
BT underperformed most other providers at giving access line speeds without being prompted by the caller (28% vs 43% average). Ofcom clarify the main reason for this through the data which indicates that some services providers ask for a MAC code or other information such as bank details early in the sales process. For BT, the MAC had been asked for in 37% of cases where the mystery shopper had to prompt for speeds (average 22%). O2 provided this data in 54% of cases without being prompted, but they did did only manage to do this in 80% of calls, slightly below other providers (average 85%).
Provision of an access line speed estimate via telephone shop
Sky varied compared to other providers in this aspect by giving a range (of greater than 1Mbps difference) of speeds in 28% of cases (the average being 10%). One reason given for providing a range is that ISP's believe that line checkers used are not necessarily accurate and a range would give a more realistic expectation of speed. Sky also gave the headline speed (the speed the product is marketed at) in 21% of cases (average 10%).
Access line speed provided via ISP website
Virgin DSL products seemed to perform worst in this category giving a headline speed in 35% of cases where the average was only 8%. They also failed to give an answer in 29%. Sky again gave a range for the speed in 21% (average 5%) and no value was given in 30% of cases (average 19%). Plusnet and TalkTalk failed to give a figure in only 12% of cases. Virgin also failed to identify that the speed was an estimate in 79% of calls (average 60%). This is particularly worrying where they are giving the headline speed of the product, which is rarely (if ever) actually achievable on a DSL based broadband service.
Consistency of access line speed provided via telephone shops and ISP
The average where figures matched exactly was only 51%, but the best performing was O2 who met this in 66% of cases. Virgin DSL products had a 1Mbps difference in 48% of checks (below the average of 69%) whilst Sky's data only had an exact match in 25% of cases and averaged a 1Mbps difference in 57%.
Consistency of access line speed provided via telephone shops and BT
Wholesale line checker estimate
BT provided the most consistency with the Wholesale checker with 39% of values given matching, more than double the average of 17%. In terms of ADSL2+ speeds matching those given by the BT Wholesale checker, BT again outperformed with 46% matching compared with the average of 17%. In 74% of cases O2 reported more than 1Mbps higher than the BT Wholesale checker.
The other area investigated is based on perceptions of the ISP staff and uses a scale of 1-5 where box 1 is a poor rating, and box 5 is a good rating. BT customer agents were deemed to be less knowledgeable scoring only 30% (average 43%) in the top two boxes where callers felt the staff were knowledgeable. Plusnet came top in perceptions of helpfulness with 60% scoring the top two boxes of 5 to indicate helpfulness (average 38%). Plusnet also came top in clearly explaining that actual speeds are likely to be lower than access speeds with 37% of callers indicating this in the top 2 boxes (average 27%).
The data is interesting in areas with the full breakdown showing that some ISPs have particular failings which will need to be addressed by these service providers. Many of the large ISPs averaged similarly in most categories. Smaller ISPs will be more likely to give a more personal experience and so may excel in being knowledgeable and helpful, but the data provided indicating speeds may not necessarily be more accurate than the larger providers.