Broadband News

Which? publishes speed test results but are they right?

Which? has analysed its speed test results for the period September 2017 to September 2018 and published the median download speed results by local authority and the press headlines are lapping it up.

Identifying areas such as the Orkney islands (3Mbps), Allerdale (5.7Mbps), Shetland Islands (6.7Mbps), Argyll and Bute (7Mbps), Moray (7.1Mbps), Fermanagh and Omagh (7.4Mbps) and Ceredigion (7.5 Mbps) as being the slowest local authority areas of the UK is not a surprise to us and also should not be a surprise to those using our own data resources which has a list of the UK local authorities updated every quarter. Visitors to thinkbroadband and others using our broadband speed test technology for the same 7 areas recorded the following median download speed Orkney Islands 11.1 Mbps, Allerdale 22.3 Mbps,  Shetland Islands 12.3 Mbps, Argyll and Bute 15.3 Mbps, Moray 18.4 Mbps, Fermangh and Omagh 18.2 Mbps and Ceredigion 16.9 Mbps.

Analysis of broadband take-up in Orkney
Plot of broadband technologies seen in Orkney on thinkbroadband speed test from 2012 to 2018

The Orkney Islands is marginally faster than the Scilly Isles (10.3 Mbps) in our data but the figures from our speed test results are for Q4 2018 and if we look at Q4 2017 Orkney was down at 8.1 Mbps so the speeds are improving since we are showing a figure of 11.1 Mbps for Q4 2018. The chart illustrates the story nicely for Orkney with ADSL/ADSL2+ services still dominating broadband on the islands. The figure of 3 Mbps as a median reported by Which? does look unusually slow especially when you consider that the median download speed for ADSL/ADSL2+ on the islands is 4.8 Mbps (upload 0.3 Mbps), for those who have upgraded to FTTC the median is 32.3 Mbps (5.7 Mbps upload). So we don't know why Which? is so much slower, it may be they have a higher proportion of visitors testing using Wi-Fi, without access to the full individual results for comparison with others in the same postcodes it is impossible to say much more.

The Which? release does make some worrying statements about the broadband universal service obligation that may cause confusion to the public who implicitly trust a household name like Which? The parts that worry us the part about the Government pledging to ensure 10 Mbps as bare minimum across the country and Wales and Scotland having additional schemes in place. The reality of the broadband USO is that it Wales, Scotland and Northern Ireland all share the same USO once finally implemented and people will need to request the 10 Mbps service from the USO provider (most likely BT). This makes it very different to the BDUK programmes and the need to request means that it may be many years before the USO has a significant impact on speed test results for different areas. The detail of how the broadband USO will work should be published in 2019 and people able to start demanding a better connection in 2020, small things like how long from request to delivery, technology used, price and usage allowances are the unknowns we are waiting to be resolved.

We cannot finish without saying some words about Allerdale, Which? claims a median download of 5.7 Mbps and our figure is 22.3 Mbps. Allerdale is a relatively mature FTTC area exceeding the current coverage levels of Orkney way back in 2015 and has superfast broadband available to 91.9% of premises, the median for FTTC in the local authority area is 29.9 Mbps (6.5 Mbps up) and ADSL/ADSL2+ 5 Mbps (0.6 Mbps) with a technology split of 69% FTTC and 31% ADSL. The Which? figures look unusually low and the trend over time for the rising take-up of FTTC looks reasonable in our data.

Comments

Without explaining their data the which figure are not worth much time.

  • Croft12
  • 3 months ago

I hope that you will use this an opportunity to talk to Which and perhaps get involved with future reports. You will get extra publicity and perhaps more followerrs.

  • drwright
  • 3 months ago

Par for the course when it comes to surveys from Which that cover broadband. As the saying goes, there are lies, damn lies and Which surveys.

  • New_Londoner
  • 3 months ago

@drwright The simplest solution for Which? would be to adopt a version of our speed tester and we can then do all the in depth analysis that we do and we would have a slightly larger dataset to talk about. Problem being that we charge for third party commercial sites to use the speed test, since maintaining the software and network infrastructure needs paying for.

We have tried to engage previously particularly around performance concerns over their tester but nothing has ever progressed.

  • andrew
  • thinkbroadband staff
  • 3 months ago

Hi Broadband Watchers.
With Which results running in the past the stats have altered very quickly even Thinkbroadband has been lagging on G/Fast and not picking up the all in Cabs plus many new build sites so the overall situation is better than advertised.

  • Blackmamba
  • 3 months ago

@Blackmamba You say we are lagging, but still doing a lot better than Ofcom who have a lot more people to figure everything out.

  • andrew
  • thinkbroadband staff
  • 3 months ago

Which stopped being a consumer champion many years ago. Their sample is severely biased to those on poor BB connections so understates median speeds dramatically. It needs everybody on FTTP in Ceredigion to use the Which tester to blow its mind and their stats.

It is also misleading in that it implies that all residents in the 15 areas get poor BB "The consumer champion’s data suggests residents in 15 UK local authority are failing to access these speeds"

  • jumpmum
  • 3 months ago

Which is long past its prime and its reports rarely represent the truth whether it be broadband, washing powder or vacuum cleaners. It seems to me that Which is more interested in self-promotion and signing up new members than it is in providing useful reliable information. Sadly many have failed to recognise these changes and mistakenly still hold Which in undeserved and unwarranted high regard,

  • MCM999
  • 3 months ago

Which used to be a great consumer champion but sadly it's gone down the moneymaking trail. I agree 100% with MCM999.

As to speed tests and just about everything else, TBB is the only one I trust.

  • mike41
  • 3 months ago

Hi Broadband Watchers.
If only the BT Checker ADDRESSES tied in with the Ofcom Postal Addresses this would give the ISP,s better correct information to sell with confidence on each Post CODE.

  • Blackmamba
  • 3 months ago

@Blackmamba Can you explain how the BT and Ofcom address level lookups differ? The Ofcom checker works off of data supplied by the providers, so are you saying BT gives Ofcom a different set of data compared to what it shows on its checkers.

  • andrew
  • thinkbroadband staff
  • 3 months ago

Hi Broadband Staff.
I was under the impression that the OfCom Addresses are tied to the Postal Service (letters) delivery which was deregulated date ?
The BT Checker addresses are off the old distribution pole records (DP) which are not 100% correct.

  • Blackmamba
  • 3 months ago

Hi Broadband Watchers.
Off the BT Checker you have the (Post Code ) which may have various DP,s and off each DP you have the homes addresses workers and spares so to be accurate I would use the OfCom postal addresses.
This could I would think alter the %,s. FTTC,Adsl.

  • Blackmamba
  • 3 months ago

@Blackmamba

Here is how it works...

Ofcom uses its powers to request address level data from operators including BT Group

BT will say the speed range for 1 Random Street, is 67 to 80 Mbps on A range and 55 to 72 on the B range
The Ofcom checker will then display 80 Mbps if you lookup 1 Random Street

  • andrew
  • thinkbroadband staff
  • 3 months ago

The Which? survey appears to have painted over the cracks all the same. Unlike the approach on this site, it has tried to use geographical data over huge swathes of Scotland (but not at a constituency level) and yet has failed to appreciate that you need to study the low-20% median dataset if you want to truly appreciate which areas of the country are hurting from poor broadband speeds. Otherwise you are just looking at a mean average across too large a dataset. You might as well just look at the Scotland average at that level!

  • p6resthome
  • 3 months ago

Hi Andrew Staff.
You have a situation here where the tail is wagging the dog in my view but many thanks for your information.

  • Blackmamba
  • 3 months ago

@Blackmamba
As ever, your ability to re-interpret a story is unique.

@Andrew
Perhaps if enough people complain to Which it might reassess how it commissions such reports, come to its senses and pay someone that knows what they’re talking about instead - I’ll spare your blushes regarding who it should turn to!

  • New_Londoner
  • 3 months ago

Andrew

Many thanks for this report.

May I suggest that you:-

1. Contact Which, as suggested, and

2. Post your findings both here & on Wikipedia

  • NJSS
  • 3 months ago

@Andrew if what you say is correct it leaves a puzzling explanation as to why OR's checker can give accurate information but Ofcom's can't!

  • Croft12
  • 3 months ago

The Ofcom data set is not updated dynamically, but usually is 3 to 6 months out of date.

On specific quirks without knowing the address and thus able to see what we have in terms of history for area hard to say, but Ofcom and Openreach are not matching. I know of some errors in the Ofcom checker but whose fault I don't know.

  • andrew
  • thinkbroadband staff
  • 3 months ago

Ofcom's speed and coverage checkers appear to serve no useful purpose other than allowing Sharon White to spend her budget

  • gerarda
  • 3 months ago

Gerarda.
Hi. At least her (Sharron) Map squaring and Post Code and addresses ( Postal) allaction are correct.

  • Blackmamba
  • 3 months ago

@thinkbroadband 0mbs there is no Internet, broadband or otherwise in the location I live whilst a couple of miles a… https://t.co/C75dlZ9PkU

  • @Houllmouse
  • comment via twitter
  • 3 months ago

@Blackmamba
You may like Ofcm’s map but surely the more important consideration is that the broadband data is both out of date and contains errors? The rest is just window dressing.

  • New_Londoner
  • 3 months ago

@BM - I suspect both BT and Ofcom will use the Royal Mail Postal Address File (PAF) for properties. There is a small industry maintaining accurate data.

  • Somerset
  • 3 months ago

Hi Somerset.
I have always used the Royal Mail Postal file which I feel all department should use so correcting the no exerting addresses which can contaminate the true figures.Openreach is the worst contender which has caused many Customers not been able to take up a better service and order FTTC. If you get the data on the Post.Correct the results will merge.

  • Blackmamba
  • 3 months ago

@BM - would you please read your posts before you send to ensure they make sense.

eg. 'all department should use so correcting the no exerting addresses which can contaminate the true'

I have this vision of you working for the GPO riding a bike carrying a ladder...

  • Somerset
  • 3 months ago

Hi Somerset.
I had one customer that had moved to a new house and changed the name and had very low speeds she had registered with local council thus making OfCom Data correct. On the Bt checker her home did not exist after many weeks she was able to order FTTC on a very long line ,this high lighted that her neighbors were still on ADSL. The Post Code was split between two Exchanges I think that there is a good chance one section will get PON.

  • Blackmamba
  • 3 months ago

Ceredigion isn't all doom and gloom. After much lobbying I'm now on 200 Mbps

  • rkimber
  • 3 months ago

I am a Which? subscriber and do wonder whether some of the comments in this blog are from people who haven't a clue about Which?
Admittedly the Which? subscription fee is likely to deter benefits claimants and those who cannot spell probably don't have the brain power to interpret the Which? test results, so perhaps we should feel sorry for them??

  • steveputman
  • 3 months ago

I am also a Which member of many years standing. Unlike the facetious comments from the member above, one thing I have learnt in this time is never to trust all of their so called reports. They have morphed from being a consumer organisation to something being much closer to a commercial operation and now charge extra for many of the services that were previously free to their members.

What is really revealing is their refusal to publish statistics showing the number of complaints they receive regarding their consumer reports, something they have severely criticised commercial companies for. The only stats they do publish are for commercial services where they are legally obliged to do so (Financial Services for example).
Just recently they re-advertised another commercial service called 'Which Tech Support', claiming to provide support for "printers, computers, tablets and smartphones to TVs, broadband and more". Needless to say, this would require a vast resource.

Their Terms and conditions for this service included the statement: "We may engage services of third parties to provide or assist us in providing the Service to you. You agree that we may pass on to such third parties that we engage for this purpose, any data or information that you provide to us with when you contact us with a request for information.” When asked if they did employ third party companies or subcontractors they refused to provide a meaningful answer. I personally would not want my information passed on to such parties.

Whilst Which? do provide a genuine service to consumers, like the BBC, they find it extremely difficult to admit they can be wrong.

  • daviem
  • 3 months ago

I have used Which Mag reports for many years for most of my major purchases. My considered opinion is that it is always worth looking at what they have to say but to treat it with a VERY large pinch of salt because it is very far from reliable. On every purchase occasion I have had very good cause to diametrically disagree with some (not all) Which comments and findings. Exactly as per BBC, they seem to employ letter answerers purely on their ability to fob off legitimate questioning and never ever admit to getting anything wrong. Each to their own opinion but I have NO faith in Which or BBC

  • Tropi
  • 3 months ago

As with many things Which make statements about products and services that they are not expert in. You cannot use a knowledge of consumer products and services in general and apply it meangfully to produce an in-depth analysis of a particular product or service.

I rembwr years ago Which reviewing camera film. In the one test they compared colour and black and white film and marked the black and white film down because you could take colour picturres!

  • sbedford
  • 3 months ago

  • sbedford
  • 3 months ago

As with many things Which make statements about products and services that they are not expert in. You cannot use a knowledge of consumer products and services in general and apply it meangfully to produce an in-depth analysis of a particular product or service.

I remember r years ago Which reviewing camera film. In the one test they compared colour and black and white film and marked the black and white film down because you couldn’t take colour picturres!

  • sbedford
  • 3 months ago

Hi Broadband Watchers.
It was Which Speed Tester that highlighted the Customers that we’re still on Adsl approx 20 Customers out of 175 in the 20 most have switched.

  • Blackmamba
  • 3 months ago

How do you know that 20 out of 175 switched? Made up numbers are easy to post online, so explaining statements is sometimes needed.

Their speed test does nothing to tell you that a better service is available.

  • andrew
  • thinkbroadband staff
  • 3 months ago

howlinwolf

  • howlinwolf
  • 3 months ago

Hi Andrew.
Checked back on the customers number via BT Checker plus returned emails.

  • Blackmamba
  • 3 months ago

So most likely something you said rather than Which? encouraged them to switch don't you think.

BTW a few new Surrey cabinets and FTTP appearing in the postcode search tonight and then the maps next time they update.

  • andrew
  • thinkbroadband staff
  • 3 months ago

Hi Andrew.
It was a customer on the Cab that suggested that his neighbors use Which speed test and if problems go via their ISP I have always let each customer to do as they think and go with the ISP of their choice.
Many thanks for looking at Surrey Post Codes available Speeds I will check on the next update.

  • Blackmamba
  • 3 months ago

Hi Andrew. After you updated the results all three results Surrey,Waverley,and G.Xxxx all improved. Thanks.

  • Blackmamba
  • 2 months ago

Post a comment

Login Register