Skip Navigation

Not all speedtesters up to the job - still!
Wednesday 11 March 2009 13:10:29 by Andrew Ferguson

Virgin Media started off looking at the ability of speedtesters to accurately test its 50Mbps XXL service back in 2008. The results surprised it, and has led to the provider carrying out further checks to consider how well checkers handle 10Mbps, 20Mbps in addition to 50Mbps.

"We’re dismayed to find that for years, many speedtest and comparison sites have been misleading the public on ISP performance. Consumers rely on these sites for clear advice and honest results. The fact that many of them cannot accurately measure speeds of 20Mb, and some cannot even accurately measure speeds of 10Mb, demonstrates there is a clear need for more transparency. As ultrafast services such as 50Mb become more mainstream, this issue will need to be addressed quickly"

Jon James, Director of Broadband at Virgin Media

To ensure the testing was as accurate as possible Virgin Media used its Next Generation platform for all the tests, this ensured that contention and capacity issues were not a problem. The report highlights the importance of also having a computer that is optimised for the broadband speeds you are able to receive, Vista automatically tunes itself, but for Windows XP you may have to manually optimise the RWIN value to get the best performance.

One aspect not covered in the report is the testing methodology used, there are some testers in the market where you can actually unplug your ethernet cable and plug it back in, and the result is the same as if you had left the cable plugged in. These testers appear to work by taking a peak reading, which if your service has a highly variable speed due to congestion or some other issue can give you a false impression of high speeds. Our tbbMeter tool is useful to allow you to monitor your speeds without wasting your allowance on constant speedtests and it will show how large downloads are varying in speed, as well as on speedtesters.

A table showing the sites tested and the results is below:

Speedtest Site 10Mbps 20Mbps 50Mbps Comment (London) Yes Yes Yes Has been 50Mb ready since start of campaign, however have made improvements to testing algorithm and plan to introduce Northern server (Maidenhead) Yes Yes Yes Hosting/Server change at Maidenhead site coupled with changes to algorithm have improved results. Yes Yes Yes Has been 50Mb ready since start of campaign Yes - Yes Site was unavailable during 20Mb testing, however use same technology as (by Ookla) Yes Yes Yes Now using thinkbroadband’s technology. Yes* Yes Yes* *Inconsistent results on 10Mb and 50Mb Yes* Yes No *Unable to measure 50Mb, with inconsistent results during 10Mb tests Yes No No Unable to measure over 10Mb accurately. Releasing monthly results using inaccurate data.
Yes No Yes* *Inconsistent results on speeds above 10Mb. Currently working on improvements to test. No No No Working on improvements to testing system. No No No Working on improvements to testing system. No No No Working on improvements to testing system.


Posted by Jimkirk363 over 8 years ago
Can i just be the first to inform u that i spoke to Virgin Medias chief execs office over a week ago and they informed me that London Server was in no way accurate on their 20mb and above packages.
Posted by 1Gopher over 8 years ago
Another speed test site which is pretty accurate as there test is performed across 3 servers rather than 1.

Posted by Dawn_Falcon over 8 years ago
Adjusting a speed tester to "cope" creates an unrealistic situation. Speed testers should connect to a popular backbone, and take precisely the speed they get. Working with any ISP distorts the results.
Posted by ceedee over 8 years ago
Ironic that VM whine about inaccurate speedtests the week after they prevent access to their own?

Incidentally, checking with (London and Maidenhead) both report my VM 10Mb/s connection at around 8500Kb/s while ThinkBroadband's test only gets 2700Kb/s...
( reckoned 7500Kb/s)
Posted by David-Park over 8 years ago
I do not see the BT speed tester there, is it ok or not on anything?
Posted by andrew (Favicon staff member) over 8 years ago
BT speedtester site is meant to only be used with BT Wholesale IPstream connections, and is not considered a public speedtest site in the way that the others listed are.
Posted by seb (Favicon staff member) over 8 years ago
@Dawn_Falcon - Working with ISPs is important as it means you can iron out 'local' problems. These might not be visible to users normally but could affect results. We're always open to talk to any provider about their results and how we can help them improve them. What is not acceptable is manually 'prioritising' traffic so the speed testers give an incorrect picture. Making sure the speed testers are well connected, etc. on the other hand is fine.
Posted by Dixinormous over 8 years ago
@Dawn_Falcon - speedtesters are subject to abnormal traffic patterns and loads from most servers so makes sense to adjust them to cope with that load. They are there to give a realistic representation of the performance a customer should expect from their broadband service. The speed tester should never be the limiting factor in the test else it's worthless.
Posted by Dixinormous over 8 years ago
I'm deeply shocked by broadbandchoices, there I was thinking it was a crappy sales driven site with an 'expert' with no clue what he's talking about and a speedtest hosted on a poorly connected colo in Milton Keynes as it's cheap.
Posted by rasczak over 8 years ago
I am confused how the Thinkbroadband speed tester can be rated for 50 Mb connections and yet struggles with the 8Mb connection I have, current IP profile is 6500. I just did a test Date 11/03/09 21:24:00
Speed Down 993.07 Kbps ( 1 Mbps )
Speed Up 376.24 Kbps ( 0.4 Mbps )
Port 80
I then downloaded the file from here and it came down at an average 684Kbps
Something wrong somewhere.
Posted by rasczak over 8 years ago
Typo in my last post, when I downloaded the file it came down at 684KBps
To clarify, the speed tester gives 993 kilobits per second, which equates to 124 kilobytes per second. The file download averaged 684 kilobytes per second.
Posted by rasczak over 8 years ago
More testing, I set my download manager to use a single thread to download and the average was 131 kilobytes per second, close enough to the tester speed to say it was the same. Seems that the speed tester, for me at least, only uses a single thread, even though the browser is set to use up to 8 threads per server.
Posted by jerrytaff over 8 years ago
Are you having problems at thinkbroadband? Until recently your speedtest gave me a solid 6.7 to 6.9M download and 500k upload. Over the last few weeks my download speed has reduced - first to 1M now to less than 500k. upload is unaffected. However, other than your site, my connection seems fine, and other speed testers still measure the download at 6.9M
Posted by Dawn_Falcon over 8 years ago
@Dixinormous - There's a difference between ensuring they have a good connection and working with any particular ISP

@Seb - And given the users will see those "problems" on those sites, it's simply making the speedtest unreliable for real-world speeds.

(The only deacent connection-test speeds are inside your ISP's own network!)
Posted by CARPETBURN over 8 years ago
Ive never found ANY speed test site that is 100% accurate this sites and get close but the ultimate way to still measure your speed is to find a largish file on a fast server and download it while observing your speed.
Frankly im shocked Virgin are still kicking up a fuss, do their services not run at a reduced rate for more hours in the day than it does full speed if you actually attempted to use all that so called speed during the day?
Posted by andrew (Favicon staff member) over 8 years ago
There is not a direct connection between your broadband connection and our servers, so people need to look at the links inbetween i.e. tools like tracert.

Some tests as mentioned in the article will report good speeds even when actual speed is up and down a lot.

Yes our tester uses a single thread, hence why tbbmeter has had an update recently to support multi-thread downloads, between 2 and 5 threads, i.e. 10MB through to 25MB of payload.

Posted by andrew (Favicon staff member) over 8 years ago
If people think there is a specific issue then post on our forums in TalkToTheStaff a server issue should affect all ISP's equally.
Posted by CARPETBURN over 8 years ago
I dont think there is any issue with your speed tester or any of the other well known ones, i do of course realise none of us are connected direct to yourself and thus the reading is never going to be 100% accurate, i personally tend to find this sites speed tester and under-estimate my real life download and upload speeds by about 10%, I suspect this is pretty normal as various speed testers have done the same thing with more than one ISP i have been with.
Posted by CARPETBURN over 8 years ago
Downloading a file either from here or elsewhere IMO is still the best way to observe your connections speed. I dont understand why Virgin are getting all hot under the collar over this... Why dont they just host their own speed test on their site like some other ISPs do if they are that bothered?
Posted by Jimkirk363 over 8 years ago
Think broadband test.
Date 12/03/09 02:06:36
Speed Down 16940.81 Kbps ( 16.5 Mbps )
Speed Up 721.79 Kbps ( 0.7 Mbps )
Port 8095

Downloaded the 200mb test file at all of 104kb/s lol London Server

As u can see Speedtest always reports my upload wrong

Soooo shows the difference between them for me here.
Posted by mcowley01 over 8 years ago
I have Vista x64. i downloaded the tbbMeter and tried to run it - first thing it said was that an update was available so I clicked on the update button. Message was that the signature was out of date. Now it just doesn't open.
Posted by SirBruceHall over 8 years ago
Ref. Andrew Ferguson's second para above, I've got Windows XP. Could someone please tell me how to "manually optimise my RWIN value"?

Posted by andrew (Favicon staff member) over 8 years ago
To mcowley, I've had this issue under XP, and a reboot fixed it. If the certificate where not present no-one would have been able to upgrade.

If you want email me and I'll let you have a link to download the latest version. Alternatively rename one of the backup versions to tbbmeter.exe and it should run.
Posted by adriandaz over 8 years ago
So much for:
Posted by t0m5k1 over 8 years ago
tell you what i would love,
when a customer calls in about speed issue's that they actually do really visit when asked to do so!!
Posted by andrew (Favicon staff member) over 8 years ago
The BT speedtester is not always the most appropriate resource to use, providers should get reports of the IP Profile for lines, so can store this themselves, and it is not intended for day to day speedtesting.

Many people will know that the BT speedtester is very hit and miss or can take ages to run, and if on the phone to support that can make it expensive.
Posted by Dawn_Falcon over 8 years ago
Carpetburn - Sure. files inside your ISP's network for connection speeds and from major peering nodes like for testing your ISP's peering.
Posted by comnut over 8 years ago
gawd guys... I see y'all still have not much sense.... (excepting notable few...)

(I am way too busy ATM, to keep up with this hopelessly ordered 'comments'... It seems they are just too lazy to give it a proper *forum* order, with *editing*, PM, threadview, email notify, etc... )
Posted by comnut over 8 years ago
I have noticed starting to 'lag' a while ago... is a bit better, but the *all* start suffering due to too many using them, most likely why .net is getting bad.. even the TBB server seems to be getting 'patchy'...

USE Tbbmeter properly to get a full 'health check', not to just lamely moan about your speed!!
Posted by andrew (Favicon staff member) over 8 years ago
For threaded etc then that is what the forums are for, and you start the thread to discuss the salient point, we even allow anonymous posting in some places.

There is more than one tbb server by the way
Posted by XRaySpeX over 8 years ago
@jerrytaff - are you on Orange?

cuz there seems to be a problem on ThinkBroadband Speed Test & Orange.

Posted by chrysalis over 8 years ago
I think what dawn_falcon may be reffering to is if they adjuting the transit routing just for the speedtest but not for generic traffic then it is in affect cheating. eg. optimal expensive routing for speedtest max speed but then other traffic cheapest routing but less optimal. A speedtest is also useful for testing transit/peering performance.
Posted by Dawn_Falcon over 8 years ago
Chrysalis, a lot of people misuse speed tests on the web to check connection speeds, when downloading from inside your ISP's network is the only good test there.

Transit/peering performance is the only really legitimate use for web-based speedtesters. And yes, it's easy to fiddle ><
Posted by chrysalis over 8 years ago
it is indeed eg. I speedtested worldwide locations on when I joined my isp, obviously I know I wont max out from asutralia but is useful to get an indication of overseas transit performance. However if the isp is playing games with the routing then its all a waste of time.
Posted by CARPETBURN over 8 years ago
Agree with Dawn Falcons and chysalis last posts.... I really dont even understand why virgin are getting their knickers in a twist over speedtesters.... If its such a big deal why dont they put a speedtester on their own site and refer people to that if the customer has speed issues. Sounds to me like they want others to do the work to support their naff service but they dont want to do any work thereself.
Posted by jbesant over 8 years ago
Frankly I just want a reliable internet connection. I have a 20MB service from VM and have just got service back after a 6 day outage and suffered shocking customer service throughout. I have had nothing but problems since VM started it's upgrade programme to facilitate 50MB in Nov 2008. They should trying managing the netwrok they have effectively before making things worse!
You must be logged in to post comments. Click here to login.