Broadband News

Speed and latency problems for Virgin Media users to some destinations

Virgin Media users have been having a bad few days with their broadband connections, though it is a bit confusing as the problems don't affect all Internet sites and services equally.

The symptoms are that during the week latency rises by 20 to 30ms between the hours of 4pm and midnight to our Broadband Quality Monitor tool, and we are also seeing a significant drop in the observed download speeds from our speed tester.

7656-virgin-media-bqm-6-march-2017-thumb
Click image for larger version

Normally congestion issues only affect a proportion of Virgin Media customers, but from looking at the full set of monitors it looks as if all Virgin Media cable broadband customers are affected. Looking at the trace route activity to and from Virgin Media customers a handover from Virgin Media to LINX (London Internet Exchange) looks like it is congesting badly (long thread on Virgin Media community forums discussing latency).

We have cross checked with other broadband providers and nothing other than the usual variation in BQM results is apparent, i.e. this blip is unique to Virgin Media.

The effect on speed test results is very noticeable, the average download speed between 4pm and midnight on Tuesday 6th March was 16.8 Mbps (upload average 4.1 Mbps, TCP latency 105ms) and at the start of March on Wednesday 1st in the same time period it was a more normal 48.3 Mbps download average speed (upload 6.3 Mbps, TCP latency 77ms. The maximum speed test seen was also affected, the best seen in the 8 hour period was 160 Mbps down, compared to 305 Mbps on the 1st March.

Other speed testers are sometimes affected, a lot depends on the routing to the site, something the average user will not normally worry about, we can serve our data for the speed tester from an alternate location but that would simply mask what is an issue for customers on Virgin Media.

What is odd is that the times change at the weekend, suggesting that maybe there is something changing in the core network configuration, or that people are suddenly doing lots of stuff at the same time each day, but as they are home at weekends the timings alter.

"Having reviewed some network data samples this does not appear to be an issue affecting all of our customers. This could be related to specific network equipment we but are currently investigating this further to determine the exact cause and solution. We apologise to any customers affected and hope to have this fixed soon."

Comment from Virgin Media spokesperson

A Virgin Media community moderator posted on Sunday that the provider was looking into the problem.

Update 4:55pm The original article was drafted at midday and shared with Virgin Media to give them some time to respond prior to publication which they have done. We have also checked to see what the situation is with regards the latency issues, and on cue at 4pm there was a rise similar to the previous days, but after a few minutes when the latency had risen by 10ms the latency returned to normal. This effect has been observed across all the BQM graphs that we can see, fingers crossed this means someone has fixed the root cause, we are currently still seeing speed tests in the 50 to 100 Mbps region so that side is looking better than Monday evening too.

7656-virgin-media-sample-bqm-7-march-201
Click image for larger version

Comments

Thank you very much for following and reporting on this problem -- it is now more likely to get solved! Sad part is, imagine some people are affected by a problem like this but it doesn't show up in popular speed tests or your BQM... good luck getting a solution in that case.

  • tcstone
  • 5 months ago

Thanks for running this story Andrew.... you seem to have got Virgins attention on this issue which I believe many have failed to do... we'll see if this issue stays at bay now.

  • mlmclaren
  • 5 months ago

I doubt that VM will tell anybody what happened. Cards close to chest and all that.

  • Suspishio
  • 5 months ago

It's been like this for many years with VM. But, I doubt VM wouldn't care about BQM. Because VM found nothing wrong with it own network. VM is horrible network with heavily congestion as always is.

BT FTTC is far better!

  • adslmax
  • 5 months ago

@Suspishio Leave it with me, will see what I can find out :)

@adslmax No operator is bothered about BQMs per se, however the potential for service impact elsewhere is an issue.

VM clearly were concerned about their own network here as they resolved the issue through managing traffic, and are aware of localised congestion as they do deal with it.... eventually.

  • CarlThomas
  • 5 months ago

@adslmax - BT FTTC might be great now but I'm sure congestion related issues are soon to appear as speeds get faster and more connections go live.... good news is the 50,000 premises on my exchange will never be connected it seems so there's 50k less subscribers to worry about #VirginOwnedAreas

  • mlmclaren
  • 5 months ago

I have a new issues that cropped up since this "fix"... will elaborate more after a wee bit of investigating

  • mlmclaren
  • 5 months ago

So after a quick tracert, can confirm Virgin have changed routing a little... used to go via Manchester but now going via somewhere else (seeing elslou instead of manc in tracert from Virgin to LINX)

So now instead of an increased base ping on just CQM's based in London I'm seeing increased/jitter on all BQM's (including one based out of Leicester!

Xbox one also impacted now :/

  • mlmclaren
  • 5 months ago

@mlmclaren

Whether there are issues of not depends on the competence and willingness of the service provider to spend enough money on capacity and performance management. I don't see any reason why g.fast would introduce performance problems.

  • TheEulerID
  • 5 months ago

@mlmclaren

FTTC(VDSL/g.fast) and ADSL is not a shared technology like DOCSIS is

worst case on VDSL is that modem speeds lock on slower as more are on your cab but latency should be unaffected or packet loss

Virgin normal issue at a local level is they don't act fast enough to add more line cards or split the cab when utilisation gets high on the COXA side of the main FTTN cab

  • leexgx
  • 5 months ago

@leexgx
The shared or not nature of DOCSIS vs DSL in the last mile looks to be irrelevant in this case, as the congestion here looks like it's further into the VM network.
Once you are on to the ISPs IP Network, the challenge of having sufficient bandwidth to prevent excessive congestion within the ISP network or through the various peering points is the same for either technology.

  • Bin216
  • 5 months ago

Very true, TheEulerID...

Yes, BT FTTC **may** be great, better, etc....
UNTIL something goes wrong!!!
Then you are stuck with the quite often hopeless, passing the buck, bad customer service, and NO internet for weeks...

the last BIG outage from VM (due to some builder puttig a digger thru a main cable) we DID at least have a service (10Mbps, but useable!!)

as they say, the 'other isp' always looks better, until a few months later when you get problems...

  • comnut
  • 5 months ago

Could you point me to an instance where a 'big' outage on the BT FTTC network, by big I mean multiple regions, caused customers to go offline for weeks please, Sir?

  • CarlThomas
  • 5 months ago

@[email protected] - I think you've misunderstood me, as already has been said the issue posted about above is a core network issue not localised... I'm happy to say I'm in an area where Virgin have kept on top on capacity and don't suffer localised congestion like many other areas currently do!

My comments regarding alternative providers networks is they both work the same beyond the local network, and I've had first hand experience of core network congestion with FTTC providers such as BT and Plusnet and even Zen for a while...

  • mlmclaren
  • 5 months ago

UPDATE - Issues I reported last night following Virgin Media's "fix" where actually fixed by a reboot of Hub 3.... all things fixed following reconnection.

Connections to ThinkBroadband are still impacted though, not as bad as before but I these issues don't fix themselves overnight, I can say that overall connections to London have improved with some returning to normal.

  • mlmclaren
  • 5 months ago

@CarlThomas - Could you elaborate as to what outage your talking about, I'm sure such a thing would have attracted a lot of complaints and media coverage... I don't recall any?

  • mlmclaren
  • 5 months ago

This was ages ago, the new build on the old twickenham sorting office site...
By BIG I mean the CABLE was supplying about 500 premises...

  • comnut
  • 5 months ago

Whoops I saw it was BT, has someone forgotten their glasses again?? :D :D

  • comnut
  • 5 months ago

Ah Brewery Wharf. Yes. 500 premises isn't a big outage. On VM it's a single node, on BT a single PCP.

If you live there, kudos. The prospect of paying those prices makes me nauseous, and property prices in that area were a really good motivation to move a couple of hundred miles north. Paying Twickenham / St Margaret's / Whitton rents was tiresome.

  • CarlThomas
  • 5 months ago

I checked my own BQM and seen the exact same thing, is it a coincidence that these are the exact same times that traffic management start (on the upload)

  • toytown
  • 5 months ago

I had the same readings since last Friday night but everything is back to normal now and I hope it stays that way.

  • cks22
  • 5 months ago

It may not be 'big' to you but when the whole of regal house has issues, as well as residential....

what would BT do, just leave you without internet for 2 weeks??? that is why my boss will never use them... :P

  • comnut
  • 5 months ago

but you're calling a single node is like a major event across the whole of the UK (which was more likely just a fiber lines they destroyed)

diggers did the same thing around my end they fixed it next day (looked like they was building a new house on the road as was a lot of other equipment there were a digger had cut the fiber that feeds the local FTTN VM cab for the block around here, was resolved over night)

i not really had problems with ADSL or VDSL when there has been problems they have norm been resolved within 2-4 working days

  • leexgx
  • 5 months ago

@mlmclaren

You are the one that's misunderstanding. I made no comment with regard to local network topologies or anything that could be inferred that way. What I said was that good capacity planning and provisioning with deal with increased requirements and that there was no reason to believe g.fast will be any different.

Capacity planning applies to core as well as distribution networks.

  • TheEulerID
  • 5 months ago

My connection has returned to normal now. Oddly, most speed tests showed no drop in my speed, during 'the bump', or an increase in latency. Didn't bother with the TBB test, 'cos it never gets it right.

  • Windblown
  • 5 months ago

because TBB is giving you a realistic speed result not a max attainable one ?

  • leexgx
  • 5 months ago

Realistic? I think not. I get the same very low speed, less than my actual speed, whatever time of day or night. I live in a non-congested area anyway. YBB ought to be indicating some difference between peak and off-peak.

I have tested by downloading a single file from a single source (5Gb). It downloaded at 41MBs (333Mb). I did a TBB test just before and just after. Both tests gave me less than 120Mb even on the single thread test. There is the real world, then there is the TBB world.

  • Windblown
  • 5 months ago

i did have to run 2 tests before it would get above 100mb on the 1x test still get 220mb ish on 6x test

this was at 3am thought day time is norm fien as well gets a bit heary around 5-7pm

  • leexgx
  • 5 months ago

Windblown:
Note that the speed you get depends VERY much on where you are testing from...

Speedtests use a special connection, so are not affected by the 'usual suspects'.. If they do not produce your paid for speed, then there is a core problem, so contact your ISP!!

A big company(and others too..) like ytube will bandwidth limit, to keep it fair for smaller users..

Any others are affected by how bad the links across the country are, and how many other people are downloading the same file!

  • comnut
  • 5 months ago

Believe these may be the tests that @windblown is talking about
http://tbb.st/1488530851677787055
http://tbb.st/14891276361600203650

Two different versions of code and similar results, so not a core speed test code issue.

The graph shape for download suggests something is not perfect, but speeds are such that most people will believe connection is running perfectly, and with multiple routes through the Virgin Media core am sure some testers will give better speeds.

  • andrew
  • thinkbroadband staff
  • 5 months ago

@Comnut Speed tests DONT use a special connection, nothing special about ours all the services we host get the same treatment and attention to throughput and low latency on our network.

  • andrew
  • thinkbroadband staff
  • 5 months ago

On looking at tests in the area for Windblown they appear to be one of the faster users.

A good speed test should look like
http://tbb.st/1489138395378821655

And its not like our servers cannot go faster since there are tests like http://tbb.st/1488564954431503855
(NOTE user lied it was not EE, but a JANET connection)

  • andrew
  • thinkbroadband staff
  • 5 months ago

Last comment is confusing,

What is EE & what is JANET?

Which user lied about this? I hope you aren't referring to me.

BTW these are not the tests I was talking about. You should not use disinformation to defend your unreliable speed tests.

http://tbb.st/1488530851677787055
http://tbb.st/14891276361600203650

  • Windblown
  • 5 months ago

EE was the provider shown in the speed test, i.e. what the user doing the test picked, but we can see it was a test across the academic network called JANET. So never have said you lied.

On the tests sorry if the tests
http://tbb.st/1488530851677787055
http://tbb.st/14891276361600203650
were NOT carried out by you, they fit your description and other metrics. Please feel free to run a test again and share the result link.

  • andrew
  • thinkbroadband staff
  • 5 months ago

Just did a series of 1Gb tests on your file download test server. The best test maxed at 39MB for a few seconds, although 34MB (sustained) was more typical. A few tests stuck in the high-20s, still higher than your Flash tests on my connection. Screen shots available if you want proof!!! Grrrrr

I followed this with your standard test. I can't offer you an off-peak test until later.

http://www.thinkbroadband.com/speedtest/results.html?id=148914984124476512751

  • Windblown
  • 5 months ago

.. continued As you can see your standard tests do not match your file download tests as reported by me. This has been my complaint about your standard Flash Player test ever since TBB introduced it.

I understand your point about the contrast between your multi-thread and single thread test picking up issues with the Virgin Media architecture, (shared single coax), but none of this explains the consistent under-performance of your Flash test compared with the performance I get using UK-based single-thread file download tests.

  • Windblown
  • 5 months ago

...continued BTW: The speed tests and 5Gb download tests I referred to in my earlier post were under a different user name, some weeks ago. Currently, I cannot access that user name because the new email I allocated to it failed to get me a verification link.

  • Windblown
  • 5 months ago

And the client for the file download was?

Browser add-ons can increase the number of threads for file downloads helping files and increasing difference.

No idea why you are getting slower performance when others with same ISP can get the speeds.

Short of driving to you with a device tested and proved on a gigabit connection this debate will go around in endless circles.

  • andrew
  • thinkbroadband staff
  • 5 months ago

Was using Firefox, with no such known add-ons. Took the hint and used Chrome. Same result. Sustained download speed on file download tests of 35MB.

Did Flash test in Chrome too. Marginally better HTTP test. http://www.thinkbroadband.com/speedtest/results.html?id=148915362116924915173

Tried IE. Same, but almost acceptable HTTP test.
http://www.thinkbroadband.com/speedtest/results.html?id=148915437870393897494

Frankly, if your download test and flash test is subject to the vagaries of browsers then there isn't much hope for consistency is there?

I rest my case.

  • Windblown
  • 5 months ago

And we will beg to differ and no understand why your PC is behaving different to other Virgin Media customers who can hit the speeds sold when there are no other issues.

If I dropped 30% of the slow samples, 10% of fastest and took an average then and never showed the graphs am sure you'd get a happier number.

  • andrew
  • thinkbroadband staff
  • 5 months ago

Yes, we will continue to differ. TBB stands alone as the only single strand and multi-strand test package that never gives test results anywhere near my actual.

  • Windblown
  • 5 months ago

@Windblown

what antivirus are you using (or its wifi) kaspersky is well known for messing with speed test results

  • leexgx
  • 5 months ago

There is another one, that you can personally set the number of up / down streams, and other stuff...

http://www.dslreports.com/speedtest/11796784

  • comnut
  • 5 months ago

Windblown: I would be interested in your dslreports result. :)

  • comnut
  • 5 months ago

here is my one :)
http://www.thinkbroadband.com/speedtest/results.html?id=1489228936866915955

  • comnut
  • 5 months ago

@Windblown - Util Virgin upgrade their core network lines with London, we will continue to see issues such as the ones shown in the TBB speedtester...

Personally I've used TBB's tester for many years with all manner of different connections/isps and always found it to be fairly consistent... even back in 2014 I saw similar issue to the ones I've seen lately and that was also related to congested core network.

  • mlmclaren
  • 5 months ago

... of course, I'm not sure just how many people will be affected by such capacity type issues, its generally depends on routing choices...

Mine was really bad last week, but after Andrew made Virgin aware of the issue, they made some minor changes to the routing between me and London...

Improved things a lot, but not 100%

  • mlmclaren
  • 5 months ago

Thanks TBB for identifying and publicising a problem.

Unfortunately I am not wanting a new technical hobby, like most internet users I just want my ISP to supply what I pay for.

If I buy a pint of milk and it's delivered half empty I complain and get a refund. If I get 10% of the speed I pay for from Virgin I get technical twaddle as excuses. A bit like being told that the cow was a bit off colour and dehydrated....

  • fox-uk
  • 5 months ago

Pay for up to 200mps. Late afternoon and evening can drop to less than 5mps.

  • fergiet
  • 5 months ago

it is always worth getting a second opinion... anyone tried yet??

  • comnut
  • 5 months ago

is it just us or any other Virgin customers seen random packet loss over the last two days

  • steven19900
  • 5 months ago

@steven19900 - I am also seeing random packet loss since Tuesday after another bump in latency Tuesday evening.

  • paulo9
  • 5 months ago

might be worth asking here??? :)
http://community.virginmedia.com/t5/Broadband/ct-p/Broadband

  • comnut
  • 5 months ago

Post a comment

Login Register