HELLO & WELCOME

[WelshPaul]

So this morning Voipfone suffered a serious outage affecting errr well everything... :? How embarrassing for Voipfone! :oops:

Phone registrations, their main trading site voipfone.co.uk, their forums voipfoneuserforum.com and errr well everything.

Follow them on twitter @Voipfone and @Voipfonestatus for live updates.

@Voipfone
We are aware of a serious connectivity issue our engineer are currently working on it
10:40am - 27 Feb 14

[WelshPaul]

So Voipfone have finally managed to get voipfone.co.uk back up and phones are registering but there are still serious problems with the service. Inbound and outbound calls are not working, geographical numbers are dead and services on their site such as logging in to accounts are limited.

This is taking Voipfone a considerable amount of time to fix, not good going by their customers comments over on twitter.

[GoofyCyborg]

I bet some heads are rolling over in Voipfone HQ! Very embarrassing for Voipfone as they always go on about failover procedures and such.

[WelshPaul]

Still waiting to read Voipfone's report on the outage...

[WelshPaul]

Took a few days but here is the report: http://www.voipfoneuserforum.com/viewto ... f=1&t=6381

At approximately 8:30 am on the 27th February our monitoring system alerted us to an issue with one of the primary links between The Bunker, our data centre in Kent, and our data centre in London, Telehouse East.

Our automated failover kicked in and the traffic was re-routed via our back up link as designed. This link takes traffic via our Goswell Road, London data centre and is our secondary route used for just such occasions.

All of the above had no impact on customers and services were unaffected.

Later that morning our monitoring system detected an issue with some services in our primary data centre at Sovereign House in London due to unusual latency on the secondary inter-data centre link that had become active earlier. Although the issue was not big enough for our automated failover to kick in, in order to provide a high quality service to all our customers, our engineers decided that it would be safer to move some of the main services to our secondary data centre in Kent.

At 10.24 and just before our engineers finished making these changes, our back up link (which is from a third completely independent supplier using separate routing) also went down which resulted in our data centre in Kent going completely off line. At that point some of the main functions of our platform had become completely unavailable as they had been moved to the data centre that was now down. All we could do was rebuild the network so that wherever connectivity returned, we’d be ready for it. Connectivity was re-established just as our engineers were bringing the main functions (registrations and call routing) on line.

This final outage did not only affect us but also the thousands of other companies in the same data centre including many high-street shops, banks and financial institutions. The outage lasted from 10.25am to 12.09pm, 1 hour 44 minutes.

All our data centres and the connectivity to them is ISO 27000 certified and we have been told that the odds of what happened could be as low as a million to one.

We have invested a considerable amount of money in the best data centres in the UK, along with tier one connectivity and multiple backups and for us to lose all connectivity with multiple redundancy in place is hard to believe – and extremely disappointing.

Of course as a company we will learn from this and you can be assured that we are working with our data centres and providers and doing everything we can to ensure that this doesn’t happen again.