BT & Plusnet outages

Both BT and Plusnet are experiencing connectivity issues meaning some Krystal clients are unable to access their services with us. The issues though are directly with BT & Plusnet and all Krystal services are operating as usual. If you are with either provider and experiencing issues please check their service status pages here:

http://www.plus.net/supportpages.html?a=2&support_action=messages&ispservice_id=adsldial

https://btbusiness.custhelp.com/app/service_status/

We have had reports from some customers that BT’s call centre are advising that there are no service issues at their end however as their status page above shows this is incorrect, and they are experiencing wide-ranging service issues this morning.

From what details BT / Plusnet have provided the issue they’re seeing is a routing failure at one of their connectivity suppliers which means that BT/Plusnet customers will not be seeing a total service failure and some websites will still work correctly, though many others (including those hosted with us) will not.

This website indicates the scale of the problems:

http://downdetector.co.uk/problems/bt-british-telecom

[RESOLVED] DDoS Attack – Artemis

[11.06] It appears that currently the Artemis.krystal.co.uk server is the subject of a 10Gbps DDoS attack, to which we are working with our protection providers to mitigate this attack.

[11.16] The attack appears to now be subsiding therefore we hope that Artemis will return to service as soon as is possible, to which we will update this ticket when this has been confirmed.

[11.25] Artemis has now been restored to full service.

We thank you for your patience and understanding whilst we worked to resolve this issue.

Pluto – Scheduled Hardware Upgrade

05:35 Problems were encountered during data synchronisation due to the poor performance of the old server’s storage system. As a result, databases are only now being copied to the new server. The hardware swap is now not expected to start until at least 07:00hrs.

07:31 services have been stopped on old server while final sync is executed and hardware is switched.

07:55 Almost there!

08:15 All services are available again, and customers can take transactional websites out of maintenance mode – any queued email will be delivered in the next hour or two. Thank you for your patience.

Valhalla stability issues

10.43 Valhalla should now be operating as usual – please accept our sincere apologies for the disruption caused.

10.30 We are currently experiencing some stability issues with Valhalla causing intermittent issues. We are working to have this fully resolved ASAP.

Gobi – Scheduled Hardware Upgrade

05:35 Data synchronisation is almost complete. Services on gobi.krystal.co.uk have been temporarily stopped during the upgrade of hardware.

05:55 New server is booted up – carrying out final networking configuration.

06:10 All services are now available again. Customer can take their websites out of maintenance mode. Thanks you for your patience while we carried out this essential work.

06:55 Some users experienced a problem with MySQL extensions in PHP for the past hour. This has now been corrected.

[RESOLVED] DDoS attack/UDP flood – Ceres.uksrv.co.uk

[10:58] It appears that a UDP flood is being directed towards ceres.uksrv.co.uk – To protect our network mitigation has been applied, access to ceres will be intermittent whilst this mitigation is in place.

[11:25] It appears that the attack vectors have changed, we are working with our mitigation provider to resolve this.

[11:36] We are still working with our mitigation providers to resolve this attack on our network.

[11:41] The attack on our network has been successfully mitigated therefore regular network connectivity has resumed – Networks staff will continue to monitor the network performance for stability following this attack.

We thank you for your patience and understanding whilst we worked to resolve this issue.

RESOLVED Sahara connectivity issues

08:15 – We are currently investigating an issue with Sahara – please bear with us whilst we get this fully resolved. We sincerely apologise and thank you for your patience during this time.

9:16 – Running Apache upgrade on Sahara to fix PHP errors.

9:21 – Websites are now loading – some sites may still display errors.

10:15 – Running upgrades on Sahara as we’re still seeing errors.

10:17 Sahara being rebooted to finalise upgrades.

10.40 We are temporarily blocking web traffic to Sahara whilst we run one last update. Thank you for your patience during this time.

11.15 Sahara is now serving requests as per usual – if you are still having any issues please do let our technical team know. Thank you for your patience this morning whilst we resolved this.

Olympus Connectivity Issue

13.30 We have now resolved this issue and all sites are now functioning correctly.

13.15 We are currently investigating an issue with Olympus which is affecting connectivity.

Please bear with us whilst we resolve this and thank you for your patience during this time.

Apollo – Scheduled Hardware Upgrade

05:15 Data synchronisation is almost complete. Services on apollo.krystal.co.uk have been temporarily stopped during the upgrade of hardware.

05:50 The new apollo server is online and responding normally. Customers with transactional websites can now switch back into production mode.

RESOLVED : Zeus PHP Issue

12:20 All users on Zeus should now be able to view and access their sites. If you notice any errors from now please let our technical team know via support.krystal.co.uk.

12.15 The majority of sites using PHP are now working correctly. There are a handful of users using older PHP versions that we are still working to fix.

11.25 We are still working on resolving the issue with PHP – this is affecting some clients on Zeus at present and we sincerely apologise. We are hoping to have this resolved in the next hour – but will keep updating the status page. We appreciate the disruption this is causing some clients and are working to resolve this as quickly as possible.

10.40 Some users on Zeus are experiencing a PHP error on their sites. We are currently investigating this and hope to have this resolved ASAP. We thank you for your patience during this time.

Follow

Get every new post delivered to your Inbox.

Join 104 other followers