Scheduled Poseidon Move – 18/07/2014

Today is the scheduled migration of Poseidon to a new server.

We have already provisioned the new server and prepared it to become the new Poseidon.

10:00 – Data transfer has started between the servers to prepare for account transfer.
12:00 – Data transfer has occurred much quicker than anticipated and we may move the servers earlier than planned
13:00 – We’ve moved over to the new server now, much earlier than anticipated – we still need to reboot the server and move it physically
14:39 – The server move is complete – please let us know if you have any problems

Network Maintenance – Wednesday 16th July 06:00 (Completed)

Dear All,

The network upgrade that was postponed from Friday has been rescheduled for this coming Wednesday morning, starting at 06:00

We have made use of the extra time to further improve our preparedness and taken on board feedback about the time of maintenance windows. Hopefully early mid-week will affect fewer people in the eventuality of disruption.

We will update this blog post as the work takes place.

Thank you for your patience during this essential network maintenance.

 

06:00 We are starting this work now.

06:30 This work has been completed successfully. Thank you for your patience.

Network Maintenance 11th July 18:00 (Postponed)

Dear All,

We will be undertaking critical core network maintenance tomorrow evening commencing after 18:00.

This has the potential to affect all servers/services and while we hope that there will be no noticeable impact there could be a loss of connectivity of up to 2 minutes.

We thank you for your patience and understanding during this maintenance window.

20:55 We have started this work

21:36 We’re experiencing intermittent issues that we believe are caused by a software bug in the switch stack. We’re rebooting it now.

21:50 Network configuration has been reverted to the previous setup for the moment, all services should now be fully restored.

——-

23:08 – There appears to be some network latency/packet loss which is unrelated to the earlier network operations, we are currently investigating the cause.

23:27 -  The network has returned to optimal preformance, we apologise again for any inconvenience that this may have caused.

Valhalla MySQL issues (Resolved)

10.20am – A cPanel update has knocked out MySQL on Valhalla. We are working to get this resolved and will post updates here as and when we have more information.

Only services using MySQL are affected – Mail (POP & IMAP), static content such as HTML and dynamic content such as PHP is working as normal.

11:39 – We have discovered that the update has caused data corruption on a few databases. We are investigating options and will provide more information shortly.

12:00 – We have decided for expediency to restore all data from the snapshot we took this morning. Due to the size of the data it may take us a number of hours to fully restore all databases to a consistent state. We will update this post again once we have more information.

14:31 – Data has been successfully copied from the ArK snapshot taken this morning. However, it may be possible to restore the current data, which we’d much rather do if possible. Thank you for your continued patience.

14:32 – MySQL has been successfully started and all databases are in consistent state. No data lost.

Thank you for your patience during this incident.

 

Network Latency (Resolved)

16:07 – There appears to be network latency, of which we are investigating – We will update when we know more.

16:20 – Connectivity has returned to normal, we are currently investigating the cause of the latency.

Update: It appears that a distributed dential of service (DDoS) attack was aimed at one of our shared hosting servers, this in turn was causing some latency on the network.

mysql issue : ceres.uksrv.co.uk

9:35 Instability of the mysql service forced a restart. We are attempting to bring the service down gracefully.

9:45 MySQL and Apache were halted for a few minutes so we could restart under controlled conditions. The services are now available again.

Poseidon Unplanned Outage. (Resolved)

16:15 – We have been alerted to a possible issue with Poseidon. It is not responding to ping, engineers are investigating.

16:20 – Poseidon has been sent for a reboot, we are awaiting its return.

16:29 – Poseidon has restarted, all services should now start to operate as expected.

Ceres Server – Apache issue. (RESOLVED)

17:30 – Rebuild of apache to add additional PHP modules.

18:18 – Build complete, apache restarted.

18:26 – Apache failing to start, identified as build issue – rebuild of Apache + PHP modules commencing again.

18:45 – Apache restarted and is now serving sites again.

Network Maintenance – 23rd April 2014, 21:00 – 23:00 GMT (Completed)

Dear All,

Please be advised that The Bunker will be performing necessary upgrades on core network devices in our Ash (Kent) data centre. This maintenance is part of a number of phased changes to fully resolve network problems experienced in February and March of this year.

Please note that customers may experience a short loss in service while networking devices are restarted.

We thank you for your patience and understanding during this maintenance period.

23/04/14 – 22:32 – Please be advised that all scheduled maintenance for this evening has now been completed.

Heartbleed openSSL Bug Updates

Tuesday 8th April,

Krystal patched all affected servers.

Thursday 10th April,

Due to the recent discovery of the HeartBleed SSL bug, we have now replaced the private keys and SSL certificates across our fleet.

Further Information: Krystal’s retail (shared, reseller and premium) servers were all patched, or not initially vulnerable to the Heartbleed issue within a few hours of learning about it on 8th April.¬†While this prevented the information leakage associated with the bug, it could provide the opportunity for those able to capture customer packets over local networks to decrypt secure data. Therefore we decided to cycle the private keys the certificates protecting cPanel, Webmail, Apache, Exim, POP3/IMAP and FTP services across the fleet. This was done by early this morning on all but the reseller servers where we experienced a little delay in obtaining the newly signed certs (perhaps the whole world is trying to do the same?). We apologise for the brief inconvenience this has caused our resellers.

Follow

Get every new post delivered to your Inbox.

Join 62 other followers