Ark Backups on Shared & Reseller Services

We are currently experiencing a problem with ARK backups on most of our shared servers. ARK is the name for our daily, incremental backups that usually get taken once per day, per server. At this time these backups will not be running until new hardware arrives from our supplier. We expect this to arrive next week.

Normal cPanel backups that are taken weekly and monthly are still occurring and are not affected. ARK backups on premium and dedicated machines are not affected.

The cause of this problem is the degradation of a node in our backup setup. Since this node failed, the rest of the backup system has been unable to take the strain, and this is why we have ordered new hardware to add more resources to the problem.

Olympus – Websites down

06:00 after a brief period of downtime, we are updating MySQL on this server to improve performance. While this takes place, the web service has been stopped to limit load. Other services remain enabled. This notice will be updated as soon as the service is re-enabled and the upgrades complete.

06:35 MySQL has been upgraded and all checks completed. Web and mysql services are available once more. A small number of users who transferred to us using cpanel accounts that mosts MySQL v4 password will be notified that passwords must be updated for their mysql users accounts. This only affects around 1% of users. Thank you for your patience.

07:00 A routine apache rebuild has failed unexpectedly – again we are investigating this and will return the server to full service as soon as possible. We apologise for the ongoing difficulties, but the work was most necessary, and will improve server efficiency.

07:20 web and mysql services are now all updated and stable. We apologise for any inconvenience this may have caused users.

Valhalla – Intermittent issues

10:00 – A handful of clients on Valhalla have been experiencing intermittent service issues this morning resulting in a 502 bad gateway error. If you are on Valhalla and are experiencing this please let our technical team know via

Thank you for your patience whilst we resolve this.

Ares – planned maintenance

5th Nov – The ares server is currently unavailable while new hardware is being installed. This was notified to all affected customers’ contact email addresses on 16th November. We will update this status as soon as the server becomes available once more, which we expect will be within the next hour or two.



Thar Down

20:45 – Thar down – Our technical team is investigating.

22:20 – Thar is back up, server rebooted and disk check, all operational.

Cpanel login issues – Valhalla & Elysium. (PROVISIONALLY RESOLVED)

14.32 – We have been alerted to a problem on Valhalla & Elysium where users are unable to login to cPanel, we are currently investigating this anomaly and will update you as soon as we know more.

We thank you for your patience and understanding whilst we investigate this issue.

14.51 – This issue has now been resolved however we are still investigating the root cause and will update when we know more.

Olympus intermittent connectivity

[09:30] – We’re investigating a problem with Olympus connectivity dropping out
[09:38] – We’re sending the machine for a reboot
[09:55] – We’re investigating a network issue affecting one of our core switches
[10.15] – We’ve fixed the network issue and everything should be working correctly again

Reseller/Premium Network Outage

[12:40] – We’re currently investigating a network outage on our Premium and Reseller networks. We will update this post as soon as we have any information. All e-mails will be queued on our mail cluster, and all servers are still online, they are just lacking network connectivity.

[13:01] – We’re investigating a failed optical networking cable that is causing alarms in out networking kit – engineers are en-route to our equipment to replace.

[13:25] – We’ve replaced the cable, and services are now all back online.

Network Attack

18:00 Commissioned new DoS filtering service, and updated network routing to mitigate future attacks. Read more about his on our blog.

10:00 All users should now be able to connect to their sites – if you are having any issues today please do let our tech team know via Once again we are incredibly grateful for the support & patience shown by everyone affected.


22:50 The force of tonights attack meant we saw over 720 million packets – a very large attack. Our tech team are working to mitigate this now.

22:20 Unfortunately another network attack looks to have just started again. We are putting in place DDoS filtering across the network.

21:00 Traffic to all servers has now been steady for a few hours but filtering still remains in place on Ordos. We will continue to monitor Ordos and will update once we can safely remove any protection in place. Please be advised this isn’t affecting website traffic to Ordos however some controls like accessing cPanel may still be interrupted for some time.

20:20 Traffic to Ordos has resumed however some filtering is still in place. We have been advised that the majority of the internet can now view sites hosted on Ordos. If you have any further issues connecting please bear with us whilst we organise the final filtering removals.

20:10 Service for Thar has fully resumed – if any clients on Thar are having issues connecting please let our tech team know.

20:00 We have requested the removal of the filters on Ordos & Thar – these should be removed ASAP now – again we apologise for the frustration this has caused and we appreciate the patience our clients have shown today through the disruption caused by these attacks.

16:50 – Currently due to the DDoS attacks on our infrastructure today we still have DDoS filters on Thar & Ordos to carry on the mitigation of the attacks. We expect this to be amended at around 17:30 -18:00 – however we are constantly monitoring so this might be sooner – and will be done as soon as is possible.

Once again we want to thank all of our clients for their support & patience during todays attacks.

14:50 We are continuing to monitor the situation – if any one has any further connection issues please let our technical team know – thank you again for your patience today.

14:10 We are continuing to suffer a heavy attack. The nature of a Distributed Denial of Service attack makes it impossible for us to tackle the problem by blocking individual attacking hosts. The hosts involved in the attack may include infected home computers, wifi routers, or any other device which can be controlled as part of a modern botnet. This kind of attack took Apple’s store down yesterday, to give some idea of how difficult it is to counteract such an attack. We cannot describe in any great detail the effect of the issue on our network, and that of our data carriers for security reasons – suffice to say the problem is large enough to be causing concern outside of Krystal. We are currently working hard with our partners to mitigate the attack, which is of an order of magnitude greater than any we have previously experienced.

Rest assured, we are all feeling incredibly sore, and we know exactly how much this is damaging our business, and yours. We will not rest until we have upped our game considerably to deal with attacks of this size in future. As soon as we have more solid information we will update again, but for now, the situation is very dynamic, and service availability continues to be affected.

Customer can expect email delivery (inbound) to be delayed, although mail should arrive eventually from the sender’s mail hosts once normal service is restored. It should also be made clear that this attack is intended to saturate/flood our network with rubbish in an attempt to prevent bone-fide traffic. This attack is not a security risk.

13:45 Unfortunately another network attack looks to have just started again. Once again we appreciate the frustration this is causing our clients and we are working to resolve ASAP.

11:40 Things are returning to normal here – we would like to take this opportunity to thank our clients for their patience this morning – we appreciate fully the frustration of these network attacks and we are currently in talks with our service provider regarding this.

10:45 We are currently investigating another attack on our network – we apologise for the disruption during this time and thank you again for your patience.

10:00 We have mitigated the attack and regular service should now be returning. We thank you for your patience whilst we resolved this.

09.45 We are currently investigating what appears to be a network attack. Our technical team are working to mitigate this as quickly as possible. We apologise for the disruption during this time and thank you for your patience whilst we resolve this.

cPanel security update – PHP/MySQL problems

This morning cPanel released an urgent security update – sadly, this is causing some problems for hosts using CloudLinux. The symptoms are loss of MySQL connectivity via the PHP scripting engine. We are working to correct these issues, and will update again here as the work progresses.


Get every new post delivered to your Inbox.

Join 93 other followers