Network Attack – 23 June 14:45

[14:45] – We’re experience heavy traffic down all of our networks
[15:00] – Confirmed as a DDoS attack against a number of ips in our range
[15:05] – We have blacklisted a number of ips in our range to keep sessions down. We are currently being hit by over 60,000 ip addresses and over 500,000 sessions.
[15:10] – Attack has potentially changed vectors, our engineers are still working on the problem.
[15:25] – Traffic has been flowing normally for some time now, we’ve requested extra DDoS protection upstream and we’ve un blacklisted some i.p addresses.

Premium Clients – XP mail

10.30am : To ensure our premium servers will pass current PCI compliancy scans we have had to update the ciphers used on these machines. Unfortunately this update has resulted in an issue with Microsoft XP – as XP is now unsecured due to Microsoft no longer supporting or providing updates for it.

If you are using XP, are on one of our Premium servers and having issues please contact our support team via support.krystal.co.uk.

Suspected Network Attack

22:55 – Our support staff were automatically alerted to an influx of traffic on our network.

22:59 – Traffic identified as UDP/TCP Syn flood, upstream provider contacted to apply mitigation techniques.

23:09 – We are actively working with our upstream providers to mitigate this.

23:30 – Attack on network appears to have been stemmed.

We will continue monitoring the situation and apply updates as and when required, we thank you for your patience and understanding whilst we work to resolve this.

Kloud Planned Maintenance

[04:33] – We are starting maintenance now, and over the next hour we will start powering down VMs.

[05:15] – All VMs now offline, we are upgrading software and rebooting Hypervisors

[05:35] – All Hypervisors are up to date and rebooting

[06:07] – All VMS are rebooted – please contact support if you’re having problems with anything.

[06:10] – We are updating all UI and web control software, you may be unable to reboot/edit your VMS for some time whilst we bring everything up to date.

[08:57] – Everything is now fully up to date. Please contact support if you’re experiencing any issues.

Olympus mail log in issue

12.10 – There is an issue affecting some clients logging in to their mail on our Olympus server. This is being investigated by our tech team – we apologise for any inconvenience caused during this time.

13.20 – Our team are still investigating the mail log in issue on Olympus. The issue is intermittent and mail is still being queued however it isn’t being delivered at present due to the connectivity issue. We appreciate the frustration of this situation and are working to resolve this as quickly as possible.

14:00 The intermittent log in issue affecting customers on Olympus should now be resolved. If you are having any further issues connecting to mail on Olympus please may you enter a support ticket via support.krystal.co.uk and include your IP address from viewmyip.co.uk.

We would like to apologise for the inconvenience caused this afternoon due to the mail log in issue and we appreciate the patience that our clients have shown during this time.

Microsoft Mail Issue

11.30am : We are currently investigating an issue affecting a small amount of our clients collecting mail using POP3 & Mircrosoft Outlook (older versions) and Live Mail.

12.40am : This issue has now been resolved.

Service Disruption – Network Attack

Our network appears to be under attack but we are still investigating, we will keep you updated.

13:15 Network disruption reported, after investigating it appears to be malicious traffic.

13:21 Upstream Scrub applied by the Data Centre.

13:25 Attack started to relent.

13:26 Network appears to be performing normally.

Change Notice – ns1.krystal.co.uk

Due to updates in our internal infrastructure, the IP address of ns1.krystal.co.uk has been changed to 46.101.24.26. The zone files and glue data for all domains that are under our control have been updated already.

The original IP address 77.72.0.11 will still answer queries until Monday morning (20/4/2105) to give all external DNS plenty of time to expire the old records. This will ensure that NO DISRUPTION will take place during the transition.

If your domain is assigned to ns1.krystal.co.uk or ns1.uksrv.co.uk then you DO NOT need to take any action.

You DO need to take action if  you have domains which are NOT registered with Krystal, but DO contain their own authoritative nameserver (NS) records, as follows:

  • any “A” records pointing to 77.72.0.11 should be changed to point to 46.101.24.26
  • glue data (nameserver records) at the parent registry must be updated from 77.72.0.11 to 46.101.24.26

Total Failure of Backup System

We are currently experiencing a total failure of our backup system.

This includes both R1-Soft and cPanel backups.

This is due to a catastrophic failure in a large number of hard drives in our backup systems.

We are expecting a 100% data loss, and we are in the process of replacing all of these HDDs (50+ drives).

We are also in the middle of splitting up our backup systems so as to negate this scenario in the future.

We expect to take delivery of the new drives, and do the installation at some point next week. Until then we are unable to offer any restoration to any clients.

Update: Backups are now working again – but there is no historical data before May 30th.

We are still implementing our secondary backup system.

Olympus WordPress log in issues

13:00 – Due to an on going attack on Olympus we have currently disabled all WordPress logins at present.

As soon as things have calmed down on Olympus we will remove the block.

Thank you for your patience during this time.

Follow

Get every new post delivered to your Inbox.

Join 81 other followers