Change Notice – ns1.krystal.co.uk

Due to updates in our internal infrastructure, the IP address of ns1.krystal.co.uk has been changed to 46.101.24.26. The zone files and glue data for all domains that are under our control have been updated already.

The original IP address 77.72.0.11 will still answer queries until Monday morning (20/4/2105) to give all external DNS plenty of time to expire the old records. This will ensure that NO DISRUPTION will take place during the transition.

If your domain is assigned to ns1.krystal.co.uk or ns1.uksrv.co.uk then you DO NOT need to take any action.

You DO need to take action if  you have domains which are NOT registered with Krystal, but DO contain their own authoritative nameserver (NS) records, as follows:

  • any “A” records pointing to 77.72.0.11 should be changed to point to 46.101.24.26
  • glue data (nameserver records) at the parent registry must be updated from 77.72.0.11 to 46.101.24.26

Total Failure of Backup System

We are currently experiencing a total failure of our backup system.

This includes both R1-Soft and cPanel backups.

This is due to a catastrophic failure in a large number of hard drives in our backup systems.

We are expecting a 100% data loss, and we are in the process of replacing all of these HDDs (50+ drives).

We are also in the middle of splitting up our backup systems so as to negate this scenario in the future.

We expect to take delivery of the new drives, and do the installation at some point next week. Until then we are unable to offer any restoration to any clients.

Update: Backups are now working again – but there is no historical data before May 30th.

We are still implementing our secondary backup system.

Olympus WordPress log in issues

13:00 – Due to an on going attack on Olympus we have currently disabled all WordPress logins at present.

As soon as things have calmed down on Olympus we will remove the block.

Thank you for your patience during this time.

Hera planned upgrade

As part of our ongoing upgrades to our fleet we are upgrading your server (Hera) to SSD drives. This will mean faster loading sites, and better all round performance.

We are planning on doing this upgrade on February the 11th (next Wednesday) starting in the evening, and finishing late Wednesday or early Thursday. We plan for the server to be offline for up 12 hours in total, but from previous experience we can usually have all the transfers done in less than 6 hours.

18:00 – The planned Hera upgrade has begun.

19:00 – Accounts are being restored on the new SSD server.

20:36 – Hera is now back online – please raise a ticket if you’re having any problems.

Service Disruption – Network Attack

It appears another large attack is being aimed at our network.

19:12 – We were alerted to latency on our network.

19:18 – Networks are now investigating and applying mitigation techniques.

19:42 – Attack identified as SSDP UDP amplified network attack.

19:44 – Dropped identified destination of amplified traffic, network latency and speeds returning to normal.

We appreciate your patience and understanding during this period.

20.32 Olympus and all other servers should now be returning to normal speeds. We would like to take this opportunity to thank our clients for their support during this time. We appreciate the frustration these attacks cause & are incredibly grateful that our clients have been so understanding.

(RESOLVED) Service disruption – server attack

22.45: Our servers were being subjected to attacks starting around 22.45 this evening.

Our team are working on this now and we will keep the status updated.

23.15: Service seems to have returned to normal for the majority of users however a few Virgin Media users have reported issues connecting still.

[COMPLETED] Scheduled Network Maintenance – 19:00 – 01:00

Dear All,

As part of The Bunker’s continued upgrade to their core network, the final meshing of additional network routes and throughput will be applied this evening between 19:00hrs and 01:00hrs.

The majority of customers will see little or no noticeable change, however as with all technology there is a small chance this will result in some latency or packet loss while changes are being made.

We thank you for your patience and understanding during this maintenance period.

BT / Virgin Connectivity Issues

10.15am – We have been made aware by some clients using BT Broadband that they are unable to connect to some Krystal services. After investigating we have found BT are having what appears to be wide spread connectivity issues which aren’t related to our hosting.

BT have a service page here which details the issues they are having this morning.

11.15am – Both BT & Virgin clients have reported difficulties reaching our network this morning. We believe this is the result of an issue at BT which they aim to have resolved by 12:30.
For more information please see here.

Valhalla – High load issues / Apache issue.

15:00 – We have experienced high loads with the Apache/Nginx on Valhalla, at the moment it is not clear what is causing the issue however we will update this post as soon as we know more.
15:07 – Apache/Nginx restarted and appear to be happy, we will continue to monitor it.

We thank you for your patience and understanding.

 

(RESOLVED) Network connectivity problems

21:05 – We have stemmed the flow of the attack on our network, however, we will continue to monitor this over the coming hours.

20:51 – We are currently experiencing network connectivity problems, initial investigations shows that there is an attack being directed at our network.

We appreciate your patience whilst we work on resolving this issue.

Follow

Get every new post delivered to your Inbox.

Join 80 other followers