Apr 23 2012
22 Comments
Outage, Server

Aphrodite Issues (Resolved)

We are experiencing some difficulties with aphrodite.krystal.co.uk – the server was rebooted due to an unknown software error around 17:05hrs. It is now experiencing an incredibly high load at what is a peak time. We are currently waiting for the server to stabilise pending an investigation into the nature of the original problem.

Update 24/04 19:32 One of the hard drives in Aphrodite’s RAID array has failed. We’re now removing the affected hard drive and will boot it as soon as it’s possible to do so safely. Please bare with us.

Update 19:48 The server is now back up, though in a “degraded” RAID mode – it only has 1 hard drive instead of the normal 2. Performance will be adversely affected until the RAID array is rebuilt. Thanks for your patience.

Update 20:33 A replacement hard drive has replaced the failed one and the RAID array is currently rebuilding. We expect this will take around 12 hours during which time server will be less responsive than normal as the data set is rebuilt. We therefore advise users to refrain from any heavy usage and be aware there is a higher than normal chance of an outage.

Update 21:41: The rebuild is at 20%

Update 00:15 The rebuild is at 80%

Update 25th April 01:00 The rebuild of the raid array is now complete. We can’t be 100% sure that this problem was caused by a failing disk, and so we will maintain our extra monitoring on this server for some time.

Apr 16 2012
Leave a comment
Unplanned Maintenance

Spamhaus RBL Aphrodite (Resolved)

If your hosting account is based on aphrodite.krystal.co.uk you may experience email messages being rejected by your recipients this morning. One of our hosting customers managed to get a botnet controller installed in their account, and this has tarnished our IP reputation. The offending site/material has been deactivated, and we are working with Spamhaus to rectify the issue. Predictably we have had an unprecedented number of calls regarding this, and we apologise to anyone who has been trying to get through.

Update: 11:07:49 BST

Spamhaus informed us that the record was removed, and should propagate within the hour.

Update 12:26:00 BST Resolved

http://www.spamhaus.org/query/ip/77.72.0.66

Tagged
Mar 24 2012
6 Comments
Network

Sahara Connection Problem (Resolved)

We are aware of intermittent packets loss with Sahara.

The server has been rebooted at 19:20 and techs are investigating.

We’ll update this ticket in due coure. Thanks for your patience.

Tagged
Feb 01 2012
Leave a comment
Scheduled Maintenance

Billing System & Client Area Upgrade (Completed)

23:00 GMT on Wednesday 1st of February 2012 – We are currently performing an billing system upgrade and will be back shortly.

Update 23:06: The upgrade has been completed successfully. Thank you for your patience.

Jan 31 2012
10 Comments
Unplanned Maintenance

DNS Disruption (Resolved)

Due to an administrative error, domains that met the following conditions:

1.) Are not registered on our nominet tag – KRYSTAL
2.) Were pointing at ns1.krystal.co.uk and ns2.krystal.co.uk

are *temporarily* set to ns1.groupabc.co.uk and ns2.groupabc.co.uk

Update: The DNS issues has now been fixed – although it may take an hour or so for things to return to normal.

We are working with Nominet to reverse the change.

We apologise for any inconvenience the DNS disruption has caused today, and would like to assure our clients that we are doing all we can to resolve this matter as quickly as possible.

Jan 18 2012
7 Comments
Unplanned Maintenance

Emails being bounced – Resellers (Resolved)

We have uncovered a client account that has been sending out large quantities of spam from our ceres reseller server. Unfortunately this has resulted in a number of blacklists adding our IP address. We have taken measures to prevent further issues, and have been contacting the list maintainers to expedite the restoration of the reputation of this server’s main IP address.

We apologise for the inconvenience this has caused, and would like to assure you that we are dealing with the matter as swiftly as we can.

Update 11:44: Normal delivery has been restored – http://www.spamcop.net/w3m?action=blcheck&ip=77.72.4.98

Tagged ,
Dec 20 2011
Leave a comment
Scheduled Maintenance

Emergency Maintenance (Completed)

UPDATE 01:34 Wednesday 21st December 

The network maintenance has now been completed. We have re-arranged the core network, upgraded software across the board and installed the first stage of new equipment to replace the failed equipment from last months outage.

—————————————————————————

Our systems have notified us that our core switches are operating at too high a temperature.

As a result we will be moving the switches tonight, after midnight, and expect that there could be up to 10 minutes of downtime, though we will seek to minimise disruption as much as possible.

Hopefully this planned action will not cause significant inconvenience, and is certainly more desirable than being plunged in to an unplanned outage through hardware failure.

All email will be kept and held br StriKe and delivered when the network comes back up.

Thank you for your patience and understanding.

Dec 20 2011
Leave a comment
Migration

Dublin & Homer Migration (Completed)

Important Update – Users may experience mail problems if still using the .purplecloud.com mail server. Please use mail.<YOURDOMAIN> instead. Webmail is now at <YOURDOMAIN>/webmail

If you’ve have MySQL errors it’s likely because cPanel doesn’t permit the main account user being a database user too (we didn’t realise this would happen and have been fixing as many as possible)

If we’ve not got you yours yet you can fix by creating a new database user via your cPanel (under “MySQL Databases”) pair it with your database and update the config file, or contact us to have us do it for you.

10:00 20th December – Dublin & Homer migrations have begun, we will keep you updated here.

18:00 20th December  – Dublin is over 50% transfered, and Homer is around 35%. We expect the transfers to continue throughout the night with a small pause for our network maintenance at 00:00 on the 21st.

20:45 20th December  – Dublin is now 95% transfered, and a few accounts that failed the initial transfer are being manually audited and transfered. Homer is around 50% transfered, and will be continuing long into tomorrow!

16:29 21st December – All accounts have now moved over.

If you’re having any issues please contact support at http://www.krystal.info

Dec 13 2011
Leave a comment
Migration

Auckland Migration (Completed)

The migration from Auckland started at 11:00 today and is still ongoing, clients will be seamlessly migrated throughout the night.

Homer and Dublin accounts will be migrated on Tuesday 20th.

Please see http://krystal.co.uk/welcome/ to determine which server you are on and for more information

Update: Auckland has now been migrated.

MAIL
Users should update their mail servers to either mail.<YOURDOMAIN> or dionysus.krystal.co.uk
Webmail is now at <YOURDOMAIN>/webmail

CPANEL
cPanel welcome email with usernames and passwords will be sent around shortly.
Your cPanel is at <YOURDOMAIN>/cpanel

DNS
Clients with external DNS should set their nameservers to ns1.krystal.co.uk and ns2.krystal.co.uk

If you have any problems please visit www.krystal.info

Welcome to Krystal

Nov 15 2011
35 Comments
Network, Outage

Ongoing Network Issue (Resolved)

We are currently experiencing an ongoing network issue that is causing full network outages, we are currently looking into this, and will keep you updated.

We apologise for any inconvenience caused. We are unable to provide an ETA for this to be resolved, but be assured we are working as fast as we can to regain connectivity.

UPDATE – 17:36 – There seems to be a problem with our core routing system. We are currently investigating, and are talking with the manufacturer. We expect there to be intermittent service for some time.

UPDATE – 18:05 – We now unfortunately have a total network blackout due to a failed piece of equipment. We are attempting to migrate to a contingency setup shortly.

UPDATE – 19:41 – We are working on the configuration for a new piece of hardware, with the aim of going live as soon as possible.

UPDATE – 20:05 – We are now running on the new equipment, and connectivity should be restored. We will be bringing up more connectivity over the evening, so there may be the occasional blip as routes get updated. We will need some time to work out the full cause of the original problem, but we will update here when we know more.

← Older posts
Follow

Get every new post delivered to your Inbox.

Join 27 other followers