Affected services:

  • Onyx

Onyx - Degraded performance

Opened on Wednesday 13th October 2021, last updated

Resolved — This is confirmed as resolved as of 22/10

Posted by Krystal

Monitoring — We believe the platform is now stable. The analysis continues and we will update this thread with further information when available. Thank you for your patience.

Posted by Krystal

Investigating — A further period of instability was experienced with Onyx for around 10 minutes. We continue to troubleshoot the difficulties with our storage vendor.

Posted by Krystal

Investigating — The issue has stabilised and we are investigating the performance degradation with our storage appliance vendor. Further updates will be provided via this thread.

Posted by Krystal

Investigating — Unfortunately the issue has re-occurred and appears to be affecting multiple compute nodes as such we are investigating this as a priority. Apologies for any disruption caused.

Posted by Krystal

Monitoring — The load spike has subsided and sites will be running normally. We're still aware of an intermittent storage performance issue with a much smaller impact, and will provide a further update on this.

Posted by Krystal

Investigating — We are seeing load issues across multiple compute nodes and are working on this as a priority.

Posted by Krystal

Identified — The vast majority of sites on the platform are loading normally, but we are still seeing occasional storage latency for some sites and are continuing to investigate this. Please contact support if you are seeing performance issues.

Posted by Krystal

Identified — We've identified an issue with reduced performance for some sites on the Onyx platform and are investigating this now.

Posted by Krystal