SYXCLOUD - Access disrupted

Reported by

• Customers and monitoring

Affected product Categorization:

• Citrix
• Netscaler

Incident history

• Start downtime – 15:42 01/08/2024
• Stop downtime – 15:54 01/08/2024 (first recovery)
• Stop issues as consequence of the above downtime – 16:30 01/08/2024 > Action: to log off all disconnected profiles (final recovery)

Issue description & analysis

• A netscaler update was scheduled on 09/07/2024 to fix possible vulnerabilities signaled in: Citrix CTX677944, Citrix CTX677998, Citrix CTX678025, Citrix CTX678035, Citrix CTX678036, Citrix CTX678037.
• The update was scheduled at 01/08/2024.
• 15:00 01/08/2024 the process started to install the security update on the netscaler.
• This is done during business hours because we use a failover system to upgrade the Citrix appliance (‘In Service Software Upgrade’) which should result in a zero downtime upgrade. We also have done multiple updates using this system in the past without issues.
• Some minutes after the installed update, the netscaler appliances suffered from memory pressure, which caused a cascading failure of these systems and the customer downtime.
• After rebooting the affected systems, all issues were resolved.
• Some connections were disconnected and reconnecting failed. This was solved by logging off those sessions manually, which was done until 16:30.

Issue root-Cause

• Upgrade netscaler appliances because of security patch.

Measures taken to avoid this issue or similar issues:

• Currently were are building a test case environment to evaluate and test the ‘In Service Software Upgrade’ and see how this could happen and how to avoid the failure of this service with our Citrix partner.
• For the moment, no infrastructure updates are allowed during business hours as our zero downtime mechanic was compromised. Until we can confirm this is not the case any longer, we will cease all during business hours infrastructure updates.

Dear Customers,

All Citrix components are up and running again.
If you have any questions or issues, please call our support center.
We apologize for this inconvenience.
and thank you for your patience.
Team Vintia

Dear Customers,

We are currently experiencing a disruption on the level of our Citrix servers, we have identified the problem and are acting as fast as we can to get everything up and running again.

We apologize for this inconvenience.

Team Vintia