Important service notification

Incident Report for W2

Postmortem

On the 12th of May at approximately 15:11, our REST and SOAP APIs intermittently stopped being able to properly authenticate with some other internal services. This manifested in a partial outage of our API services until the issue was resolved at 19:55. A proportion of client calls to our services were affected.

The root cause of the issue was that our internal authentication provider made some changes to their authentication mechanism which had an unforeseen impact on our internal services, even though the change had been reviewed by Engineering ahead of time. The issue was esoteric in nature and required a concerted effort to resolve from the whole team to detect and resolve.

Posted May 21, 2021 - 11:39 BST

Resolved

Following the fix deployed at 19:55, we have been continuously monitoring the affected components and all behavior is as expected. We are now closing this incident but will continue to monitor the situation.

Posted May 12, 2021 - 20:19 BST

Monitoring

We are pleased to be able to confirm that at 19:55 a fix was deployed to the affected service and since this time all services have been behaving as expected. We will continue to monitor the situation until we are confident we can close the incident. Thank you for your patience during this time and once again offer our apologies for any inconvenience caused.

Posted May 12, 2021 - 19:57 BST

Update

We are continuing to work on the affected service and are currently testing a fix. Please be aware that this matter is our number one priority and we are deploying all available internal and external resources to resolve the problem. We will continue to provide updates on a 60-minute basis, should anything change in the interim we will update our status page accordingly. Please accept our sincere apologies for any continued impact this is having on you.

Posted May 12, 2021 - 18:57 BST

Identified

Upon investigation, we can confirm that since 16:30 we have been experiencing service issues and we have identified the problem and we are actively working to resolve this as a matter of priority. We will continue to provide updates through our normal channels. Please accept our apologies for any inconvenience this may be causing you. We hope to be able to provide updates every 60 minutes.

Posted May 12, 2021 - 18:05 BST

Investigating

We are currently experiencing a potential incident.
Our internal monitoring systems have alerted us to a potential issue and we are currently investigating the cause and impact. We hope to provide a further update in the next 60 minutes. Please accept our apologies for any inconvenience this may be causing you.

Posted May 12, 2021 - 17:23 BST

This incident affected: REST API and SOAP API (Legacy).