Important service notification

Incident Report for W2

Postmortem

On the 2nd July, at approximately 17:55 BST (16:55 UTC), we were alerted to an issue with our API endpoints, whereby customers were unable to authenticate correctly against the endpoint.

We raised an incident via our status page (https://status.w2globaldata.com/) to alert our customers as the API, and Portal were both affected by this incident.

We began fixing the issues, which was caused by an error when completing an internal process, and the fix was in place by 19:15 BST (18:15 UTC) whereby all customers could utilize the API endpoints again. Due to the nature of the incident, some customers will have seen successful interactions with the API sooner than 19:15. If you would like further information about your call history during this period, please contact your account manager, or the W2 support team (https://support.w2globaldata.com)

Once the API was working correctly, we began working on the Portal errors. This issue arose due to the incident with the API and caused errors attempting to login to the Portal, however all information in the dashboard was unaffected. The portal was fixed at 19:45 BST (18:45 UTC) and moved into our Monitoring phase.

At 23:35 BST (22:35 UTC), we had encountered no further issues, so resolved the Incident.

We have since updated our internal process so that the error cannot happen again, and will continue to build on our processes to ensure these issues don't recur in the future.

Posted Jul 03, 2020 - 15:35 BST

Resolved

Following the fix deployed at 19:45 we have been continuously monitoring the affected components and all behavior is as expected. We are now closing this incident but will continue to monitor the situation and a Root Cause Analysis will follow within the next 2 working days.

Posted Jul 02, 2020 - 23:35 BST

Update

We are continuing to monitor for any further issues.

Posted Jul 02, 2020 - 23:35 BST

Monitoring

We are pleased to be able to confirm that at 19:45 a fix was deployed to the affected services and since this time all services have been behaving as expected. We will continue to monitor the situation until we are confident we can close the incident. Thank you for your patience during this time and once again offer our apologies for any inconvenience caused.

Posted Jul 02, 2020 - 20:00 BST

Update

We are continuing to work on a fix for this issue.

Posted Jul 02, 2020 - 19:15 BST

Identified

Upon investigation, we can confirm that since 17:55 we have been experiencing service issues and we have identified the problem and we are actively working to resolve this as a matter of priority. We will continue to provide updates through our normal channels. Please accept our apologies for any inconvenience this may be causing you. We hope to be able to provide updates every 60 minutes.

Posted Jul 02, 2020 - 18:26 BST

Investigating

We are currently experiencing a potential incident.
Our internal monitoring systems have alerted us to a potential issue and we are currently investigating the cause and impact. We hope to provide a further update in the next 15-20 minutes. Please accept our apologies for any inconvenience this may be causing you.

Posted Jul 02, 2020 - 18:08 BST

This incident affected: SOAP API (Legacy) and Portal.