Dear customers,
On 12 January 2022 around 21:20 we noticed that some of the endpoints of our Ygate API were failing to respond. Our teams started directly on finding the root cause and solving the problem.
The impact
Some of our users who use the Ygate API’s “ygate.ymonitor.nl” endpoints to retrieve measurement and alert data could not get any response from the API. The endpoints of the “ymonitor.nl/ygate” were not impacted. Therefore users who use those endpoints did not have any problems. Other functions of the Ymonitor system were not impacted.
The resolution
We immediately noticed that the problem was caused by one of our service providers, namely Bitbucket. They have later reported on their status page that some repository archive downloads on AWS CodePipeline were disrupted. We decoupled the Bitbucket downloads from our AWS CodePipeline as soon as we could and replaced the deployment manually to solve the problem. Then the Ygate API service returned back to normal around 21:55.
The details of the Bitbucket incident can be seen here.
Corrective and preventive actions
The root cause of the problem was a third party service that we depend on, so we could not solve the root cause. Therefore we applied the quickest workaround that we could to ensure that our service resumed normally.
Although it seems like there is not much that we can do to prevent such an incident from happening again in the future, we will investigate what extra measures we can take.