Partial Outage

Incident Report for Ymonitor

Postmortem

We started to experience some delays in measurements on 07-11-2019 around 7:00. The measurements were not lost, but they were not displayed on the dashboards as they were not being stored in the database. The main reason for that was that our messaging queue was building up due to a memory utilization problem. We temporarily solved this memory utilization problem, which resolved the incident. We are going to investigate the cause of it and work on a permanent solution.

We apologize for the inconvenience that the incident might have caused.

‌If you have further questions please do not hesitate to reach to your contact person at Ymor.

Posted Nov 07, 2019 - 15:54 CET

Resolved

The issue has been fixed and currently all measurements and alerts are visible in Ymonitor.

Posted Nov 07, 2019 - 07:48 CET

Monitoring

The root cause has been found and we are working hard on fixing the problem.
A fix has been implemented and we are monitoring the results.
We apologize for this inconvenience.

Posted Nov 07, 2019 - 07:33 CET

Update

We are continuing to investigate this issue.

Posted Nov 07, 2019 - 07:11 CET

Update

We are continuing to investigate this issue.

Posted Nov 07, 2019 - 07:09 CET

Investigating

We are seeing that for some customers there is currently no new data being added to Ymonitor. This means that these customers won't see new measurements on ymonitor.nl or on dashboards. Also alerts won't be triggered currently for the affected customers.

We are investigating the issue and we will post updates as we progress.

Posted Nov 07, 2019 - 07:04 CET

This incident affected: Ymonitor Dashboards, ymonitor.nl, API, and Alerting.