Data is not available on Splunk

Incident Report for Ymonitor

Postmortem

Dear customers,

As you might have followed the updates on this page, we have gone through several incidents because of service disruptions of our provider AWS Amazon MQ. As indicated in the explanation of the Emergency Maintenance of November 10th, we took some measures to contain the impact of this series of incidents. Although we succeeded for Ymonitor systems, we experienced another incident on the same day with Splunk. During that incident unfortunately no data could be sent from Ymonitor to Splunk from November 10th, 01:00 (CET) to November 11th, 01:30 (CET).

Because of this, Splunk databases now has a gap of measurements and alert data. Our teams are working hard to fill this gap with data from the Ymonitor databases. Because it involves a lot of manual work, we cannot estimate when it will finish. Once it is done you will be able to see data from the above mentioned period on Splunk.

Should you have more questions please do not hesitate to reach out your consultants.

Kind regards,

Sentia part of Accenture

Posted Nov 14, 2022 - 14:17 CET

Resolved

After running the final checks we see that the measurement and alert data are ingested and processed by Splunk. This incident is closed now. We apologize again for the inconvenience and thank you for your patience.

Posted Nov 11, 2022 - 01:31 CET

Update

We are still continuing to work on the new queueing platform.

Posted Nov 11, 2022 - 00:36 CET

Update

We are preparing a new queueing platform to process our monitoring messages. This new queueing platform will be used by our Splunk instances. We expect to bring the new queuing platform live around 22:30.

Posted Nov 10, 2022 - 21:28 CET

Update

We are continuing to work on a fix for this issue.

In the meantime, we are continuing to manually check our monitoring systems and intervene where necessary.

We expect to publish the next update around 21:30 CET

Our apologies for the inconvenience

Posted Nov 10, 2022 - 16:47 CET

Identified

After the Emergency Maintenance of last night we are having issues with receiving Ymonitor measurements in our Splunk instances.

This is impacting customers who are using http://customer.apm.sentia.com/ and the Ymonitor Splunk App.

We also expect a delay in the maintenance of our Ymonitor scripts. The DEM Control Center normally uses Splunk to see what needs to be handled. Because our automation is failing due to current issues, we are currently doing manual checks.

Our teams are working to fix the Splunk integration as soon as possible. We will keep you up to date with the status.

Posted Nov 10, 2022 - 11:09 CET

This incident affected: Splunk.