CDP - Delayed data processing

Incident Report for CM.com

Resolved

This incident has been resolved.
Posted Mar 14, 2025 - 18:50 CET

Update

Based on our efforts from yesterday, the system has stabilized overnight and the system is now operating as expected.
Our team is actively monitoring the system to ensure continued stability and is focused on retrying any failed event data to keep all processes up-to-date and accurate.
Posted Mar 13, 2025 - 08:46 CET

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Mar 12, 2025 - 19:55 CET

Investigating

Following the latest update, we have decided to proceed with another system maintenance window. Although errors are decreasing, we aim to fine-tune and expedite this improvement further to enhance performance.
Please note that during this maintenance window, no events will be processed, and workflows will not be triggered. We will perform this maintenance between 19:00 and 20:00 CET time.
Posted Mar 12, 2025 - 18:12 CET

Update

We observe a positive decrease in errors and queues following our recent maintenance window. Our team will closely monitor these encouraging trends over the next hour to assess whether further emergency maintenance would be beneficial. We will provide our next update in 60 minutes or sooner if significant developments occur.
Posted Mar 12, 2025 - 17:24 CET

Update

Our ongoing database maintenance is progressing successfully. However, to ensure we achieve the best possible outcome, we require an additional hour beyond our original schedule. We now anticipate concluding the maintenance around 16:00.
Posted Mar 12, 2025 - 14:50 CET

Update

As mentioned in the previous update, we will perform maintenance between 14:00 and 14:30 CET time. During this timeframe no events and workflows will be processed, after the maintenance window this will be picked up again and continue.
Posted Mar 12, 2025 - 13:24 CET

Update

We are going to perform maintenance on our systems to help mitigate the ongoing issues. During this timeframe no events and workflows will be processed, after the maintenance window this will be picked up again and continue.

Please understand we are doing everything we can to mitigate the ongoing issues with the help of several partners. As soon as we know the timeframe we will post another update, but please be aware this is upcoming.
Posted Mar 12, 2025 - 12:52 CET

Update

Update: We are actively working together with our third-party providers to resolve the ongoing issues. Our team is fully committed to restoring normal operations as quickly as possible, we are sorry for the inconvenience.
Posted Mar 12, 2025 - 09:59 CET

Update

We are working on implementing the solution to resolve these issues.
Posted Mar 11, 2025 - 16:56 CET

Identified

We have identified the issue causing delays, as well as failures in processing events and syncing segments to the address book. We are working on implementing a solution to resolve these issues.
No events have been lost during this issue, and failed events will be reprocessed once the fix is implemented.
Posted Mar 11, 2025 - 14:55 CET

Update

We are continuing our investigation into this issue.
Posted Mar 11, 2025 - 13:48 CET

Investigating

We are experiencing delays in data processing and loading times. Our team is actively investigating the issue.
Posted Mar 11, 2025 - 09:42 CET
This incident affected: Mobile Marketing Cloud (Customer Data Platform).