Hardware failure Voice platform ZA

Incident Report for CM.com

Resolved

This incident has been resolved.
Posted Jan 23, 2020 - 19:28 CET

Update

Resolved
Posted Jan 23, 2020 - 18:54 CET

Monitoring

The platform has been restored to its original state. All services are up and running. We will continue to closely monitor the platform.
Posted Jan 23, 2020 - 17:05 CET

Update

In order to resolve the issue, we need to perform some emergency maintenance. This maintenance will potentially cause a series of short service interruptions.

Gateway 2 (85.119.55.52) will go offline, please switch to Gateway 1 (85.119.55.51).

Start: 12:30 UTC
Duration: Approx. 1 hour

Next update: 13:00 UTC
Posted Jan 23, 2020 - 13:27 CET

Identified

We managed to get call traffic flowing again, but reliability cannot be guaranteed at the moment.
Posted Jan 23, 2020 - 09:01 CET

Update

More research into the effects now shows that most of our customers will not be able to call via this platform. The cause has been found and we are working on a resolution.
Posted Jan 23, 2020 - 08:51 CET

Investigating

It seems like something is wrong with the hardware located in Johannesburg, ZA, causing some Virtual Machines that are part of the Voice Outbound Platform to have stopped.
These VMs are running in groups, split over separate hardware, so it's unclear whether this problem actually affects traffic. First tests allow us to place calls on the platform, but customers might experience problems.
We are investigating both the cause and the effects.
Posted Jan 23, 2020 - 08:41 CET