Outage on Outbound Voice gateway - Breda (NL) (31.169.63.1, 31.169.63.2)
Incident Report for CM.com
Postmortem

We hereby update you with an official RCA (Root Cause Analysis) regarding the reported issues with outbound calls via our Outbound-01 platform.

At this moment, all traffic has been restored to regular quality standards and customer expectations.

Date & time:

Start: Thursday August 1st, 06:30

End: Thursday August 1st, 09:40

Reported issue(s):

  • Outbound calls set up via the Outbound-01 (Breda – NL) platform received a SIP 500 error code.
    During the reported time window, no outbound calls could be set up.

Root cause:

Due to the addition of new carriers and outbound routes to the platform, the routers ran into internal memory limitations during their daily routing calculation procedure.
This was not picked up by our monitoring and alerting systems, which caused a delay in finding the cause and implementing a solution.

Measures taken:

  • Increased memory size on all routers, on all voice platforms
  • Additional monitoring and automated checks in our Network Operation Center

If you have any further questions and/or are still experiencing issues, please do not hesitate to contact us via support@cm.n

Posted 3 months ago. Aug 02, 2019 - 10:08 UTC

Resolved
This incident has been resolved.
Posted 3 months ago. Aug 01, 2019 - 07:59 UTC
Monitoring
A fix has been implemented. We are monitoring active calls.
Posted 3 months ago. Aug 01, 2019 - 07:40 UTC
Identified
The problem has been identified. The routers on the Outbound Voice gateway - Breda (NL) ran out of memory. We are currently increasing it's memory size and expecting the platform to be up and running on short notice.
Posted 3 months ago. Aug 01, 2019 - 07:35 UTC
Investigating
We are currently investigating a (partial) outage on our Outbound voice trunk in Breda, the Netherlands. Affected IPs: 31.169.63.1, 31.169.63.2

We advise you to connect with one of our back-up trunks instead:
31.169.58.10, 31.169.58.11
or
31.169.63.81, 31.169.63.82
Posted 3 months ago. Aug 01, 2019 - 07:23 UTC
This incident affected: Voice (Outbound Gateway Breda 2 (NL)).