Outage on Outbound Voice gateway - Breda (NL) (31.169.63.1, 31.169.63.2)
Incident Report for CM.com
Postmortem

We hereby update you with an official RCA (Root Cause Analysis) regarding the reported issues with outbound calls via our Outbound-01 platform.

At this moment, all traffic has been restored to regular quality standards and customer expectations.

Date & time:

Start: Thursday August 1st, 06:30

End: Thursday August 1st, 09:40

Reported issue(s):

  • Outbound calls set up via the Outbound-01 (Breda – NL) platform received a SIP 500 error code.
    During the reported time window, no outbound calls could be set up.

Root cause:

Due to the addition of new carriers and outbound routes to the platform, the routers ran into internal memory limitations during their daily routing calculation procedure.
This was not picked up by our monitoring and alerting systems, which caused a delay in finding the cause and implementing a solution.

Measures taken:

  • Increased memory size on all routers, on all voice platforms
  • Additional monitoring and automated checks in our Network Operation Center

If you have any further questions and/or are still experiencing issues, please do not hesitate to contact us via support@cm.n

Posted Aug 02, 2019 - 12:08 CEST

Resolved
This incident has been resolved.
Posted Aug 01, 2019 - 09:59 CEST
Monitoring
A fix has been implemented. We are monitoring active calls.
Posted Aug 01, 2019 - 09:40 CEST
Identified
The problem has been identified. The routers on the Outbound Voice gateway - Breda (NL) ran out of memory. We are currently increasing it's memory size and expecting the platform to be up and running on short notice.
Posted Aug 01, 2019 - 09:35 CEST
Investigating
We are currently investigating a (partial) outage on our Outbound voice trunk in Breda, the Netherlands. Affected IPs: 31.169.63.1, 31.169.63.2

We advise you to connect with one of our back-up trunks instead:
31.169.58.10, 31.169.58.11
or
31.169.63.81, 31.169.63.82
Posted Aug 01, 2019 - 09:23 CEST
This incident affected: Voice (Outbound Gateway Breda 2 (NL)).