Voice - Degraded performance NL (+31) outbound calls
Incident Report for CM.com
Postmortem

We hereby update you with an official RCA (Root Cause Analysis) regarding the reported issues with outbound calls to Dutch (+31) numbers.

At this moment, all traffic has been restored to regular quality standards and customer expectations.

Date & time:

Start: Monday July 8th, 06:30

End: Tuesday, July 9th, 14:30

Reported issue(s):

  • One-way audio
  • Robotic voices
  • Increased call setup times, resulting in an increase of cancelled calls
  • Increased % of failed calls

Root cause:

The cross-connection in our Amsterdam datacenter with our primary carrier for NL (+31) traffic was malfunctioning.

This caused the reported issues to occur for a majority of the traffic to NL numbers.
All traffic to NL numbers was routed to secondary carriers. Due to insufficient capacity at some of these secondary carriers, an increase of failed and cancelled calls occurred.

Measures taken:

  • Replaced faulty cross-connect, enabling us to reinstate routing for primary carrier
  • Requested capacity upgrade with secondary carriers
  • Initiated interconnect with 2 additional carriers
  • Added additional monitoring and alerting tools regarding carrier capacity

If you have any further questions and/or are still experiencing issues, please do not hesitate to contact us via support@cm.nl

Posted Jul 10, 2019 - 09:25 CEST

Resolved
We have identified the cause to be with one of our primary interconnects for NL (+31) outbound traffic. Until further notice this carrier has been disabled and traffic is being routed through our other carriers. An official RCA via e-mail will follow to all clients that were affected by the quality disturbance. In case you still experience issues with calls through one of our platforms, please contact our Support department or your account manager.
Posted Jul 08, 2019 - 19:55 CEST
Update
We are continuing our monitoring for further issues. We currently see traffic to NL (+31) destinations performing as expected. If you still experience specific quality issues, please contact us directly with call examples.
Posted Jul 08, 2019 - 15:00 CEST
Monitoring
We have identified an issue with one of our downstream carrier interconnects to be the likely cause of the degraded quality to +31 calls. We have rerouted the traffic meanwhile and are monitoring performance. In case you are still experiencing quality issues, please contact our support desk directly.
Posted Jul 08, 2019 - 10:08 CEST
Update
We are continuing to investigate this issue.
Posted Jul 08, 2019 - 09:53 CEST
Investigating
We are currently investigating an issue that potentially causes quality issues and degraded performance on calls towards NL (+31) destinations.
Posted Jul 08, 2019 - 09:52 CEST
This incident affected: Voice (Outbound Gateway Breda 1 (NL), Outbound Gateway Breda 2 (NL), Outbound Gateway Rijen (NL)).