- Priority - Critical
- Affecting Other - Magnavoip
Between 2:30pm and 3:00pm on 10/7/2017, Data102 observed multiple T1/PRIs lose connetivity to an upstream voice carrier.
These circuits carry the majority of traffic for the Colorado Springs/719 region of our service. Customers with service via this carrier have been unable to make or recieve calls for the duration of the outage.
As of 4:30pm, we have remapped all of our outbound dialing to avoid these circuits; inbound service continues to be impacted.
** Update 4:40pm **
We have learned that our upstream carrier suffered a power outage due to vandalism at their head end. Power has been restored, and services are being repaired.
** Update 6:05pm **
Notified of inbound service restoration, and was confirmed at approximately 6:10pm.
** Update 6:43pm **
Notified of outbound service restoration, reconfigured trunks and tested outbound service at 7:38pm (not urgent, as outbound was being processed through alternate upstreams without issue)
** Update 10/9/17 @ 11:30am **
We are aware of intermittent inbound dialing, and of an "All circuits are busy" message occasionally playing on connect. This is due to various underlying circuits being down on our trunk group, and some calls attempting to use those downed services. We have remedied more than 50% of those, and are troubleshooting the additional capacity.
** UPdate 10/9/17 @ 12:32pm **
Technicians are en route to the far end of one of our down hi-cap circuits to replace equipment that was damaged during the outage. Upon replacement and cable swapping we expect full operation.
** Update 10/9/17 @ 2:16pm **
Full service has been restored across all uplinks and carriers.
- Date - 10/07/2017 14:29
- Last Updated - 10/09/2017 14:21
- Priority - High
- Affecting System - MagnaVoIP
At 2017-08-09 10:24am MST Data102 observed that the SIP proxy labelled SIP2 stopped routing SIP packets. Two separate alert systems informed of this issue, and at 10:25am, automated remediation restarted the SIP routing service on that server. By 10:27am the server resumed full functionality and service was restored.
Customers with endpoints that register to the SIP2 server may have been unable to create new outbound calls, and were unable to receive calls during this brief period; in-progress or calls in the connecting phase may have been interrupted. Calls that were already established were not impaced. Customers with endpionts registered to any other SIP proxy were not impacted.
The service interruption was caused by a software crash on SIP2. This software crash is due to a known memory allocation bug in the SIP proxy software. We have been advised that this kind of issue will continue to occur more frequently as our customer base and call volume increases.
Resolution & Scheduled Maintenance
Due to this being a known bug causing the SIP proxy software to crash, the resolution is to upgrade our software to avoid hitting the bug again. As such, we are scheduling an urgent software upgrade for 2017-08-10 at 12:30am MST.
- Date - 08/09/2017 10:24 - 08/09/2017 10:27
- Last Updated - 08/09/2017 13:03
- Priority - Medium
- Affecting System - DirectMX Email Firewall Platform
- Date - 02/11/2016 22:00 - 02/12/2016 01:00
- Last Updated - 02/05/2016 12:42