حالة الشبكة أخبار و معلومات

At 21:51 MST Data102 saw multiple PRIs from our Colorado Springs local carrier to down simulatenously. This temporarily impacts inbound calling to 719 numbers on that trunk group.

Data102 is in contact with the upstream provider and is working for a speedy remedy.

** Update: 10:21pm: All of the PRIs are back up and appear to be functioning normally.

Data102's nimbus platform experienced a partial switch feailure at 15:25:27MST. Network connectivity was fully restored at ~15:31:49. Due to partial switch failure impacting storage availability, some Windows virtual machines automatically rebooted. No Linux-based server appears to have been impacted.

Data102 is assessing every server to ensure their operation at this time. Data102 will evaluate the switch failure and remedy accordingly.

Over the last few hours we have observed some issues with our Cogent uplink, including heavy packet loss. We are temporarily removing Cogent from our transit mix at this time.

** Update 15:34 **
We believe we have isolated and repaired the packet loss issues we have been seeing with Cogent and, after monitoring, have returned the network to full service.

Data102 saw our BGP sessions with Level3 this evening fail around 21:43, 21:48, and 21:52 MST. Our logs and routers indicate that the BGP hold timer failed. We have opened ticket a ticket with Level3 for more information as to what occurred – we did not observe any other BGP flaps, router reloads or other issues, just Level3 flapping on both v4 and v6.

Event Summary
At 2017-08-09 10:24am MST Data102 observed that the SIP proxy labelled SIP2 stopped routing SIP packets. Two separate alert systems informed of this issue, and at 10:25am, automated remediation restarted the SIP routing service on that server. By 10:27am the server resumed full functionality and service was restored.

Customers with endpoints that register to the SIP2 server may have been unable to create new outbound calls, and were unable to receive calls during this brief period; in-progress or calls in the connecting phase may have been interrupted. Calls that were already established were not impaced. Customers with endpionts registered to any other SIP proxy were not impacted.

Root Cause
The service interruption was caused by a software crash on SIP2. This software crash is due to a known memory allocation bug in the SIP proxy software. We have been advised that this kind of issue will continue to occur more frequently as our customer base and call volume increases.

Resolution & Scheduled Maintenance
Due to this being a known bug causing the SIP proxy software to crash, the resolution is to upgrade our software to avoid hitting the bug again. As such, we are scheduling an urgent software upgrade for 2017-08-10 at 12:30am MST.