Estat de la xarxa News & Information

Data102 is aware of an inbound dialing issue in the Colorado Springs region, stemming from a CenturyLink interconnect problem. We are working aggressively to resolve this as soon as possible.

At 21:51 MST Data102 saw multiple PRIs from our Colorado Springs local carrier to down simulatenously. This temporarily impacts inbound calling to 719 numbers on that trunk group.

Data102 is in contact with the upstream provider and is working for a speedy remedy.

** Update: 10:21pm: All of the PRIs are back up and appear to be functioning normally.

Data102's nimbus platform experienced a partial switch feailure at 15:25:27MST. Network connectivity was fully restored at ~15:31:49. Due to partial switch failure impacting storage availability, some Windows virtual machines automatically rebooted. No Linux-based server appears to have been impacted.

Data102 is assessing every server to ensure their operation at this time. Data102 will evaluate the switch failure and remedy accordingly.

Over the last few hours we have observed some issues with our Cogent uplink, including heavy packet loss. We are temporarily removing Cogent from our transit mix at this time.

** Update 15:34 **
We believe we have isolated and repaired the packet loss issues we have been seeing with Cogent and, after monitoring, have returned the network to full service.

Data102 saw our BGP sessions with Level3 this evening fail around 21:43, 21:48, and 21:52 MST. Our logs and routers indicate that the BGP hold timer failed. We have opened ticket a ticket with Level3 for more information as to what occurred – we did not observe any other BGP flaps, router reloads or other issues, just Level3 flapping on both v4 and v6.

Between 2:30pm and 3:00pm on 10/7/2017, Data102 observed multiple T1/PRIs lose connetivity to an upstream voice carrier.

These circuits carry the majority of traffic for the Colorado Springs/719 region of our service. Customers with service via this carrier have been unable to make or recieve calls for the duration of the outage. 

As of 4:30pm, we have remapped all of our outbound dialing to avoid these circuits; inbound service continues to be impacted.

** Update 4:40pm **
We have learned that our upstream carrier suffered a power outage due to vandalism at their head end. Power has been restored, and services are being repaired.

** Update 6:05pm **
Notified of inbound service restoration, and was confirmed at approximately 6:10pm.

** Update 6:43pm **
Notified of outbound service restoration, reconfigured trunks and tested outbound service at 7:38pm (not urgent, as outbound was being processed through alternate upstreams without issue)

** Update 10/9/17 @ 11:30am **
We are aware of intermittent inbound dialing, and of an "All circuits are busy" message occasionally playing on connect. This is due to various underlying circuits being down on our trunk group, and some calls attempting to use those downed services. We have remedied more than 50% of those, and are troubleshooting the additional capacity.

** UPdate 10/9/17 @ 12:32pm **
Technicians are en route to the far end of one of our down hi-cap circuits to replace equipment that was damaged during the outage. Upon replacement and cable swapping we expect full operation.

** Update 10/9/17 @ 2:16pm **
Full service has been restored across all uplinks and carriers.

We have become aware that the DirectMX platform has failed to update anti-spam settings for at least 24 hours, resulting in a slight increase in spam being delivered to end users. We are aware of this issue and are actively investigating.

12:21pm
This issue has been isolated to a hung process, and has been resolved, and definition updates have been completed.

Issue Description
Earlier this morning Data102 became aware of intermittent outbound calling problems, where end users would place a call and receive a message indicating "Destination Unavailable" or similar. Inbound calls were not impacted.

Issue Root Cause
After debugging, Data102 determined that an upstream carrier's voice gateway became unavailable and unresponsive at an unknown time in the morning. Due to this failure, an alternate service was used for call completions. This failover service has a limited number of call paths available (46), and intermittently this alternate service became congested, resulting in a failure to complete outbound calls.

Issue Resolution
After determining that the main voice gateway was unavailable, Data102 re-configured the call platform to forward calls to an alternate voice gateway. This promptly resolved the call flow issue, and service was restored. Data102 has put additional monitoring in place of upstream service provider availability, and setup additional alarming and auto-redirection services in case of gateway failure.

Event Overview

• Outage Scope: Customers using MagnaVoIP phone services behind NAT firewalls without ALG
• Outage Date: 2016-07-11
• Outage Type: One-way audio / loss of service
• Severity: Critical
• Root Cause: Bug causing incorrect NAT setting on customer-facing SIP proxy

Issue Description
Service: CDP Backups
Impact: Critical

Issue Detail
Yesterday at approximately 3:05 PM MST, Data102's Colorado Springs-side SAN02 lost two hard drives during a patrol read, destroying a 6-drive RAID5 set. At approximately 5PM, we completed swapping hard drives and began background-initializing the array, with estimate completion time of 48 hours.

This morning at ~9:15AM, we began the resync process - copying the data from our Denver-side SAN01 to SAN02 to restore all services. This process has an estimated completion of ~33 hours.

In the meantime, to ensure data consistency, all CDP services have been temporarily disabled. 

Updates
01/01/16 @ 12:45pm: The initialization and copyback are proceeding, at 37% and 8.5% respectively.
01/02/16 @ 08:30am: Process continues, 57% , 59%
01/02/16 @ 11:50pm: Rebuild and data transfer complete
01/03/16 @ 08:15am: All backup jobs started
01/03/16 @ 11:30am: Backup jobs audited, repaired as needed
01/04/16 @ 09:28am: All backup jobs appear to completed as intended


Our apologies for this significant inconvenience. We understand that backups are important, and are working as diligently and as swiftly as possible to restore service without losing any data. Should the need arise to take a backup or to restore a backup, we MAY be able to facilitate your request, depending on the location of your data. Please open a trouble ticket with your request and we will respond promptly.

Data102 Operations Team
VP/Ops Randal Kohutek
888-328-2102 / 719-387-0000