Beginning on Jan 8th at 1:25AM ET we lost connectivity at the Cincinnati 1 data center during a scheduled maintenance window on a shared fiber connection upstream from the data center. During this maintenance multiple other unforeseeable and unpreventable events occurred which caused a complete outage until 6:20AM ET.
RFO: ISP 1: Cincinnati Bell (Fiber Entrance 1) Root Cause: Construction equipment severed fiber cabling hauling IP Services into the data center.
Fix: Fiber was repaired and services restored.
ISP 2: Spectrum (Fiber Entrance 2)
Root Cause: Scheduled maintenance on local loop fiber optic services impacted IP Services not scheduled for impacting maintenance.
Fix: Fiber being spliced in the same local loop network was severed by a piece of construction equipment north of the existing maintenance site. Fiber was repaired at both locations and service restored.
ISP 3: CenturyLink (Fiber Entrance 2)
Root Cause: An equipment issue in conjunction with a planned maintenance in Norwood, OH impacted IP services.
Fix Action: Field Operations completed the planned maintenance and the IP NOC remotely bounced a node, restoring services.
Century Link Reason for Outage (RFO) Summary: On January 8, 2019 at 15:16 GMT, CenturyLink reported a service impact in Norwood, OH. The IP NOC was engaged and suspected an equipment issue was impacting IP services. Field Operations was engaged and advised that a planned maintenance to replace a splice enclosure was in progress. Further investigation revealed that during the maintenance, the fiber was tested as splicing activities were performed. After the fiber was tested, a card had shutdown, causing a portion of the protected path to fail. Later, an unknown issue impacted the remaining portion of the protected path and causing the nodes to become isolated. The IP NOC remotely bounced a node, restoring a subset of the impacted services. After, Field Operations completed splicing the fiber, completing the planned maintenance and restoring services back to a loss of redundancy state. IP services were not intended to be impacted, and as such were not notified of the planned maintenance as services should have been operational on the protected path. The IP NOC will continue to work with Field Operations to determine what caused the protected path to become impacted, as well as restore full redundancy. Any additional details will be provided upon ticket closure.
Next Steps: Beyond Hosting will work with its data center provider to increase the diversity of physical fiber optic cabling coming into the building and ensuring all paths are diverse and physically separated by considerable amounts of distance.
- This document may be updated as new information is received.
Posted Jan 10, 2019 - 11:00 EST
We are continuing to monitor for any further issues.
Posted Jan 10, 2019 - 11:00 EST
At this time all internet providers and services are operating normally. We will continue to monitor and provide an RFO and Action Plan on correcting this issue in the future.
Posted Jan 08, 2019 - 12:47 EST
The connection has been repaired. We will continue to monitor and provide all customers an RFO shortly.
Posted Jan 08, 2019 - 06:26 EST
Crews are still working on repairs. We have no estimate but can assure you that we are working as fast as possible.
We apologize for the inconvenience and will provide a complete RFO and action plan once resolved.
Posted Jan 08, 2019 - 06:12 EST
Repair crews are on site working to restore connectivity.
Still no ETA.
Posted Jan 08, 2019 - 05:17 EST
We are currently working with the upstream provider for more information. We believe that a shared redundant fiber optic pair has been severed and is currently impacting multiple data center customers including us.
Posted Jan 08, 2019 - 03:39 EST
CIN1 is currently experiencing a network outage caused by a fiber cut at an upstream ISP (Spectrum/CenturyLink). We currently have no ETA for restored services.
Posted Jan 08, 2019 - 02:48 EST
This incident affected: API, Cloud Dashboard, Cloud Environment Infrastructure, Cloud Hosting, and Support Website.