• We scheduled internal maintenance from 5:00 AM to 6:00 AM CT to deploy a core mitigation update to our global network.
• As part of our standard process, we apply configuration changes and core updates during off-peak hours on weekdays to minimize risk, and we typically do not expect any customer impact.

Incident Timeline

• 5:00 AM CT: The mitigation update was deployed to AMS as part of a staged rollout. As a result, AMS began sending synchronization pulses to other edge PoPs. These pulses unintentionally triggered a previously unknown critical bug in an unrelated module.
• 5:05 AM CT: ASH began experiencing packet loss due to the bug.
• 5:06 AM CT: DFW began experiencing packet loss due to the bug.
• 5:09 AM CT: Alarms indicated downtime in DFW and ASH. We immediately applied a change to disable the synchronization pulses, preventing further impact to other PoPs.
• 5:15 AM CT: DFW connectivity was fully restored.
• 5:16 AM CT: ASH connectivity was fully restored.
• 5:56 AM CT: We began deploying an emergency mitigation update to patch the critical bug and prevent recurrence.
• 6:16 AM CT: The emergency update deployment was completed.

We are committed to continuously improving our network resiliency and taking proactive measures to prevent similar issues in the future. Should you have any questions or require further information, please don’t hesitate to reach out.

CosmicGuard Dallas has gone offline.