Previous incidents

Post Mortem Report:

We asked CosmicGuard to add a new /24 IP subnet to our Ashburn location earlier today.

CosmicGuard stopped announcing our prefixes in Ashburn because of a configuration error when adding them to the prefix list.

This misconfiguration caused an outage until the error was corrected.

DAL-5-01 is now back online, we will provide an official report later.

Official Report from Cosmic:

Hi everyone, During an emergency update to our DDoS mitigation systems, an error caused the automated rollback functionality to fail. As a result, the systems entered a continuous boot loop and, due to limited connectivity, were unable to restore the latest version automatically. What would typically be a five-second operation took over 15 minutes as we manually restored the systems.

Although we normally do not perform updates on weekends, a critical bug require...

CosmicGuard Outage & Failover Explanation

According to CosmicGuard’s official outage report, their London PoP experienced downtime due to an unexpected ASIC restart on their Arista core router. They are conducting a root cause analysis, suspecting it may be linked to new capacity planned for next week.

When our DDoS provider goes offline, we always have a fallback ISP in place, ensuring only a brief service disruption—typically no more than 5 seconds. Automated monitoring is in place to d...

We have implemented a patch to correct this.

The node is back online, and all VPS servers are operational. We will now begin migrating the VPS servers from this node to a different node.

We’ve just completed adding additional members to the LAG (Link Aggregation Group) as part of our efforts to address the issue impacting our blips.

While this adjustment is based on what we suspect could be causing the problem, we’re treating it as a potential solution given the information available.

We’ll continue monitoring the situation closely and will update you as soon as we have more details.

Thank you for your patience as we work to resolve this.

We replaced and swap the motherboard for a new one.

CosmicGuard Official Update:

We are addressing a mitigation issue. A temporary fix has been applied, and we are currently testing a comprehensive solution in our lab environment, which will be deployed later tonight.

All servers are now coming back online.

We’re working with our backup, there’ll be a small “lag” once we switch over to our full routes.

For now, we’ll mark as resolved but will provide a better detail report later.