Previous incidents

As part of the final transition to our new Ashburn data center and the decommissioning of our legacy facility, we migrated CosmicGuard to a new router. However, we quickly discovered that CG did not support NSR (Nonstop Routing). Once identified, we promptly restored network connectivity.

CosmicGuard had a network device issue while deploying a DDoS patch. It was quickly resolved.

Sorry about that.

Server is back online. The connection to it had flopped.

Ashburn, Virginia - CosmicGuard DDoS recovered.

This incident auto recovered, we will post a post incident report regarding this.

Post Mortem Report:

We asked CosmicGuard to add a new /24 IP subnet to our Ashburn location earlier today.

CosmicGuard stopped announcing our prefixes in Ashburn because of a configuration error when adding them to the prefix list.

This misconfiguration caused an outage until the error was corrected.

DAL-5-01 is now back online, we will provide an official report later.

Official Report from Cosmic:

Hi everyone, During an emergency update to our DDoS mitigation systems, an error caused the automated rollback functionality to fail. As a result, the systems entered a continuous boot loop and, due to limited connectivity, were unable to restore the latest version automatically. What would typically be a five-second operation took over 15 minutes as we manually restored the systems.

Although we normally do not perform updates on weekends, a critical bug require...

CosmicGuard Outage & Failover Explanation

According to CosmicGuard’s official outage report, their London PoP experienced downtime due to an unexpected ASIC restart on their Arista core router. They are conducting a root cause analysis, suspecting it may be linked to new capacity planned for next week.

When our DDoS provider goes offline, we always have a fallback ISP in place, ensuring only a brief service disruption—typically no more than 5 seconds. Automated monitoring is in place to d...

We have implemented a patch to correct this.

The node is back online, and all VPS servers are operational. We will now begin migrating the VPS servers from this node to a different node.

We’ve just completed adding additional members to the LAG (Link Aggregation Group) as part of our efforts to address the issue impacting our blips.

While this adjustment is based on what we suspect could be causing the problem, we’re treating it as a potential solution given the information available.

We’ll continue monitoring the situation closely and will update you as soon as we have more details.

Thank you for your patience as we work to resolve this.

We replaced and swap the motherboard for a new one.

CosmicGuard Official Update:

We are addressing a mitigation issue. A temporary fix has been applied, and we are currently testing a comprehensive solution in our lab environment, which will be deployed later tonight.

All servers are now coming back online.

We’re working with our backup, there’ll be a small “lag” once we switch over to our full routes.

For now, we’ll mark as resolved but will provide a better detail report later.

Official CosmicGuard report regarding the degraded performance:

This morning, a bug in the filtering code caused lag spikes and, in some instances, outages in specific locations. The issue was promptly identified and addressed, and throughout the day, we implemented several fixes to prevent it from happening again. These updates will not only resolve the problem but also enhance filtering performance on our end.

CosmicGuard has posted an update located here, we'll update that.

https://status.as30456.net/cm2p2qiwq0076miqcgbpcscuc

The server is now back online, and we apologize for the extended wait time. The motherboard on the previous server was faulty, and we have replaced it with a brand-new one. Compensation will be credited to your accounts shortly.

We’ve identified that the recent issue occurred because the default route was utilizing the session that experienced disruptions, even though the specific routes themselves weren’t affected.

To address this, we’ve added a backup default route within the routing instance. This change ensures that if the primary default route to Cosmic becomes unavailable for any reason, the system will switch to our main ISP for outbound traffic. While this failover may not be completely seamless, it will pre...

CosmicGuard rerouted around the NTT outage to minimize impact for our customers.

CosmicGuard is currently in the process of transitioning away from their Dallas data center. Unfortunately, a data center technician prematurely unplugged a patch panel that was not scheduled for disconnection until later tonight, resulting in an outage.

We sincerely apologize for any inconvenience this may have caused.

The Ethernet cable has been replaced, and the server is back online. However, the remote hands connected the cable to the wrong port. We’ll switch it back tomorrow after peak hours.

Please expect maintenance on this node tomorrow around 1 PM EST when our on-site team arrives.

Server is back online, there is a faulty cable that needs to be replaced.

We will replace it on Monday with our on site tech. For now, we will leave it alone but we will keep an eye on it and on stand by in case we need to replace it before then.

• We scheduled internal maintenance from 5:00 AM to 6:00 AM CT to deploy a core mitigation update to our global network.
• As part of our standard process, we apply configuration changes and core updates during off-peak hours on weekdays to minimize risk, and we typically do not expect any customer impact.

Incident Timeline

• 5:00 AM CT: The mitigation update was deployed to AMS as part of a staged rollout. As a result, AMS began sending synchronization pulses to other edge PoPs. T...

Cable is now fixed and server is back online.