We are in the process of bringing VPS servers back online. While we were able to recover a significant portion of them, not all could be restored.

If your VPS is in the status Stopped or Paused from your VPS dashboard, then it was affected and could not be recovered, you have a few options:

1. You can visit your VPS dashboard and click on "Rebuild" to start with a fresh Windows installation.
2. If you had scheduled backups, you can use the "Backups" feature to restore your VPS.
3. Alternatively, you can restore your VPS using your own backups. As part of our compensation, we will be offering everyone a 50% discount on their next invoice.

We understand this isn't the outcome everyone hoped for, and we sincerely apologize. We’ve always been transparent about data responsibility—it's crucial to ensure your data is backed up at all times. This is the first hardware drive failure we've encountered in three years since launching our company, and we are committed to improving our disaster recovery plan to prevent future incidents.

We are still actively working on this and proceeding with the backup process. Thank you for your patience!

Since our last update, we've been actively working on restoring the affected VPS.

We’ve identified that one of the disks has failed, which is preventing the impacted VPS from rebooting.

Why did the disk fail?
We routinely check our disks every quarter (every three months) and the last check for this particular disk was actually completed on July 7th, 2024. At that time, it showed a health score of 97%, indicating it was in excellent condition. Unfortunately, disk failures can still happen unexpectedly, and in this case, we're unsure of the exact reason. For those curious, this node is equipped with WD 3.84TB Ultrastar WUS3BA138C7P3E3 SSD NVMe disks, which are designed for datacenter environments and high-load applications.

What does this mean for my data?
Fortunately, we maintain regular backups as part of our disaster recovery plan. We are in the process of restoring the data from the last available backup, which was made three days ago—two days before the issue occurred. While this means there may be a slight rollback, it's better than losing all the data. Please note that while restoring data, there is a chance of some corruption, but we are doing our best to restore the 27 affected VPS as quickly as possible.

Alternatively, if you have created VPS backups through our dashboard, you can restore them from there. However, please do not attempt to restore them at this time as it will fail.

We will notify you when it is safe to proceed with the restoration.

What if I have my own backup?
As always, we recommend that clients maintain their own backups. If you have your own backup, please open a ticket and let us know. We will provision a new VPS for you right away to get you back online.

How long until data is restored?
We expect the data restoration process to be completed by tomorrow afternoon, as we need to download and copy over 3TB of data.

We sincerely apologize for this issue, which is unfortunately beyond our control, and we will be offering compensation to all affected users. If you have any special requests or need assistance during this time, please don’t hesitate to reach out. We’re here to help in any way we can.

Our team has arrived at the datacenter and is actively working on resolving the issue. We’ll keep you updated with any progress.

The remote hands at the datacenter have been unhelpful and have raised concerns about their handling of our hardware. We have requested that they stop working on the server, as they are not following our instructions and are acting independently.

We will wait for our staff to arrive in the morning, and we plan to have the server back online early tomorrow.

Although it’s 4 AM for us, we are still working on this with the help of remote hands. Thank you for your patience.

We apologize for the extended delay on this, we are working with remote hands still.

Apologies for the delay in resolving this issue.

We are relying on remote hands at the data center, and they are not as fast during weekends.

We expect to have this resolved within the next two hours.

We will provide updates if we receive any additional information.

Please note that this issue is not yet resolved.

After rebooting the node, none of the drives are being detected by the motherboard, and our access through internal tools isn't functioning as expected.

We are submitting a ticket to our remote hands team to assist in regaining access to the node and investigating why some servers aren’t booting.

Please note that response times from remote hands tend to be slower over the weekends.

Following yesterday's incident of the damage ethernet cable, we will need to issue a reboot to fix a drive that became unmounted and it is preventing a small amount of VPS from booting up.