We have tested the server, and it is functioning as expected.

We will begin the migration back to the original server on 9/12/2024 at 11:30 AM EST.

A maintenance notice will be posted for the migration.

This issue is now resolved.

We’ve received a brand new server, and we’re currently in the process of testing it. We will provide an update on the migration back to the original server once testing is complete.

Our vendor provided us with a new CPU and RAM but did not include a new motherboard. We have already requested a replacement motherboard, which is expected to arrive on 9/10/2024.

We have received an update from our vendor:

"We are preparing a NEW replacement CPU, motherboard and a replacement memory kit. These will be sent to site in Dallas via Overnight shipping. Will advise tracking details as soon as they are available. "

We will be getting that Wednesday at our datacenter to replace.

All VPS servers are now online. If your VPS is still offline, you can boot it from your VPS dashboard. If you encounter any issues, please open a ticket.

Next, we will be addressing compensation for our customers.

We will mark this as resolved since there's no current outage, however, we will continue to update once we receive information from our vendor regarding the RMA.

We are now bringing back VPS servers in batches, your VPS should be online within the next 20-30 minutes.

All disks are now attached! We’re now going to be merging some VPS away to a new Node due to not having enough RAM on this replacement server.

DO NOT attempt to turn on your VPS or you may corrupt your server.

It appears that the target server has 12 bays, but only 4 are compatible with U2 drives. We are about to shut down the server, relocate the drives, and then power it back on. After this, our drives will be online and we will start bringing back all VPS servers.

The wait it almost over, thank you for your patience and understanding.

We located the disks and are now attaching them to the server.

We’re trying to reach one of the datacenter staff that did the swap for us because the target/temp server only has one NVMe installed out of 5 disks needed. They were set aside for some reason. We apologize for any inconvenience this may have caused.

Our staff is currently on-site and addressing the situation.

Even after directing the datacenter staff to the correct server, they were unable to locate the second drive. Rest assured, we do have the disk. This issue will need to be reviewed by one of our data center team member. We're arranging for someone to be there at 8 AM tomorrow to get this resolved and back online.

One disk from the old server is still missing, but it's being retrieved and added by remote hands.

We apologize for the delay; the long weekend has made things challenging due to the understaffed datacenter.

Apologies for the delay. The new machine is now online and operational. However, we're currently addressing an issue with Linux detecting one of the drivers. We're actively working to resolve this.

We’re nearly finished configuring the drives and network. We’ll notify you here as soon as it’s safe to power on your VPS servers. Thank you for your continued patience.

Please DO NOT power on your VPS servers through your VPS dashboard.

This temporary server has 512GB of DDR5 RAM, compared to our usual 768GB DDR5, and we need to manually move a few servers off the node first.

We’ve successfully moved all the drives to the temporary server and are now configuring the internet and virtualization. Please bear with us as we complete this process.

We've secured a replacement server and are in the process of moving the drives over. Please bear with us as we complete this task.

We’re currently working on securing a spare server to transfer the disks to. Please stay tuned.

There are a few corrections to the previous update. Unfortunately, the BIOS update did not complete because the server keeps powering on and off repeatedly, indicating that the motherboard is officially dead. Despite replacing all 768GB of DDR5 RAM with our spare units to test, we've determined that the motherboard is indeed faulty.

We have already requested an overnight RMA for a server replacement and will keep you posted on whether we can have a new server by Monday, though there is concern about potential delays due to the upcoming Labor Day holiday.

What does this all of this mean?

Rest assured, your data is completely safe. There is no corruption or damage. Once the new server arrives, we will simply move the disks over and bring everything back online.

In the meantime, if you have a backup of your server (which we strongly ALWAYS recommend everyone maintain at all times), we can set you up with a new VPS in Dallas to get you back up and running immediately. If you would like this, please create a ticket on our Discord, and we’ll assist you in getting a server delivered. We will also provide compensation for the downtime once everything is resolved.

The server remains racked and powered on, and if it happens to boot back up, we will immediately begin migrating servers off of it. However, there is no guarantee this will happen, as the issue is unpredictable.

We understand that this situation is particularly unfortunate given that it’s Friday and the weekend, the busiest time for both us and your players. I sincerely apologize for the inconvenience, which is unfortunately beyond our control.

Please don’t hesitate to reach out if there’s anything we can do to assist you.

The BIOS update is complete, and we're bringing the unit back online. We will initiate an RMA with the vendor to receive a replacement unit. In the meantime, we'll closely monitor the server, and if any issues arise, we will migrate all VPS servers to a different node until the replacement unit arrives.

Thank you for your patience, and I apologize for any disruptions this may have caused for your players.

The BIOS update process is still ongoing, upgrading from version 2.03 to 2.04. We're hopeful that this update will resolve the issue with the server's random disconnects. If the problem persists, we'll need to transfer the VPS on these servers to our vendor for an RMA and request a replacement server.

A similar situation occurred with our ASH-5-01 server, which also experienced random crashes shortly after we installed a brand new ASRock motherboard. We'll keep you updated on any further developments.

We’re now working on doing BIOS update, last time this server had issues which was yesterday a reboot fixed it, this time it’s taking awhile to load into the OS.

This is similar to what happened with one of our previous ASRock motherboard and we had to send it back to our vendor replacement.

Stand by and we will provide more updates as we get them.

Thanks for your patience.

We’re aware of this issue and we’re looking at this.

We’re going to be doing a BIOS update since this server went offline yesterday causing some VPS servers unable to reboot.