Back to overview
Downtime

DAL-5-01 Frog Node | TX went down

May 16, 2024 at 06:10pm UTC

Resolved
Jun 04, 2024 at 06:35pm UTC

We have now completed the migration of all VPS back to the new node. If your VPS is still on the incorrect node (now DAL-5-01) and you were previously on Frog Node, please open a ticket. We may have missed your VPS.

We will also restart the uptime for the DAL-5-01 node. This is a brand new server, and we will begin monitoring this new node now.

Updated
Jun 03, 2024 at 10:19pm UTC

We will begin moving VPS servers back to this node starting at 1 AM on 6/4/2024 until 4 PM EST.

During this process, your VPS server may briefly disconnect and experience intermittent internet connectivity loss once during migration.

Updated
May 30, 2024 at 01:08am UTC

We have conducted a comprehensive test on this server, running it for 24 hours without any issues. We plan to continue monitoring the server until Monday to ensure its stability.

Starting on June 3, 2024, we will begin migrating users back to this node.

Our goal is to avoid any potential problems during the peak hours this upcoming weekend.

Updated
May 28, 2024 at 08:04pm UTC

The new node has been delivered, and we are currently testing and racking it.

Updated
May 25, 2024 at 01:53am UTC

Hello,

We have received notification of a brand new unit being shipped to our Dallas location.

The expected arrival is by the end of the day on Tuesday, May 28, 2024.

We will coordinate the transfer of all VPS back to this Node once we confirm that the new unit is functioning properly.

Updated
May 20, 2024 at 08:41pm UTC

We have dropped off the unit at FedEx today, and it is expected to arrive at our vendor for repairs tomorrow.

Updated
May 20, 2024 at 04:31am UTC

We expect to receive the RMA approval on Monday and will ship the item to our vendor the same day using Next Day Air for a quick delivery. They have confirmed that all parts are in stock, so we anticipate getting it back by Friday, 5/24/2024.

Updated
May 17, 2024 at 10:00pm UTC

We have now migrated all remaining VPS on this node to a stable one. Your IP might have changed, take note of the new one.

We will be shipping this unit back to our vendor for repairs.

We will provide updates on the ETA for the unit's return as we receive more information.

For now, we are marking this issue as resolved since there is no longer any downtime for our customers, but we will continue to update you here.

Updated
May 17, 2024 at 09:40pm UTC

We have now migrated all customers who had open tickets with us. We will now proceed with migrating the remaining VPS servers on this node.

Updated
May 17, 2024 at 09:08pm UTC

Unfortunately, the server's motherboard is malfunctioning and causing random boot issues.

We haven't found a server that supports these hot-swappable NVMe disks, so we are shipping one of our AMD EPYC systems from Ashburn to Dallas via Next Day Air. It is expected to arrive tomorrow, Saturday. If it does, we will have the server online; otherwise, it will be up on Monday.

Important:
This node occasionally boots up and allows server access. During these periods, we have managed to migrate a few servers based on your open tickets. If your VPS has been migrated, we will inform you accordingly.

Updated
May 17, 2024 at 05:00pm UTC

We encountered an issue with the replacement server: it does not support hot-swappable NVMe disks.

We are seeking an alternative solution for this problem.

Updated
May 16, 2024 at 11:51pm UTC

This new node is expected to be online tomorrow around 9:00 A.M EST. Your VPS will be back online, and we can begin migrating your VPS to the new nodes until the replacement unit arrives.

We can also provide a new DDR5 VPS if you have a complete backup of your current VPS, which you should always have. It's your responsibility to ensure your data is backed up for situations like this so you can quickly move things around.

Compensation for the downtime will be provided once everything is back in order.

Any new updates will be provided here and nowhere else.

Updated
May 16, 2024 at 10:27pm UTC

Adding to the previous update:

Your current data is safe, uncorrupted, and fully backed up.

We also have dedicated servers available for immediate delivery.

Updated
May 16, 2024 at 10:15pm UTC

The entire unit is currently non-functional and requires replacement, which will take some time—at least until next week. We've already submitted an RMA to our vendor.

I’d like to emphasize that this unit is brand new with all new parts; we're unsure what went wrong.

Here’s what we plan to do next:

We will remove the disks from the current node and attach them to a different node we are currently assembling. This new setup will use Dual Xeon V4 CPUs with DDR4 RAM instead of DDR5 RAM and AMD EPYC CPUs. This approach aims to bring your servers back online as quickly as possible. We also have dedicated servers available for immediate delivery.

If you prefer to move your VPS to another DDR5 node, we have some space available in Dallas and Virginia. Please open a ticket if you would like your VPS moved to a new DDR5 node. It will be on a first-come, first-served basis due to limited space.

Once we receive the replacement unit, we will revert everything back to normal.

We will keep you posted with further updates soon.

Updated
May 16, 2024 at 07:40pm UTC

Server has been de-rack and now being tested.

Updated
May 16, 2024 at 07:34pm UTC

We are going to de-rack this unit and perform a full test on all components, RAM, CPU, NVMes.

Updated
May 16, 2024 at 07:32pm UTC

Frog Node went down.

Updated
May 16, 2024 at 06:15pm UTC

The node is back online but we are going to keep an eye on it. We cannot find the root cause on why the server is restarting. We have already added logs to record and grab the status if it shuts down again.

We apologize for the past few days with this node and the issues.

Updated
May 16, 2024 at 06:12pm UTC

We are looking this issue and trying to resolve this once and for all.

Stand by while we diagnose and see what is happening right now.

Created
May 16, 2024 at 06:10pm UTC

Frog Node | TX went down.