E-commerce · Hardware incident
Rescuing an online store from a failing PCIe slot and dying NVMe drives
A failing PCIe slot and two dying NVMe drives under a live online store. We got the data off with only 35 bad sectors lost, rebuilt the RAID and planned the move to a current platform.
The situation
An online store’s server started failing at the hardware level. A PCIe slot was failing, and two NVMe drives were dying. The store’s data sat on exactly the hardware that was giving out.
In a situation like this, every restart and every heavy read is a risk. The aim is to get the data off before the drives stop answering, without making them worse in the process.
What we did
Recover the data first. We copied the data off the failing drives while they still responded. Only 35 bad sectors were lost. Everything else was recovered.
Rebuild the storage. With the data safe, the RAID was rebuilt.
Plan the way forward. A rescue fixes today’s problem. The store also needed a platform that would not lead to the same situation again. We planned a move to Magento 2.4.8 with OpenSearch for search, Redis for caching and object storage for media.
Outcome
The store kept its data: of everything on the failing drives, only 35 sectors were lost. The RAID was rebuilt, and the owner got a concrete plan for a current platform instead of another patch on an old one.
If your server is showing disk errors, do not wait for the next reboot to find out how bad it is. Start the subject line of your email with URGENT and we help the same day when we have capacity.