Home / Blog / How a RAID 5 Set Is Lost
Business · blog post

One four-bay box, eighteen months, and the weekend it stopped being a system

Nobody rings on the day a disk dies. They ring months afterwards, on the Monday following the repair, about a box that had been serving files perfectly well since a fault back in the spring and now refuses to assemble at all. Everything worth deciding sat in the gap between those two events, which is a good reason to walk one of them through from beginning to end.

March: an amber light with nobody's name on it

Picture a four-bay unit on a shelf in a cupboard at an engineering firm on the Rugby side of the county, sharing the space with the router and a box of toner. In March it decides one of its disks has gone and says so twice: an email to the address typed in when the box was commissioned, and a light on the front that changes colour. The email goes to somebody who left two years ago. The light is behind the toner.

Nothing else happens, and that is the awkward part. Drawings open, the accounts package saves, everyone works. The unit is doing precisely what it was sold to do — carrying on as though nothing had happened — and it will keep doing that indefinitely. The only sign that anything is different is a colour nobody sees.

So the set runs for six months with its protection already spent. That is not carelessness. In a firm of thirty people the storage is nobody's actual job, and a device whose entire selling point is that it hides a failure will hide one from you as thoroughly as it hides one from the software. The arithmetic underneath is what makes the next part inevitable, so it is worth two minutes.

What the redundancy actually bought, in one sum

Take three disks and ignore everything else about them. Two hold blocks of your data, say a 4 and a 7. The third holds the total, 11. Lose the disk carrying the 7 and it can be worked out from what is left: 11 minus 4. That is parity, in full. A production array uses a different sum and scatters the totals over every member rather than parking them all on one disk, but the principle does not change.

Two things drop straight out of that, and neither is printed on the box. One equation solves for one unknown, so a set tolerates exactly one missing member — lose two and there is nothing to solve, in any software, on any budget. And to work out what an absent disk held, every surviving member has to be read from one end to the other. Not the busy folders. Not the part in use. The entire surface of every remaining disk, with no sector along the way refusing to answer.

That second condition is the one that kills sets. A degraded array is not a car running on a space saver, because a space saver still gets you home. It is a car with nothing in the boot at all, being asked to complete a long run at speed this afternoon, on tyres bought in the same order, fitted on the same day and worn by the same journeys ever since.

September, Friday, six o'clock: the repair starts

Somebody finally notices the light, buys a matching disk and slots it in on the way out on Friday. The unit starts putting itself right. At around two in the morning it stops at seventy per cent and reports a second member dead. By Monday the volume will not mount, and a firm that had a fully working system with one hidden fault has no system and two faults, having changed nothing except pressing the button labelled rebuild.

Here is why that happens then rather than earlier. Disks collect sectors that will not read, quietly, and a sector nobody has asked for in two years announces nothing at all. An office array might touch a third of its capacity in a busy month, so decay across the other two thirds is completely invisible and the box reports itself well, which by its own instruments it is. A rebuild inverts that overnight: suddenly every block on every surviving member must be read, including regions untouched since the day the unit was commissioned. Whatever went wrong out there surfaces at that moment, and a controller that reads a timeout as a dead disk ejects the member on the spot. The genuinely damaged part of it might amount to a few kilobytes.

The reason this got worse rather than better over fifteen years is volume. Putting right one member of a four-bay set of 8TB disks means reading somewhere near 24TB without a single fault. Capacity grew by orders of magnitude in that period; the published rates for sectors that will not read did not improve at anything like the same pace. What was a comfortable bet on 500GB disks is a much less comfortable one now, which is why storage engineers moved to arrangements carrying two calculated blocks per stripe, and why a great many boxes in cupboards never heard about it.

Ejected and finished are not the same word

This is the paragraph that rescues more situations than anything else here. When a controller announces two failed members, it is common for only one of them to be genuinely done for. The other was pushed out — for answering slowly during a busy hour, for a small handful of bad sectors, for a backplane connector that was never quite seated — and is sitting in the tray in serviceable condition, holding everything it ever held.

A member that was merely ejected can usually be imaged in full and brought back into the reconstruction, which turns an unsolvable sum into a solvable one. The move that destroys that chance is the obvious one: telling the box to bring the set back online regardless. Forced assembly writes fresh metadata across the disks, and among the things it can bury are the sequence numbers recording which member is carrying out-of-date content, which are frequently the only trustworthy account of what happened and in which order.

That stale member deserves more respect than it usually gets. A disk that dropped out three weeks before the rest is holding a three-week-old version of every block on it. Slot it back in blindly and the volume may well assemble, mount and look entirely convincing while quietly serving a mixture of old and current data. That is a worse outcome than a volume that refuses to mount, because a set that will not mount announces itself on Monday morning and a set full of stale blocks waits until the year-end figures disagree with the bank.

If you are reading this on the evening it happened

If the volume still mounts, copy it. That is the entire first instruction and it outranks everything else on this page. A degraded set is still handing over every byte you own, so get the working data onto something else today — an external disk bought in a lunch hour beats every clever decision available later. Once that copy exists, trying a rebuild is a perfectly sensible thing to do. Without it, a rebuild is a wager placed with the only copy in existence.

If the volume has already gone, everything useful is something you stop doing. Stop writing to it and shut down the applications pointed at it, because a database still trying to commit to a broken volume makes the reconstruction harder with every attempt. Do not begin a rebuild. Turn down every offer to initialise, create, clear or repair. Be especially wary of the file system checkers, which are built to make a volume consistent rather than complete — what CHKDSK does to a damaged volume is a short read and worth having before anyone points one at a set of members.

Then write down the things nobody will remember in three weeks. Photograph the front of the unit with the disks still in it. Label each disk with its bay as it comes out and note whether you counted left to right or top to bottom. Record which member was reported failed and roughly when, whether a replacement went in, how far the repair got before it stopped, and how many times the box has been switched off and on since. Every one of those facts steers the reconstruction, and reassembling them from memory a fortnight later is unreliable in a way that costs real money.

While you are in the cupboard, check what the box actually is

Two discoveries wait in a lot of small offices, and both are cheaper to make this week than during an incident. The first is that a great many two-bay units were configured as a stripe rather than a mirror, because a stripe shows the full combined capacity and looks more generous on the shelf. A stripe holds no redundancy whatsoever — either disk failing takes the whole volume immediately. This county sends more of that work than most, which is why RAID 0 data recovery has a page of its own here. Five minutes in the storage manager settles which mode yours is in.

The second is what parity was never watching. It defends against one event, a disk dying, and against nothing else at all. A deleted folder is deleted faithfully across every member at once. Ransomware encrypts the files and the array then protects the encrypted versions with exactly the diligence it would have shown the originals. Fire, theft, a burst pipe over a comms cupboard on one of the trading estates off the M69, a controller writing corruption, a power cut halfway through a write — every one of those sits outside the problem RAID was invented to solve.

For what it is worth in a small office, a mirror behaves far better in recovery than parity does, because each half is a complete readable copy rather than one term in a set of simultaneous equations. Where the budget stretches that is the trade worth making. RAID 1 recovery covers mirrored pairs, RAID recovery sets the levels beside each other, and NAS recovery deals with Synology, QNAP, Netgear and Buffalo units specifically.

What happens at the bench, and what the bill looks like

Nothing is rebuilt on your equipment and nothing is written to your disks. Each member is copied on its own first, on hardware that sets its own timeouts and works around a difficult region rather than hammering at it until the drive gives up entirely. That stage is where the hours go, and it is the reason arrays are priced by the number of members: a four-bay box means four separate imaging jobs before any array work begins, and one sick member can occupy a day and a half by itself.

The parameters are then derived from the data rather than taken from the label — stripe size, block order, the direction parity rotates, which member is stale and by how long — and the volume is assembled virtually from the images so the file system can be checked before anybody claims a result. The endings that are genuinely bad are worth naming: two members that are mechanically finished with no overlap between the damaged areas, and a set that has already been re-created with different settings or rebuilt onto, where the outcome depends entirely on how far that got.

Arrays, NAS boxes, SANs and servers open at £500 + VAT and rise with the member count. One drive or SSD is £300 + VAT, a card or USB stick £250 + VAT, a recorder disk or an encrypted volume £400 + VAT. The diagnostic costs nothing, closes 2 working days after the disks are booked in, and produces one written figure. Logical faults run under no fix, no fee; the four published exclusions are electronic and mechanical failures, chip level work, DVR jobs and forensic jobs, and those take half the figure in advance because donor parts get bought for one named drive.

A NAS travels whole with its disks in their bays. A rack server or a bare parity set sends the member disks only — no chassis, no controller — each marked with its bay number and the photograph of the front panel tucked in beside them, packed so nothing can shift. Post it tracked and insured to Oxford Data Recovery, John Eccles House, Oxford Science Park, Robert Robinson Avenue, Littlemore, Oxford OX4 4GP. From Coventry it is about fifty-five miles down the M40 and roughly an hour if you would rather drive it; reception signs devices in Mon–Fri 9:00am–5:30pm. There is no collection round and no Coventry counter. The pages next to this one are RAID 5 data recovery, server data recovery, VMware recovery where the volume held datastores, and data recovery cost for the bands in full. The freephone is 0800 689 0668.

The dangerous hour is the repair, not the failure. A set running without cover has nothing left in reserve, and a rebuild asks every remaining disk for a flawless read from one end to the other in a single evening. Get a copy off first, or have the members imaged before a replacement disk goes anywhere near an empty bay.

// what people ask about this one

Common questions

You do not have to stop, but you should treat today as the last easy day. The danger is not that the unit collapses this afternoon; it is that the next fault of any kind now takes the volume with it. Get the important data copied somewhere else before making any decision about replacement disks. That copy is worth more than every other choice available to you put together.
You can, and thousands of firms do, but understand what a week of ordinary use means. Every working day is another day of reads and writes on disks that are the same age and have had the same life as the one that already failed. Nothing is protecting you in the meantime. If a week is genuinely what it takes to organise, spend the first hour of it making a copy rather than the last.
Not before somebody has looked at why it went offline. Forcing an assembly writes new metadata to the disks and can overwrite the sequence numbers that record which member holds out-of-date content, which is often the only reliable evidence of what happened. If it works, it works. If it does not, it has made a solvable job harder. Ask what the plan is if the forced assembly mounts something stale.
It shortens the window in which you are unprotected, which is genuinely worth having, but it does not change what a rebuild asks of the other disks. The spare starts the same heavy end-to-end read of every surviving member, at the same moment those members have nothing in reserve. A hot spare improves your odds of never noticing a single failure. It does not make the repair a safe operation.
Not automatically, because ejected and dead are different states. Controllers throw members out for slowness, for a few unreadable sectors and for connectors that were never quite seated, and a disk ejected for one of those is often in good order. It can usually be imaged and brought back into the reconstruction. Two disks that have genuinely failed mechanically, with no overlap between the damaged regions, is a far harder proposition and you will be told so at the diagnostic.
It lengthens the job without ending it. Bay order is one of the parameters worked out from the contents of the disks, but that is bench time a two-second photograph would have made unnecessary. Where the disks are still seated, take the picture now and write on them as they come out. Where they are already loose in a bag, say so, because an honest gap is much easier to work with than a confident guess that turns out to be wrong.
Multi-disk work of any kind opens at £500 + VAT and grows with the member count, settled in writing when the free diagnostic closes, 2 working days from booking in. The hours go into imaging each member on its own before a reconstruction is attempted. No fix, no fee covers logical faults. Four categories sit outside it — electronic and mechanical failures, chip level work, DVR jobs and forensic jobs — and those take half the figure first.

Rather put it in front of an engineer?

Oxford charges nothing to examine it, and that closes 2 working days from the moment it is booked in. Whatever figure comes out of it is written down and stays there: £250 + VAT a card or stick, £300 + VAT one drive or SSD.