Home / VMware Recovery

VMware Recovery VMDK, VMFS and snapshot chains

A guest is a file, that file lives on a datastore, and the datastore lives on an array, so one failure can present itself as three disasters at once. This page covers volumes that will not mount, chains a backup job walked away from, thin disks and the space inside them nobody ever wrote to, guests deleted in a hurry, and the case nobody enjoys: a virtual machine on a RAID that has given way underneath it.

Guests live on arrays, so the band is from £500 + VAT, climbing with the number of member disks, or £300 + VAT where the host ran on a single drive. Looking costs nothing and the answer arrives on the second working day after the drives reach the bench.

// where a guest actually goes

Six ways a machine stops existing

Virtualisation relocates a problem rather than removing one. A guest is a file, the file sits on a datastore, the datastore sits on an array, and when something breaks the screen rarely says which of the three it was.

The datastore will not come up

Host boots, storage is plainly visible to it, and the VMFS volume is either missing from the list or offered as unformatted. Damage has landed on the partition table, the volume header or the metadata region. What sits underneath is normally in perfect order, which is why letting anything reformat or repair the volume is the single worst move available.

Snapshots that nobody ever consolidated

Someone took a snapshot, a backup job left another behind, and the machine has been running on delta files since the spring without a word about it. Break that chain and the guest reverts to an afternoon months ago, or declines to start at all. Every link counts, and the descriptors joining them can be written again when they go missing.

The datastore filled and everything froze

Thin disks and forgotten snapshots grow quietly until there is no room left, and the guests stop dead in the middle of writing. File systems and databases inside them are then out of step with themselves. Making space is only the opening move, because those guests will need work on the inside afterwards.

A machine deleted, or merely unregistered

Taking a guest out of inventory leaves its files exactly where they were. Deleting from disk unlinks them, and on VMFS those blocks become available to everything else still running on that volume. How fast the host was shut down governs how much of it returns, and the difference between an hour and a week is enormous.

The array underneath let go

Two faults on top of one another. The RAID has to be reassembled with the right geometry before the datastore can be read at all, and only after that can anything inside the guest be repaired. A slightly wrong stripe order gives a virtual disk that opens cleanly and holds gibberish, which is why the layout is proved against content.

The guest boots and its own volume is broken

Virtual disk sound, contents not. A Windows guest with a mangled NTFS index, a Linux guest whose journal will not replay, a database that refuses to attach. That recovery takes place inside the virtual disk, on a copy, in exactly the manner it would on a physical machine — the virtualisation makes no difference to it.

// the file formats, in plain terms

A virtual disk is a file with a file system living in it

A VMware disk normally arrives as a pair of files. One is a short text descriptor holding the geometry, the disk type and, where snapshots exist, the parentage of each file. The other is a large flat file holding contents, and inside that flat file is an ordinary file system — NTFS, ext4, XFS — laid out precisely as it would be on real hardware. That last point is the one that matters: as soon as the flat file is readable, every technique used on a Windows or Linux volume applies to it without alteration.

Descriptors are tiny and easily mislaid, and mislaying one is far less serious than it appears, because the geometry can be derived from the flat file and the descriptor written again. Losing a link out of a snapshot chain is the harder case, since a delta holds only what changed after its snapshot was taken. The chain must be put back in sequence before any of it signifies anything, and a chain assembled in the wrong sequence will happily deliver a machine that boots and contains a blend of two different months.

Thin disks, thick disks and space that never existed

A thick disk claims its whole size on the day it is made. A thin disk expands as it is written, in grains tracked by a directory of its own, and damage to that directory loses the correspondence between where a block sits in the file and where the guest believes it sits — which is usually rebuildable from the file system inside. The consequence that catches people out is a different one. On a thin disk, space that was never written holds nothing at all. Two terabytes provisioned against three hundred gigabytes used is 1.7TB of absence, and no laboratory manufactures files out of ground that was never occupied.

VMFS repays being careful with it

VMFS is a cluster file system built for enormous files and several hosts writing at once. Its metadata occupies a defined region, its blocks are large, and it keeps enough duplication that a damaged volume header is often survivable on its own terms. Being written over is what it does not survive. A host told to reformat the datastore, or a version-6 volume laid across a version-5 one during an upgrade that went sideways, puts new metadata straight through the area recording where each virtual disk starts. Part of that can be rebuilt by locating the virtual disks in raw blocks. Part of it cannot be rebuilt by anybody.

So when a datastore stops mounting, the right instruction is to stop. Do not accept a repair, do not build a fresh datastore over the same disks, and do not run a consolidation or an expansion on the theory that movement might help. Put the host into maintenance, shut it down, and let the disks underneath be imaged.

Hyper-V is the same trouble in different clothing

Microsoft uses VHD and VHDX files, with differencing chains and checkpoints of their own, sitting on NTFS or a cluster shared volume. The vocabulary changes and the failures do not: a chain short of a link, a dynamically expanding disk whose allocation table is damaged, a checkpoint merged badly, a guest whose own file system has come apart. Those run identically here and land in the same band.

// odds, deliverables and the figure

What comes back, set out honestly

Routine: a datastore with mangled metadata and sound blocks beneath it; a virtual disk that has lost its descriptor; a chain that can be returned to sequence; a guest whose own file system is corrupt; a machine deleted from a volume that was powered down shortly afterwards; virtual disks on an array that can be rebuilt.

Partial, and quoted as partial rather than dressed up: a deleted machine on a busy datastore that kept running for days, because whatever else is writing will help itself to freed blocks; a chain short of a delta, which yields the machine as it stood at the last surviving link rather than at the crash; a guest caught mid-transaction when the volume filled, where the files return and the database inside still wants repairing on its own account.

Beyond reach, with nothing to gain by pretending otherwise: a datastore fully reformatted and used since; regions of a thin disk that were never written; blocks taken by other guests that carried on working after somebody deleted one; and a snapshot discarded deliberately whose changes were never merged anywhere. Where that is the position you hear it at the examination stage, before any money changes hands.

The order the work happens in

Disks underneath are imaged one by one, as for any array job. The set is rebuilt on those copies and the parameters confirmed by checking that VMFS structures appear exactly where the chosen geometry predicts. With the datastore readable, virtual disks are extracted, chains restored to sequence, and each guest opened as a file system in its own right. Whatever is broken inside a guest is repaired at that stage, once more against copies and never against your drives.

Two shapes of deliverable exist. Either the virtual disk files themselves, ready to register on rebuilt hardware, or the contents of the guest as plain folders and files. Decide early, because it changes how the final stage runs, and asking for both is entirely reasonable.

What to send, and what it comes to

Datastore disks, bare, each marked with its bay, with the front panel photographed beforehand. Not the host, not the card, not the enclosure. Add a note giving the hypervisor version, the array level if you have it, how many guests were on the volume and which one matters most — on a job this size, priorities genuinely reorder the work. Post it tracked and covered, or drive it to Oxford and hand it in on a weekday between 9:00am and 5:30pm; the run is about fifty-five miles down the M40.

Guests almost always sit on arrays, so the array band applies: from £500 + VAT, climbing with the number of members. Where a host ran a single disk it is £300 + VAT. Looking is free and the answer arrives on the second working day after the drives are logged in, with the figure fixed in writing at that point. Logical work carries no fix, no fee; chip-level work, DVR jobs, forensic jobs and mechanical or electronic failure sit outside it, and invasive work takes half upfront. Virtualisation reaches this bench from the technology campuses at Ansty and Whitley, from providers looking after a dozen small Warwickshire clients on one host, and from university spin-outs between Coventry and Warwick who put the whole company on a single well-specified box.

// related pages

Nearby on this site

// questions about virtual machine recovery

Common questions

No. Put the host into maintenance and shut it down. Repair options rewrite the very metadata that records where each virtual disk begins, and that metadata is usually the only casualty. The disks below it are generally untouched, and they stay untouched for exactly as long as nothing else writes to the volume.
Often, and how much of it depends on speed alone. Unregistering leaves the files alone; deleting from disk frees the blocks, and the other guests on that volume will use them. Power the host down as soon as you realise. Within the hour usually means everything; a week of continued running usually means fragments.
It depends which part. A delta holds only the changes made after its own snapshot, so a chain restored to the correct sequence gives the machine as it stood when everything stopped. A missing delta gives the machine as it stood at the last surviving link, which may be weeks earlier. Descriptors can be rewritten; absent blocks cannot.
Not at all, though it is two jobs in sequence. The set is rebuilt first, with the geometry tested against the contents rather than accepted from a card, and only then does the datastore become legible. A wrong stripe order produces a virtual disk full of rubbish, so the parameters are confirmed before anything is extracted.
The array band, so from £500 + VAT climbing with the member count, or £300 + VAT where the host ran on one disk. Looking costs nothing and closes two working days after the drives are booked in, and the quotation is fixed in writing before any work is authorised.

Do not let the host repair the datastore.

Virtual disks are usually perfect beneath a damaged VMFS header. A repair pass writes over the one record that still knows where they begin.

// getting your media to Oxford

Posting a device in — what goes in the box

Multi-disk jobs travel as a set of bare drives sharing one padded box, and that parcel wants tracking and covering for what is written on it rather than what it weighs. Booked over a Coventry post office counter in the afternoon, it usually reaches the Oxford bench the next working morning — faster than most firms manage to spare somebody for the run down the M40.

As a rule the storage comes out and the machine stays where it is. That applies to a laptop, a tower, an iMac and to the recorder sitting under a counter. Taking equipment apart is not something this bench does, and a repair shop will free a drive in a few minutes. Two things go the other way: an external drive stays sealed inside its own case, and a NAS travels as a complete unit with its disks still in their bays. A Fusion Mac is a third case — both of its drives come out and travel together, each one labelled. The single situation nobody can work around is memory soldered flat onto a mainboard, which is how Apple Silicon Macs and a good many slim laptops are built: if the storage will not unbolt, there is no parcel to send.

  • Use a box or padded mailer with some rigidity to it, and pack around the drive until nothing shifts when the parcel is tilted. Mains adaptors, docks and leads are not wanted at this end.
  • Sending a RAID or a server? Only the member disks travel — not the chassis, not the controller — and each one wants its bay number written on it. Take a photograph of the front of the unit before anything is pulled; it costs nothing and now and again it saves a day.
  • Print the shipping and booking-in form (PDF), add a name, a number you will answer and a sentence on how the trouble began, and drop it in beside the media.
  • Most people use Special Delivery, which is tracked and covered; a courier of your own does the same job. You can also bring it: the Oxford reception takes devices over the counter, Mon–Fri 9:00am–5:30pm. Neither a Coventry counter nor a collection round exists.
// write this on the label

Oxford Data Recovery

John Eccles House
Oxford Science Park
Robert Robinson Avenue
Littlemore, Oxford, OX4 4GP

↓ Print the shipping & booking-in form (PDF)

Put Oxford Data Recovery on the label. From Coventry it is roughly fifty-five miles straight down the M40, about an hour if you would rather drive it in than post it. Either way you are told the moment it is logged, and the free diagnostic finishes two working days later.

Unsure what ought to go in the box? Ring 0800 689 0668 before you seal it, or work through the free online diagnostic and let it do the asking.