VM-sickbay

Welcome - looks like something is wrong with your virtual machine.

Before we go into details ...
relax ...
drink a coffee and follow the VM-sickbay-rules

1. DON'T PANIC
2. do not try to start the VM again
3. do not mount disks from other VMs
4. do not use vmware-mount or vdiskmanager
5. do not use vdk in read/write mode
6. make a copy of all vmware-log files
7. relax


Are you right now running a VM that seems to be in an older state than expected ?
Shut it down now - do not run checkdisk and do not defragment.

In case you need to ask for help at VMTN or VMware-forum it is important that you provide sufficient information.
Without exact details the person trying to help you may miss-judge the cause of the desease
and so prescribe in-adequate treatment.
Anamnesis describe symptoms and conditions - post at VMTN or vmware-forum.de
Therapy fix or repair
Typical Symptoms
Post mortem recover data
Plan B for the desperate

DISCLAIMER:

The procedures I suggest for some typical problem often require advanced skills.
I can't take any responsibilty when you mess up your data following this tips.
This just lists up what I would do in such a case.
Please also note that official VMware support will very likely claim that those cases listed as "restore" are lost.
Don't give up too early - you may still be able to rescue some data.

For in depth discussion of the listed procedure use the search function at VMTN , german forum and sanbarrow forum.
Search for username "continuum" and terms from desaster description table.

Ulli Hankeln

encypted

Anamnesis

In most cases this data is sufficient for a first analysis:

- a detailed file-listing of all files in the VM-directory - this list should mention date, size and permissions
- all files smaller than 100 kb in the VM-directory
- screenshot of error message - if possible
- screenshot of datastorebrowser - if ESX
- screenshot of snapshotmanager

Please download the VM-sickbay-help-request.txt
and answer all further questions listed there.
Please simply edit the text-file.

Then pack all the files mentioned above along with your edited help-request
into a single zip-archive. Give that zip-file a reasonnable name -
I already have more than enough vmware.zips and log.zips ...

IMPORTANT:
if you decide to ask someone for help first power down the VM.
Then create the report-zip and send it.

Do NOT power on the VM before you have a good theory of what went wrong and a fix

Therapy

often used procedures ...
disktypes-table ...
the CID-chain ...

In case you want to try to fix the problem on your own - the next table lists some common-deseases and the therapy I would prescribe.

Typical symptoms

In the following I list some typical error-messages along with the suggested fix ...

Sometimes a crash of the host - no matter if Windows, Linux or ESX - results in a blank vmx.
Not always the error message is useful.
Anyway - if possible restore the vmx-file from the last vmware.log if that is available.
If you restore the vmx from scratch you may easily assign the wrong snapshot and so do more harm than good.

This may happen with sparse disks on hosted platforms.
In the vmware.log this may appear as "Grain #* @* is orphaned. "

Sometimes a crash of the host - no matter if Windows, Linux or ESX - results in a blank vmdk descriptor.

There are several possible reasons for this message:
- a vmdk was used with a second VM
- unwise manual edit of the vmx-file
- unwise operations with vmware-vdiskmanager or vmkfstools
- failed operations with snapshotmanager
- bugs, host crashes ....

In case a VM with snapshots does not start anymore because a basedisk was expanded you have two options:
- fix the snapshots
- fix the basedisk
The first option only makes sense with VMs that only have one snapshot.
In all other cases it is very probably easier to undo the expand of the basedisk.

Did you move your VM to Fat32 ?
Are you using NTFS with Linux or EXT2 with Windows ?
Obviously it is a bad idea to copyvmdk- files larger than 2 GB to Fat32.
If you think your filesystem should support the files you use
you may try this entry in the vmx-file.
diskLib.sparseMaxFileSizeCheck= "false"

Open the descriptor of the vmdk and comment offending lines.

Read vmware.log - find out expected screen resolution - change host resolution and try again.
If that does not work remove *.vmss to force a fresh start.

Disable cpu-compatibilty-test in the vmx-file by adding this lines for the next start.
checkpoint.overrideVersionCheck = "true"

checkpoint.disableCpuCheck = "true"

If that does not work remove *.vmss to force a fresh start.

top

Post mortem

In the following cases there is little hope to recover the vmdk as is.
So first thing to try is to repair the vmdk so that it can be mounted with vdk or a helper VM.
If that fails - and the data is important - try commercial tools - I have good results with UFS-explorer.
If that does not work there is always Plan B as the ultima ratio.

top

Plan B

Cases:
Rescue files from a virtual disk that is corrupted beyond repair - no matter what platform.
Rescue files from a standalone snapshot.
Rescue files from single chunks of split vmdks

Overview of the procedure:
create large new monolithicFlat vmdk
wipe disk with zeros using a helper VM or a LiveCD like Knoppix, UBCD4Win or MOA
format with filesystem used by the corrupted chunks
copy corrupted chunks into the disk
analyse with helper VM or LiveCD - using tools like UFS-explorer, GetDataBack ....
search for lost, deleted files

Problem:
Folks may tell you this can't work.
Ignore them. You are desperate - aren't you ?
I had good results with rescueing files from NTFS formatted corrupt vmdks.
Procedure is worth a try no matter what the original platform was.
I have used it with broken vmdks from ESX as well as from hosted platforms

Limitations:
This technic can only recover files that were newly created and saved during the timeframe the given chunk was active.
You can not restore files that were created in snapshot1 and last saved in snapshot2.

top

the CID-chain

VMware uses CID-values to verify if the snapshot chain is clean before starting a VM.
If a snapshot chain is corrupt the CID-chain is broken.

Snapshot chains may get corrupted when
- snapshots were deleted manually
- the host crashed
- the system run out of diskspace
- unwise manual edits of vmdk or vmx-files
- a virtual disk was attached to a different VM
- the basedisk was expanded
- ....
All those mentioned cases will be noticed by VMware at startup because of a break in the CID-chain..
As starting the VM in this conditions would further corrupt the virtual disks the VM will not be started.

Good CID-chain

In this simplified listing of one basedisk and its two snapshots in all
cases the child references the CID-value of its parent correctly.

###################### Windows Vista.txt ########################
CID=9a1f1a1f
parentCID=ffffffff
RW 104857600 SPARSE "Windows Vista.vmdk"
ddb.geometry.cylinders = "6527"

###################### Windows Vista-000004.txt ########################
CID=5cdd6af0
parentCID=9a1f1a1f
parentFileNameHint="Windows Vista.vmdk"
RW 104857600 SPARSE "Windows Vista-000004.vmdk"

###################### Windows Vista-000002.txt ########################
CID=c750afeb
parentCID=5cdd6af0
parentFileNameHint="Windows Vista-000004.vmdk"
RW 104857600 SPARSE "Windows Vista-000002.vmdk"

Broken CID-chain

In this simplified listing of one basedisk and its two snapshots
the parentCID in the first snapshot does NOT reference the correct CID-value of its parent.
This means that during the last use of the "windows Vista.vmdk" it probably was used
by another VM, or it was expanded or ...

###################### Windows Vista.txt ########################
CID=a123b123
parentCID=ffffffff
RW 104857600 SPARSE "Windows Vista.vmdk"

###################### Windows Vista-000004.txt ########################
CID=5cdd6af0
parentCID=9a1f1a1f
parentFileNameHint="Windows Vista.vmdk"
RW 104857600 SPARSE "Windows Vista-000004.vmdk"

###################### Windows Vista-000002.txt ########################
CID=c750afeb
parentCID=5cdd6af0
parentFileNameHint="Windows Vista-000004.vmdk"
RW 104857600 SPARSE "Windows Vista-000002.vmdk"

Special CID-values

Therapy : fake basedisk

Cases:
Rescue files from a standalone snapshot.
Rescue files from a snapshot chain

Overview of the procedure

find out what type of vmdk you need
find out what nominal size you need
find out which name the fake basedisk must use
create new disk with vmware-vdiskmanager.
find out which filesystem you need
format vmdk using a helper VM or a LiveCD
copy the vmdk into right path
find out which CID value is needed
attach snapshot
mount snapshot with helper VM or LiveCD
use recovery tools like UFS-explorer, GetDataBack ...

Chances:
There is a good chance to recover files which were unfragmented inside the snapshot

Disktypes with external descriptor

vmfs
vmfsSparse
monolithicFlat
twoGbMaxExtentSparse
twoGbMaxExtentFlat

fullDevice
partitionedDevice
custom

Disktypes with embedded descriptor

monolithicSparse

streamOptimized

to extract the embedded descriptor run

dsfo.exe monolithicSparse.vmdk 512 800 descriptor.bin

to inject the descriptor again run

dsfi.exe monolithicSparse.vmdk 512 800 descriptor.bin