Disk Image
Overview
Using this Protected Item type may incur a Booster charge.
Comet supports taking disk image backups.
This Protected Item type is only applicable when running on Windows or Linux. Disk Image backup on other operating systems is not currently supported by this Protected Item type.
When using the "Disk Image" Protected Item type, on the Items tab, you can select any currently-attached drives for backup, or individual partitions from any drive. It is possible to select "all drives" and exclude individual disks or partitions.
Any change to the partition structure of a drive will cause that drive to be recognized differently in Comet. If you had selected such a drive, Comet will warn you that the drive can no longer be found. You would need to reselect the drive and/or partitions in the Comet Backup app interface.
Comet feeds raw data from each disk partition directly into its chunking deduplication engine. The disk image is deduplicated, compressed, and encrypted as it is being saved to the Storage Vault. No extra temporary spool data is generated and no additional disk space is required.
The backed-up disk image data will deduplicate with other data inside the Storage Vault. A 'Files and Folders' type backup of the same data volumes should achieve a high degree of space savings. The effectiveness of any such deduplication may be negatively affected by: (A) filesystem fragmentation on the physical volume; and/or (B) small file sizes.
Comet does not allow additional file exclusions within a partition.
Unused disk sectors
On supported filesystems, Comet will exclude unused space from the disk in the backup image. Unused space is represented as zero ranges, that are compressed during the backup phase. When restoring the disk image, the file will include uncompressed zero ranges. Please see the "Supported volume types" section for more information about what filesystems are compatible with this feature. You may disable skipping free space by enabling the "Include unused disk sectors for forensic data recovery" option.
When using this Protected Item type on Windows, the disk must be set as Online for Comet to exclude unused space. If the disk is set as Offline in Windows, Comet is unable to exclude free space, even from a supported filesystem. You can change a disk's Online/Offline state from Disk Management (diskmgmt.msc) or from diskpart.
If a disk extent does not contain a filesystem (e.g. if it is a raw byte range), then Comet is unable to determine which disk sectors are needed. If you select a "Raw byte range" extent, it is backed up in its entirety, even if the "Include unused disk sectors for forensic data recovery" option is selected. If the raw data contains mostly zero bytes, it will be highly compressed during the backup phase and when stored as chunks in the Storage Vault; however, if the raw data contains mostly random data, it will not compress well.
Disk Image for Windows
Comet always skips backing up the pagefile of the booted Windows installation (pagefile.sys / swapfile.sys), even if the "Include unused disk sectors for forensic data recovery" option is enabled.
Drive letter support
Selecting disks/partitions by drive letter requires Comet 24.3.7 or later.
You can select disks/partitions by drive letter. Please note we recommend using the Live Picker over this method, however, this is useful for policies or for when you don't have a live connection to the device.
Selecting a drive letter that has only one mount point on the drive will include every partition on the drive for the backup. Meanwhile, if there are multiple mount points on the drive, only the partition of the selected drive letter will be backed up.
To access this option via the Comet Server web interface, when adding a Disk Image you must change the Type value to Include drive letter.
To access this option from the Comet Backup desktop app, on the Items tab you must first enable the option Select by Drive Letter (advanced), then swap to the Drive Letters tab.
Supported volume types
Please refer to the following table of filesystem support notes:
| Filesystem | Skip unused space | Consistency |
|---|---|---|
| NTFS (Microsoft) | Yes | Snapshot |
| ReFS (Microsoft) | Yes | Snapshot |
| FAT32 (Microsoft) | Yes | If volume is not in use |
| exFat (Microsoft) | Yes | If volume is not in use |
| UDF (Microsoft) | No | If volume is not in use |
Third-party filesystem drivers (e.g. WinBtrfs, Ext2Ifs, Paragon Linuxfs, ZFSin) have not been officially tested against Comet.
Please refer to the following table of special volume type notes:
| Volume type | Supported | Notes |
|---|---|---|
| Basic disks | Yes | Fully supported |
| Dynamic disks | Yes | The underlying volume will appear as "Raw byte range". For a span or striped volume, you should make sure to only select the dynamic volume for backup, not the underlying raw disk. Please also note that Dynamic disks are deprecated in Windows 8 and above. |
| Storage Spaces | Yes | The underlying volume will appear as "Orphaned volume". You should make sure to select only the Storage Space for backup. |
| Bitlocker | Yes, while unlocked | The backup can succeed if the Bitlocker volume is unlocked. If the Bitlocker volume is locked, it should be unlocked before running the backup job, otherwise you may experience an error This drive is locked by BitLocker Drive Encryption. You must unlock this drive from Control Panel.. The resulting partition backup is not protected by Bitlocker and you may extract single files from it without the Bitlocker encryption key, as described below. If you restore to a physical partition, you may wish to re-enable Bitlocker after restoring, via the Windows Control Panel. |
| Cluster Shared Volume | not tested | not tested |
| Truecrypt / Veracrypt | not tested | not tested |
Please refer to the following table of physical media notes:
| Physical media | Supported | Notes |
|---|---|---|
| Hard drive (512n) | Yes | Fully supported |
| Hard drive (AF) | Yes | 512e and 4Kn (Advanced Format) harddrives are supported. |
| Mounted VHD / VHDX | Yes | Fully supported |
| Removable USB drive | Yes | Some removable drives cannot be completely offlined by the operating system; a restore operation back to the physical removable USB drive may be interrupted by other programs on the PC. |
| Remote iSCSI LUN | Yes | Tested successfully against the Windows Server File and Storage Services iSCSI implementation |
| Mounted ISO | No | Only harddrive (HDD / SSD) disks are supported. |
| Optical drive | No | Only harddrive (HDD / SSD) disks are supported. |
| Floppy drive | No | Only harddrive (HDD / SSD) disks are supported. |
Please refer to the following table of partition table notes:
| Partition table | Supported | Notes |
|---|---|---|
| MBR | Yes | Fully supported, including Extended partitions (EBR) |
| GPT | Yes | Fully supported |
Consistency
Comet tries to take a VSS snapshot of the selected partition (without invoking any specific writers for quiesence). If this succeeds, the partition backup is crash-consistent.
Comet tries to lock the volume handle. If this succeeds, the partition backup is crash-consistent.
Otherwise, Comet will print a warning to the job log, and back up the partition in a rolling way. The backup may be inconsistent if other processes are writing to the partition at the same time.
Disk Image for Linux
Using Disk Image Protected Items on Linux requires Comet 24.9.10 or later.
Supported volume types
Please refer to the following table of filesystem support notes:
| Filesystem | Supported | Skip unused space |
|---|---|---|
| EXT4 | Yes | Yes (with Kernel 4.12 or later supporting the GETFSMAP feature) |
| XFS | Yes | Yes (with Kernel 4.12 or later supporting the GETFSMAP feature) |
| Other | Yes | No |
Please refer to the following table of special volume type notes:
| Volume type | Supported | Notes |
|---|---|---|
| Basic partition (hdX, sdX, nvme) | Yes | |
| LVM2 | Yes | You can select either the Logical Volume (LV) or the underlying disks. Selecting the LV is recommended. |
| MD Raid | Yes | You can select either the RAID volume or the underlying disks. Selecting the RAID volume is recommended. |
| dm-crypt / LUKS | Yes | You can select either the encrypted volume or the decrypted volume. Selecting the decrypted volume is recommended. |
| Loop devices | No | Loop devices are prevented from selection. |
| RAM disks | No | RAM disks are prevented from selection. |
| Optical drive (srX) | Yes | Optical drives will appear for backup only if a disc is inserted in the drive. |
| Btrfs | not tested | |
| Bcache | not tested | |
| ZFS | not tested |
Consistency
For each selected partition, Comet attempts to take a temporary, point-in-time snapshot of the disk using the following methods:
- Check if the partition has no mount points
- Snapshot with LVM2
- For this option to succeed, the selected partition must be a Logical Volume (LV). Additionally, there must be sufficient free space within its parent LVM Volume Group (VG) to support creating a temporary snapshot LV. If the VG is completely filled by the available partitions, this method will not succeed unless partitions are first shrunk to accommodate.
- Comet supports both LVM Thin VGs and standard LVM VGs.
- It is possible to convert a system from flat partitions to using LVM with the lvmify program.
- Snapshot with backupsnap61
- backupsnap61 is a custom Linux kernel driver that can snapshot any block device. If it is installed on the system, it will be used to snapshot the block device for backup.
- The driver requires Kernel 5.9 or later and DKMS support.
- A copy of the backupsnap61 installer is bundled with Comet Backup. To install this driver, run
backupsnap61-install.shfrom your Comet Backup install directory.
If any of the above methods succeed, the partition backup is crash-consistent. The partition backup also becomes eligible for skipping unused filesystem sectors.
Otherwise, Comet will print a warning to the job log, and back up the partition in a rolling way. The backup may be inconsistent if other processes are writing to the partition at the same time. In this situation, the backup is also not eligible for skipping unused filesystem sectors.
Restoring
Comet stores the disk image files in VMDK format. You can restore these files normally using Comet.
There is one plain-text VMDK descriptor file representing metadata about the whole drive, plus separate raw image files for each partition's extent on the disk.
Partitions of the disk that were not selected for backup are represented as zero extents in the VMDK descriptor file. This means the restored disk image appears to have the full disk size, even if only a small amount of partitions inside it were selected. The zero extents will be compressed inside the Storage Vault.
On Windows, the Comet Backup desktop app offers the option to restore the disk images either back to physical partitions, or as files.
Recovery of single files
Comet Backup supports the "Granular restore" option to restore individual files and folders from inside the Disk Image backup. See the appendix for more details.
Booting into a recovered Windows OS installation
When migrating a Windows OS installation to different hardware, any products which use hardware identifiers as a software licensing component may lose their activation status. This includes, but is not limited to
- Windows OS activation, and
- Comet device detection. See "Change of hardware causes registration dialog to appear".
The "C:" does not contain everything needed to boot an operating system. For best results when creating a bootable image, you may wish to ensure that your backup includes
- the disk's non-partition space (that includes the GPT/MBR partition table)
- the "System Reserved Partition", if present (that contains the volume boot record)
- the EFI ESP partition, if present (on GPT disks and/or UEFI-booting machines)
Windows 8.1, Windows 10, and later
Current versions of Windows do generally handle being booted on dissimilar hardware without any issues.
Earlier versions of Windows
When you boot a Windows OS installation, it may automatically become specialized for the running hardware (physical or virtual). This improves performance, but can prevent the same OS installation from booting on different hardware if the hardware is sufficiently different. The tolerable differences depend on the hardware in question.
If you experience errors booting a backed-up Windows OS disk image on different hardware (physical or virtual), it may be necessary to prepare the Windows installation for hardware-independence. You can do this by running sysprep inside the installation before taking the disk image; or, you can do this by booting a Windows recovery environment, mounting the image, and running sysprep against the attached disk.
The sysprep tool is installed in the C:\Windows\system32\Sysprep\ directory and is available on all Windows SKUs. From Windows 8.1 onward, its GUI is deprecated in favor of command-line use.
Filesystem smaller than target volume
When restoring a smaller partition into a larger one, Comet will automatically extend the restored filesystem to the fill the target partition.
- This feature is available on Windows if the filesystem driver supports it (the NTFS and ReFS file systems).
- This feature is available on Linux for supported filesystems if the necessary system utilities are installed (
resize2fsfor EXT,xfs_growfor XFS,ntfsresizefor NTFS,fatresizefor FAT).- These tools are automatically installed when using the Linux Recovery Media feature.
In other cases, the result will be a large partition containing a small filesystem. It appears to have the large size in Disk Management (that looks at the partitions) but the small size in This PC (that looks at the filesystem). The extra space from the new larger partition cannot be practically used until the filesystem is extended, to fill the partition around it.
On Windows, you can independently repeat Comet's attempt at manually extending the filesystem to fill its containing partition by
- opening Command Prompt as administrator
- run
diskpart.exe - type
list volume - Identify the target volume from the list, and then type
select volume TARGET_NUMBER - type
extend filesystem
On Linux, you can resolve this issue by using the resize2fs, xfs_grow, ntfsresize, or fatresize command as appropriate for the target filesystem type.
Filesystem larger than target volume
Comet does not support restoring a large backed-up partition into a smaller physical partition. If you are trying to do this, please shrink the partition using the OS's partition manager prior to performing the backup.
Recovery to physical hardware
In order to restore to physical hardware, the target disk or partition should be unmounted. Comet Backup may be able to do this automatically from your current booted OS, if no programs are using the target drive (e.g. for a non-boot drive); but in order to restore to your boot drive, you should first reboot the PC into a recovery environment.
For more information about creating a recovery environment, see the Recovery Media documentation.
From the boot environment, run Comet Backup, and open the Restore wizard. The Restore wizard inside Comet Backup allows restoring the backed-up disks and partitions directly to your physical disks and partitions, without requiring any temporary spool space.
You can use the "edit" button to repartition the local drives. After doing so, use the "refresh" button to refresh the local disks and partitions for restore.
To do so:
- Select a backed-up disk or partition to restore, from the left-hand pane
- Select a target disk or partition to write to, from the right-hand pane
- Click the "Add to restore queue" button
- Repeat steps 1-3 as necessary
- Click the "Restore" button to begin the restore job.
Recovery to local VM
You can attach the *.vmdk disk image files to a new- or existing Virtual Machine. If the disk image contains a Windows OS installation, it may be bootable.
| Virtualisation platform | Supports Comet's *.vmdk file format |
|---|---|
| VMware Fusion / Player / Workstation | Yes |
| VMware vSphere | No - Restore as VMware virtual disk to generate compatible disk images. |
| QEMU | Yes |
| Virtualbox | Yes |
| Hyper-V | No - must convert to VHD or VHDX format |
If your PC boots using EFI - for instance, if the source disk contains an EFI System Partition (ESP) - then you should configure the VM to boot in EFI mode ("Generation 2" in Hyper-V). Otherwise, you should configure the VM to boot in "Legacy" / MBR mode ("Generation 1" in Hyper-V).
When planning to restore as a VMware vSphere virtual machine, backing up all disks and partition is recommended to ensure that the expected disk structure is intact.
Converting VMDK variant for VMware ESXi
This feature is built-in to Comet 23.9.5 and later. The following text can be used for earlier versions of Comet.
VMware ESXi supports booting a restored Disk Image from a Comet backup job (Physical-to-virtual / P2V). However, the VMDK file format used by Comet uses multiple extents, which is not supported by ESXi until it is converted first.
You can boot your restored Comet disk image vmdk files in ESXi following these steps:
- Make Comet's restored vmdk files accessible to your VMware Datastore.
- For instance on a single ESXi server, you can upload the files through the Datastore browser; or on a SAN environment, you can copy them out-of-band to the SAN storage.
- SSH into the ESXi server.
- You can temporarily enable SSH from the ESXi web console.
- Enter the datastore directory containing the restored Comet vmdk files:
cd /vmfs/volumes/my-datastore-name/path-to-comet-disk-images/ - Start to convert the disk.
vmkfstools -i ./disk.vmdk converted.vmdk -d thin- This will require additional temporary disk space, up to the size of the disk image.
- You can replace
-d thinwith-d zeroedthickto preallocate the full disk size. For more information on the available VMDK types, see VMware's documentation. - Once the conversion has completed, you should see a new
converted.vmdkandconverted-flat.vmdkfiles.
- Delete the original Comet vmdk files:
rm ./disk*.vmdk - Continue to configure the ESXi virtual machine
- on the Virtual Hardware tab, ensure the converted disk is attached to the VM, not the original disk
- on the Virtual Hardware tab, ensure it is connected to the SATA controller, not the SCSI controller
- on the VM Options tab, ensure the VM is configured for either BIOS or EFI boot as appropriate
Converting VMDK to VHDX for Hyper-V
This feature is built-in to Comet 24.9.2 and later. The following instructions can be used for earlier versions of Comet.
Hyper-V supports booting a restored Disk Image from a Comet backup job (Physical-to-virtual / P2V). However, Hyper-V does not support Comet's file format; it must be converted first.
You can convert Comet's vmdk files to Hyper-V-compatible vhdx files using qemu-img:
- Download
qemu-imgfor Windows from qemu.org - From a command prompt, run
C:\path\to\qemu\qemu-img.exe convert -f vmdk -O vhdx C:\path\to\comet\disk.vmdk C:\path\to\output.vhdx -p- This will require additional temporary disk space, up to the size of the disk image.
- You may also need to run the command
fsutil sparse setflag C:\path\to\output.vhdx 0if you run into sparse file errors. - Delete the original Comet vmdk files.
Recovery to cloud server
You can upload the *.vmdk disk image files to a cloud provider. Depending on the cloud provider's capabilities, it may be possible to boot a new VM from them, or to attach them as extra disks to an existing VM.
If the disk image contains a Windows OS installation, it may be bootable. Not all cloud providers support booting Windows OS installations.
| Provider | Supports *.vmdk file format | Information |
|---|---|---|
| Amazon EC2 | Yes | https://aws.amazon.com/ec2/vm-import/ https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html |
| Azure | No - must convert to VHD format | https://docs.microsoft.com/en-us/azure/virtual-machines/windows/upload-generalized-managed |
| DigitalOcean | Yes | https://blog.digitalocean.com/custom-images/ |
| UpCloud | Yes | https://upcloud.com/community/tutorials/import-vmware-images/ |
Converting Windows System to Disk Image
With the deprecation of our alternative "Windows System Backup" Protected Item type, we have implemented a way to automatically convert a "Windows System Backup" Protected Item into a Disk Image Protected Item.
Please note that when the "Windows System Backup" Protected Item type is deprecated this conversion will take place automatically as part of the Comet Server startup process.
The conversion is handled using our new user bulk action option on the Comet Server web interface. Simply navigate to the "Users" tab, select the users you would like to perform the conversion for, click the "Bulk Actions" dropdown button and select "Convert Windows System Backup Protected Items". A notification should display with the number of Protected Items that were converted.
For "Windows System Backup" Protected Items that weren't backing up disks with only a single assigned drive letter, we recommend converting the Protected Item manually, as this conversion option only supports converting disks with a single drive letter.
Please note that the "All Critical Volumes" option is assumed as the C: drive.
