Written by Mason Giles - CTO.
"What's New?" is a series of blog posts covering recent changes to Comet in more detail. This article covers the latest changes in Comet Voyager over September and October 2024.
During September and October we shipped four and six Comet releases respectively. The releases were primarily in our Voyager series with the exception of one update per month for the Tethys release series.
Let's take a look at some of the new features and enhancements added over this period:
Disk Image
When going to restore a Disk Image backup, there is a new option to restore the data in Hyper-V VHDX file format:
The disk format conversion happens on-the-fly as part of the restore process. It does not require any additional temporary disk space or memory. The resulting VHDX file can be used immediately in Hyper-V, or uploaded to Microsoft Azure to boot in a cloud virtual machine.
This joins our existing support for restoring Disk Image backups to VMware ESXi-compatible VMDK files, which works in the same streaming way without any temporary disk requirements.
This new capability was a highly upvoted request from our Feature Voting system. We encourage you to check out this system and let your voice be heard - at the time of writing, we have completed over 180 user feature requests. The votes are not the only thing we base our decision-making on, but they are an important input into the process.
When restoring Disk Image backups to physical hardware, using Recovery Media is essential in order to allow overwriting the device's primary operating system partition. Users of the Disk Image Protected Item can enjoy another new feature this month: the ability to use the Create Recovery Media feature when the Comet Backup desktop app is running on Linux.
Hyper-V and VMware
Comet's current billing model for Hyper-V and VMware Protected Items has a component based on the number of guest Virtual Machines included in the backup. Comet works with a wide range of service providers, both highly managed and low-touch, and everywhere in-between. If you are a service provider who offers the software directly, you may be surprised to find your customer backing up additional virtual machines and causing a rise in your Comet Backup bill. To help mitigate this, we have added a quota feature for the number of Virtual Machines:
Adding this quota limit to your customer accounts will ensure that as a service provider, you do not receive any unexpected charges for your users.
In the 24.9.x Voyager series, we have made additional improvements for the Hyper-V and VMware Protected Items:
When restoring data from a Hyper-V, VMware, or Disk Image Protected Item type, Comet offers a granular restore option, to extract individual files from an NTFS, EXT4, XFS, or FAT partition. This allows you to back up only the full disk image while maintaining the ability to rapidly recovery individual files.
If you use the "Latest VM State" backup option in Comet, this produces a single VHDX file inside the Storage Vault. However, if you back up Hyper-V using the "All VM Checkpoints" option, running the backup job in Comet will produce a base VHDX file and a chain of differencing disks in AVHDX format. In this case, a limitation in Comet meant that only the base VHDX file was supported for granular restore. For many service providers who do not actively maintain their checkpoint trees, this could result in only a surprisingly older file being available for granular restore. Now, in the latest 24.9.x versions of Comet, this limitation has been lifted, allowing you to perform granular restore even from each AVHDX checkpoint file.
The VMware Protected Item only had the "Latest VM State" option available and so was not affected by this limitation.
In 24.9.x, VMware users can enjoy simplified configuration as we have consolidated to supporting only the vSphere API, not any of the previous beta connection formats.
Restore
When going to restore data, you can control what happens if you restore data over the top of existing files on disk. Some common choices are to replace the existing file with the restore file; or, to skip restoring the file if an existing file is already present.
However, even if you instruct Comet to overwrite existing files on disk, an issue can occur if the existing file on disk has been set as read-only in the operating system. By default, Comet respects this setting and refuses to overwrite the existing file in this case. Respecting the file permissions is a sensible baseline behaviour, but we have encountered situations where Comet should restore the file regardless. As a result, the Comet 24.9.x Voyager series introduces a new "Allow read-only files to be overwritten" advanced option when restoring data, to bypass read-only file permissions.
Bulk actions
Comet Server has many features to make it easier for a service provider to administer a large user base. In the latest 24.9.x Voyager series, we've extended the "Bulk Actions" button on the Users page to support assigning user policies in bulk.
Policies are specific to a given Tenant. In previous versions of Comet, there were some situations by which the top-level administrator could assign a top-level policy to a Tenant, in a way that would have required the Tenant to access a policy existing outside their own Tenant. To clear up this edge case, when upgrading to the latest 24.9.x versions of Comet, any such policies will be copied into the target Tenant, ensuring that Tenants have tightly bounded permissions that only provide access to their own policies.
Larger service providers with multiple Comet Servers, used in a cluster with replication and/or constellation, may also appreciate the new ability to copy and paste your server settings from one Comet Server to another. This feature is also particularly helpful when migrating between the Self-Hosted and Comet-Hosted servers.
MySQL
Comet has supported backing up MySQL and compatible (e.g. MariaDB) databases ever since our earliest releases. The MySQL Protected Item type is based on the mysqldump technology in a special way that streams the database dump directly through our chunking deduplication engine and into the Storage Vault location, without needing any temporary files or spool space.
In the many years since the MySQL Protected Item was first launched, the available MySQL and MySQL-compatible engines have diverged in terms of feature set and functionality. For example, MariaDB introduced JSON functionality prior to MySQL's implementation, and feature differences persist to this day between these two major players. Therefore, to achieve a high fidelity backup job, it is important that Comet finds a copy of the mysqldump tool that is appropriately specialized for your database to support all of its latest particular features.
In the event that mysqldump cannot be found on the system, Comet will proceed with the backup job using a bundled fallback version. This fallback is not specialized for any particular version of MySQL and may be missing support for advanced database features.
In the latest 24.9.x versions of Comet, we have extended this fallback system from only applying on Windows, to now also applying on Linux and macOS devices. Comet now also adds a notice into the MySQL backup job logs if the fallback tool is being used. This can help alert you to the fact that the backup job may be operating at a lower fidelity.
Comet Backup desktop app
Comet is used by service providers of many different sizes. Many service providers choose to use the Comet Server web interface to remotely manage their customer base; the Comet Backup desktop app is also available for self-service. Depending on your focus as a service provider, you may use Policy controls to allow or prevent end-user access to the Comet Backup desktop app.
Throughout September and October, the Comet Backup desktop app has been improved to better show cross-device names in the Activity Log dialog. This includes both a new Device Name column, and, support for showing the Protected Item names from other devices.
The Comet Backup desktop app also allows the end-user to configure their own email reports. This month, the email filters for "Time since job start" and "Time since job end" have been improved to more easily allow configuring the seconds, minutes, hours and days for the time range query.
Troubleshooting
Troubleshooting customer backup jobs is an essential part of being a service provider. The 24.9.x Voyager series of Comet includes many small changes that may make this important job a little easier.
There has been a focus on improving how Comet connects to SMB network shares. On the Protected Item side, when using the "Log in to a network share" option in a Files and Folders Protected Item, a "Test Connection" button is now offered. When using a File and Folder Storage Vault on an SMB location, the Test Connection button now more accurately represents the service user account that would perform backup and restore operations. Additionally, our dedicated SMB Storage Vault type has also been improved to seamlessly handle both domain- and workstation-based authentication, using standard Microsoft syntax.
When the Comet Backup desktop app tries to test a connection and encounters a failing error, an extra info tooltip now appears, to show you additional information when hovering over it with the mouse cursor. If the Test Connection button had been explicitly clicked, a popup will show you the full error message in its complete detail.
When looking at a backup job log, Comet will now display additional progress information during some long-running Object Lock operations. This can help you identify any performance bottlenecks you may be facing. Additionally, when viewed in the Comet Server web interface, if a given backup job was prevented from running by another job taking place to the same Storage Vault at the same time, a banner message will now show you information about the conflicting job, helping you to easily identify and resolve the cause of the conflict.
Reliability and performance
Building upon our key focus from the 24.8 Tethys release series, the 24.9.x Voyager series this quarter has continued to prioritize work on bugfixes and reliability improvements. To date, this quarter's Voyager series has included fixes for over 50 issues, ranging from minor cosmetic to important functionality.
Our Office 365 Protected Item type has seen another major improvement, making incremental backups of Calendars, Contacts, and Messages more efficient and faster.
Comet is now much more resilient at handling certain types of network interruptions. When a large network download is interrupted, Comet is now able to resume from the point of failure in more cases, rather than needing to restart the entire transfer. Similarly if a Storage Vault has many concurrent jobs running, Comet is now more able to anticipate file changes within a remote Storage Vault and continue its operation. This can improve the performance of connecting to a Storage Vault at the start of a backup or restore job.
