What's New?
"What's New?" is a series of blog posts covering recent changes to Comet in more detail. This article covers the latest changes in Comet Voyager over August 2022.
This month at Comet we've been busy with behind-the-scenes changes. We've been lucky to continue to welcome many more new staff members that have joined the development team. Throughout August, there were three "Voyager" releases across the 22.6.x series; the launch of the new 22.8 "Carme" quarterly release track; and the slightly early reopening of Voyager for the 22.9.0 release.
Comet 22.8.0 "Carme"
Carme is the latest entry in our quarterly rollup series. It branches off from our main rolling Voyager development, into a fixed target for our partners to qualify and build upon.
Comet 22.8 "Carme" takes its name from another moon of Jupiter. It was discovered along with our previous Comet release "Lysithea" by Seth Barnes Nicholson in 1938, and both discoveries were reported in the very same publication. Jupiter's moons are classified into groups, and despite sharing a discovery event, Lysithea and Carme come from different groups of moons and have very different orbital properties. The moon itself is named after the Cretan goddess Carme, one of Zeus' lovers.
Photograph by OHP (Observatoire de Haute-Provence), CC BY-SA 2.0-fr
Carme brings 7 features and 14 enhancements, including IDrive e2 support, presigned clients, the ability to spread out peak job load across a random period, custom SMTP servers for Tenants, support for fallback domain controllers, more restore options, and improved performance.
For this last month of August, there have been very few features landing in the Voyager release track, with mostly a focus on fixing issues. This has resulted in a trouble-free release process and a great release that we have confidence in. It does mean that there are only a few new features and enhancements to cover, as all the larger changes were held back for the 22.9 "Voyager" release.
Simpler codesigning for macOS
The main enhancement in the 22.6 "Voyager" series over the month of August, and included in our latest 22.8.0 "Carme" release track, is the removal of the requirement to install Apple iTMSTransporter on your Comet Server machine for the purposes of macOS client codesigning and notarization. This builds upon an announcement from Apple at the WWDC 2022 conference regarding their improved support for automation in this area. We're very grateful to the macOS codesigning community for contributing this feature.
In early versions of Comet, macOS codesigning required Comet Server to SSH into a real Mac machine somewhere on your network in order to run the official XCode toolchain. We were very proud to integrate rcodesign as a native, built-in alternative in Comet 22.3.3; the bundling of pre-signed clients in Comet 22.6.2; and now, no longer requiring the third-party iTMSTransporter application as of Comet 22.6.9.
Go SDK
Another feature released in August is our latest SDK, which makes it easy to call the Comet Server API using the Go programming language.
The SDK is available on GitHub under the MIT license. It includes all the definitions you may have seen in the PHP and Ruby SDKs, as well as in our public API documentation. It comes with built-in examples and is designed for use with the Go module system.
In our company we use the Go programming language extensively, so it's perhaps a little unusual that we didn't have a public Go SDK until this point. That's largely because PHP and Ruby produced the largest demand from external developers looking to integrate Comet with their web sites and ecommerce platforms; and in our in-house Go development environment, we can use the type definitions directly from Go without any SDK at all. However, having a public Go SDK definitely makes it easier to write and develop auxiliary tools, and we've already seen several utilities being ported to this great and efficient programming language.
Announcement about legacy operating system support
The 22.8.0 "Carme" release is the culmination of everything up through the previous 22.6.x "Voyager" release cycle. From this point, we're planning to make some bigger changes.
During the 22.9.x timeframe, we expect to remove support for running the Comet Backup desktop application on Windows Vista, Windows Server 2008, and macOS 10.10 "Yosemite".
If you have end-users running these operating systems, you should either
- (A) stay with the 22.8 "Carme" release cycle, or
- (B) install the 22.8 "Carme" Comet Backup desktop app and use it along with an upgraded version of Comet Server, or
- (C) arrange for your end-users to upgrade to Windows 7, Windows Server 2008 R2, macOS 10.11 "El Capitan", or newer.
This is the corresponding client-side change following our previous announcement of the end-of-life for running Comet Server on Windows Vista and equivalent Server 2008 late last year in Comet 21.12.1, and I'd like to provide an updated table for comparison:
| OS | Comet Backup | Comet Server |
|---|---|---|
| Windows Server 2008 (NT 6.0 / Vista-based) | ⚠ No longer supported in 22.9.x, please continue to use 22.8.x | ⚠ No longer supported in 21.12.x, please continue to use 21.11.x |
| Windows Server 2008 R2 (NT 6.1 / Windows 7-based) | ✅ Still supported | ✅ Still supported |
We don't take this removal lightly, but it's clear that in the present day, Windows Vista is not widely used or relevant, and we expect minimal issues as a result of this deprecation.
Although the Vista-based Windows Server 2008 and the Windows 7-based Windows Server 2008 R2 have the same extended support lifecycle, the underlying technology of the R2 update is significantly more advanced than the Vista-based RTM edition, and correspondingly easier for us to develop for. It's likely that Windows 7 and Server 2008 R2 will remain supported by Comet for a much longer period of time.
New design for the Comet Backup desktop app
In Comet 22.9.0 "Voyager", we've put a new coat of paint on the Comet Backup desktop app. A picture says a thousand words:
The desktop app now shows you a summary of your Protected Items, Storage Vaults, and recent job history at a single glance. We're very happy to have struck a balance between information density, visual identity, and efficient performance as the charts remain using an optimized native-code renderer. We are excited to continue improving the UI going forward and we expect to make tweaks over the 22.9 "Voyager" series in response to your feedback before this lands in the next quarterly release.
This dashboard screen is divided into five main areas:
The Storage Vaults chart at the top left shows up to three Storage Vaults. If the Storage Vault has a quota, the sawtooth segment will be split into two colors showing the partial usage percentage of your Storage Vault quota. If you have more than three Storage Vaults, three are chosen for display based on whether they have Quota limits in place, and also whether they have been recently used on this device. Further details about your Storage Vaults remain available on the Settings tab.
The Currently Protecting chart at the top right shows a breakdown of your Protected Item size, across this device and across other devices in your account. If your Comet user account has a limit on the number of total devices allowed to be registered, or a limit on the total Protected Item size, these figures will be displayed here.
The Storage Vault size chart at the lower left looks at the past 10 days' of job history to build a visualization of how your Storage Vaults have grown or shrunk in that time. Comet measures the size of a Storage Vault during backup operations, so this chart is most effective if there are recent backup jobs for your Storage Vaults. A maximum of three Storage Vaults are shown, and as long as the necessary data points are available, the same choices and colors will be used as per the Storage Vaults chart at the top left.
The Transfer chart in the lower right looks at the past 10 days' of job history as well, to show the upload and download size for your jobs, including both backup and restore operations. These figures correspond to the Uploaded and Downloaded job report columns and at a technical level they reflect the transfer size to- and from- the Storage Vault. Notably this means that Upload/Download sizes may be reported even for Local Path storage vaults, and, Upload/Download transfer amounts for over-the-network Protected Item types (such as remote MySQL and MongoDB servers, or Office 365 tenants) are not currently reported here.
The news widget at the bottom shows bulletin messages created by the Comet Server system administrator. It's useful for contact details, notices of recent service changes, or promotional messages. Any news messages you create in Comet Server are shown immediately in the desktop app. If you have no news entries, this bottom section is hidden entirely.
Storj.io
We're excited to announce a partnership with Storj.io, a decentralized cloud storage provider. It's been possible to use the Storj service with Comet's S3-compatible option, but in Comet 22.9.0 "Voyager" it's now possible to connect directly to the Storj P2P network without involving the centralized gateway. This allows for faster performance during download operations and lower latency without the centralized bottleneck.
At the time of writing, Storj coordinate a network of over 13,000 storage nodes worldwide and can offer great low-latency to nearby storage nodes at an industry-leading price of $4 / TB and a generous free tier of 150GB.
Please note that when using Storj in this way, Comet performs client-side erasure coding to distribute data redundantly throughout the Storj network. The file upload size reported in Comet reflects the amount of data delivered to Storj, but the upload size experienced over the network may be higher as a result of client-side redundancy.
A key aspect of the Storj integration is the ability to configure Storj as a Storage Template provider:
Storage Template providers (formerly known as Requestable Storage providers) allow your Comet Server to automatically provision new, private Storage Vaults for each of your customers, with isolated credentials - including Constellation Role support for automatically cleaning up Storj buckets that are no longer used by any Comet user account.
As with all of our supported storage platforms, the new Storj integration is also available as a backend for Comet Server Storage Role, if you would prefer all customer data to flow through your Comet Server.
Streaming restore to MySQL
Comet supports backing up MySQL using our dedicated Protected Item type. This was one of the original Protected Item types included with the very first Beta versions of Comet, and is available at no additional charge. It works by streaming the output of mysqldump directly into our chunking deduplication engine, without requiring any spool space or temporary files.
In Comet 22.9.0 "Voyager" we've introduced corresponding support for streaming MySQL restores:
The data will be streamed from Comet's Storage Vault into the MySQL server without needing to temporarily store any .sql files and without needing to manually import them.
You can configure the MySQL restore feature with all the same connection settings as for backup. The settings will be prefilled based on the current matching Protected Item, or they can be changed as required. This feature is available for restores performed in the desktop app, or remotely via the Comet Server web interface, or via the Comet Server API.
This is a great feature that simplifies the restore process for this highly popular database. We expect to announce a similar feature for another popular database very soon, so keep an eye on our release notes for the latest news.
Configure webhooks from the Comet Server settings page
Webhooks are a feature to send live events from Comet Server to your other systems. They are largely of interest to developers who want real-time information to flow from Comet Server into other systems, such as CRM and ERP systems.
Comet Server has supported webhooks since 20.6.1, with major enhancements happening for custom header support in Comet 21.9.2. However, they've largely been hidden and confined to users who read the documentation and can make manual changes to the cometd.cfg file. In Comet 22.9.0 "Voyager", we're improving the discoverability of this feature by allowing you to view, edit and manage your current webhooks from the settings page in the Comet Server web interface.
This feature is available for webhooks both at the top-level, and for individual Tenants, including Comet-Hosted users.
As a top-level administrator, you can control permission to access this feature via the admin permission checkbox or via the AllowEditWebhooks API field.
Constellation Role for tenants
Constellation Role has previously been restricted only to the top-level administrator. In Comet 22.9.0, it is now available at the Tenant level as well:
A "Bucket Users" report generated by a tenant Constellation is entirely independent of the top-level Constellation. The tenant also has an independent choice for the "Data deletion" setting.
Access to this feature can be controlled by a top-level administrator. If your Tenant administrator has all settings features enabled, this feature is included. If your Tenant administrator has been configured to have no- or limited- settings available, the Constellation Role feature is enabled by default. This is because Constellation Role is an essential partner of the Storage Template feature, to clean up disk space or cloud bucket space that was used by since-deleted user accounts.
Misuse of the Constellation Role feature can cause data loss, so please take care when configuring this setting. Our documented advice is that you should have exactly one Constellation Role for your Comet Server cluster; when running multiple Constellation reports on a per-tenant basis, the key advice is likewise to not overlap Constellation responsibilities - one set of user accounts and one set of buckets (local, Cloud Storage, Storage Role or otherwise) should be managed by one Constellation Role configuration only.
If you are using Constellation at the top-level of the Comet Server with top-level credentials, then it is able to see Tenant user accounts and deallocate their buckets as required. However, if you are in the situation of not using Constellation at the top level, or if you are not in control of which remote Storage Templates are used by your Tenants, then this new feature is especially useful to enable data cleanup.
Upcoming webinar
If you prefer to watch and listen rather than read, I'm hosting a webinar next week to discuss the latest 22.8 "Carme" software release. Please register for a notification before we go live on September 6th (PST) to catch up on all the latest Comet news with me - there will a free live Q&A session after the presentation.
As well as that, we have many more videos available on our YouTube channel, including guides on getting started with Comet, individual features, and webinars for previous quarterly software releases.