How to configure cloud storage with Comet

This guide instructs users on the different ways cloud storage can be configured with Comet.

For a more technical overview on how cloud storage works with Comet, as well as detailed information on requirements to configure specific cloud storage types, please visit our cloud storage technical documentation.

Options for configuring cloud storage

Comet is storage-agnostic and can support any S3-compatible cloud storage provider, as well as SFTP and WebDAV. At a high level, there are two ways that cloud storage can be configured with Comet:

Direct-to-cloud, where the backup data goes directly from the Comet client to the cloud storage provider. This is the most common use case and is supported in both Self-Hosted and Comet-Hosted Servers.

Through the Server, to the Cloud, which means the backup data will pass through the Comet Server first, and from there, gets uploaded to the cloud storage provider (in memory).

This is handy for the following:

a. For backup replication, when the backup data passes through the Server first, the Server can direct a replica to a secondary storage location. This setup is only possible on a Self-Hosted Server.

b. If no Storage Template exists for your chosen S3-compatible provider, you can configure it on Storage Role, and then configure the Comet Server Storage Role as a Storage Template.

c. For airtight white-labelling, as it may be possible for a very savvy client to break through any direct-to-cloud white labelling.

Direct-to-cloud configuration

Direct-to-cloud backups can be configured manually on a Storage Vault level, or as a Storage Template which allows you to easily provision a Storage Vault for your user.

Storage Template

The Storage Template section is used to manage default storage options. Setting up a Storage Template allows you to easily provision a Storage Vault for your user.

When adding a Storage Vault for a new or existing account, you can simply assign a Storage Template instead of manually creating a storage location per account. Comet will automatically separate user data and therefore there is no need to create separate bucket locations manually.

Steps to configure a Storage Template

Step 1:

Go to Settings > Authentication Role section, and scroll down to the Storage Templates. Here, click on ‘Add’.

Step 2:

The Storage Template can be renamed in the Description field. Then click on the ‘Type’ dropdown to select the storage type / provider to be configured.

Several cloud storage providers are readily available to be configured as a Storage Template. If your cloud storage provider is not found in this list, they cannot be configured as a Storage Template and will have to be configured as a Storage Vault instead.

Or, they can be configured in Storage Role as an ‘S3-compatible’ storage type, and then configured as a Storage Template, with the knowledge that the backup data will then pass through the Comet Server first.

Step 3:

In the Storage Template wizard, configure the following:

a. Storage type: Put in the storage information required by your selected provider. Then click on 'Test Connection' to check that the configuration is accurate.

b. Quota: A storage quota can be set per template, which will be applied to each Storage Vault requested from that template.

c. Rebrand Storage: You may hide the storage brand from your end users in the Comet Backup client and web portal interfaces.

Step 4:

Save your changes within the Storage Template configuration window and within the Authentication Role page.

You can now request a Storage Vault from this template.

Through the Server, to the Cloud configuration

This method can be configured at the Storage Role location on your Comet Server's Settings page.

Once configured, the cloud storage can then be configured as a Storage Template from which you can automatically request a Storage Vault.

An end user will see the Storage Vault as your server’s storage and not the actual destination (such as, not seeing cloud storage credentials).

Storage Role

Storage Role is only available on a Self-Hosted Comet Server.

When using cloud storage in Comet Server as the Storage Role, the upload stream is proxied through to the cloud storage account (if possible) or buffered in memory. No data touches the local disk (no IOPS penalty). However, under memory pressure, the operating system may write to the swap file in proportion to the transfer load.

Steps to configure cloud storage in Storage Role

Step 1:

Go to Settings > Storage Role, and click on the plus sign.

Step 2:

Choose the cloud storage you would like to configure. Some storage providers are easily available and the configuration fields are customized to their required information (bucket name, secret key, etc.).

If your storage provider is not in the list, choose ‘S3-compatible.

Step 3:

Configure your selected cloud storage and click on 'Test Connection' to check that the configuration is accurate. Make sure that ‘Enable Storage Role’ is on.

Step 4:

After configuring cloud storage in Storage Role, go to the Authentication Role tab and set up the Comet Server as a Storage Template. Make sure that the storage Type is pointed to Comet Server (Storage Role), and the address is set to Local.

Once the Storage Role has been configured as a Storage Template, a Storage Vault can be automatically requested.

Storage Role Replication

With the cloud storage set up in the Storage Role to go through the Comet Server first, the backup data could be replicated onto another storage location through the Storage Role Replication feature.

This is covered in more detail in our separate Storage Role Replication guide.

Storage Vault Configuration

A Storage Vault is a location where the backed-up data can be stored. All data within a Storage Vault is compressed, encrypted, and deduplicated. A Storage Vault is the unit of deduplication; data is not deduplicated between multiple Storage Vaults.

Cloud storage and on-prem storage (e.g., SFTP, local path, FTP, SMB) can be configured on a Storage Vault. Storage Vaults are configured per user profile.

Configuring a Storage Vault on the Comet Server

Step 1:

Access the user profile you would like to configure a Storage Vault for, and go to the Storage Vaults tab.

Step 2:

Click on ‘Add new vault’, and –

a. If a Storage Template is available, click on the Storage Template and a Storage Vault will automatically be requested from the template. If the cloud storage is configured directly as a Storage Template, then this configuration will be a direct-to-cloud backup.

If the cloud storage is configured on Storage Role, then this will go through the Comet Server first, and then sent to the cloud storage.

b. If no Storage Template is available because the storage provider cannot be configured as a Storage Template, click on ‘Custom’. This configuration will be a direct-to-cloud backup.

Step 3:

In the General tab in the Storage Vault setup wizard, the following can be configured:

a. Name: The Storage Vault can be renamed.

b. Storage Quota: A quota can be applied per Storage Vault. This is a great feature for limiting an end user is allowed to back up. If a quota is about to be exceeded, the next backup will throw an error.

c. Rebrand Storage: Option to hide the storage brand in the Comet Backup client software and in all the web interfaces.

d. Retention: Storage Vault retention rules are the default for all data stored, unless a Protected Item has its own set of retention rules. When a Storage Vault is first created, the default, no-change-by-policy retention rule is 'Keep all data forever'.

Please visit our retention guide for more information on possible retention configurations.

Step 4:

Click on Next, and in the Storage tab, click on the plus sign next to ‘Choose a storage location’.

Step 5:

Choose the storage type and click Save. Some storage providers are readily configurable and showing they require for setup (bucket name, secret key, etc.).

If your storage provider is not in the list, choose ‘S3-compatible’ or ‘S3-compatible (Object Lock)’ if using object lock.

Step 6:

Configure the access and secret keys, bucket name, and other required information. For more information on required information per storage provider, please access our cloud storage technical documentation.

Click on ‘Next’ when done.

Step 7:

The Commands tab is for configuring commands to run before and after any backup job. For more information about the Commands feature, please see details under the Overview and Concepts section.

Step 8:

After configuring the Commands tab, click on Save. The Storage Vaults will now be available to use for backups.

When setting up a backup schedule or running a manual backup, make sure you are directing the backup to the correct Storage Vault.

Configuring a Storage Vault on the Comet Client

A Storage Vault can also be configured, or requested from a Storage Template, on the Comet Backup Client.

Step 1:

In the Comet Backup Client, click on the ellipsis, then click on 'Settings'.

Step 2:

Here, you will see all the Storage Vaults already available for this device to back up to, or to restore from. To add a new Storage Vault, click on the plus sign.

Step 3:

You will see the available Storage Templates here and you can automatically request a Storage Vault by selecting the Storage Template and then clicking on 'OK'.

If there are no Storage Templates configured, you can set up a cloud storage by clicking on ‘Custom’, then 'OK'.

The Storage Vault setup wizard is the same here as in the Comet Server, except for the re-branding option. This can only be ticked or un-ticked in the Comet Server.