Skip to main content

Storage methods

This document provides an overview on the different storage methods that can be configured with Comet.

Storage can be configured using the Comet Backup desktop agent as well as using the Comet Management Console. While the complexities of different storage methods vary, the configuration is the same on both interfaces.

Important terminology

  • Storage Vault: This is the location where your backup data is encrypted and stored. This can be anything from a local disk to a network share or cloud-based storage such as AWS S3 or Wasabi.

  • Storage Template: This is a predefined configuration of storage settings that allows new Storage Vaults to be automatically created for new devices or users. One predefined configuration can be used to create thousands of Storage Vaults with no additional effort.

  • Comet Storage Gateway: Find out more about Storage Gateway here.

Methods

At a high level, Comet supports two different methods of uploading data to a Storage Vault.

The first method is where the Comet Backup desktop agent uploads data directly to the Storage Vault. The supported methods are:

  • Direct-to-Cloud with Storage Templates (recommended)
  • Direct to Storage Vault

The second method is where the Comet Backup desktop agent uploads data directly to a Comet Storage Gateway. The supported methods are:

  • Direct to Storage Gateway (using Storage Templates)
  • Proxied through Storage Gateway to the Cloud

A Storage Template is used to manage default storage options which are available when requesting a new Storage Vault for a user. Several S3-compatible cloud providers are supported.

This is the easiest and recommended way to configure cloud storage with Comet. A Storage Template only needs to be configured once and can be used to create thousands of Storage Vaults.

Direct to Storage Vault

A Storage Vault is a location where the backed-up data can be stored. All data is compressed, encrypted, and deduplicated within the Storage Vault.

It is possible to configure storage directly as a Storage Vault if it is not supported as a Storage Template, such as local path, SFTP, and S3-compatible cloud storage.

Using this method requires manual configuration each time a new Storage Vault is required. Comet does not recommend this method for large scale deployments.

Direct to Storage Gateway

This method uses the Comet Storage Gateway as the Storage Vault location. This method is recommended for customers wanting to back up data at scale to on-prem locations.

Once a Self-Hosted Storage Gateway is configured, the Comet Backup desktop application will upload data to the Storage Gateway first, which will then store the data in the location it has been configured to do so.

Comet recommends using the Storage Template system to configure this option.

Proxied through Storage Gateway to the Cloud

A Comet Storage Gateway can also be utilized as a proxy to upload data. This method allows for a wide range of advanced Storage configurations such as:

  • Designing infrastructure to have on-prem storage for fast backup and restores as well as maintaining an offsite copy for complete disaster recovery.
  • Using it to hide Cloud Storage credentials from the Comet Backup desktop application.

Appendix

Utilizing cloud storage at the client level (Direct-to-Cloud)

With this method, the cloud storage credentials will be exposed; even though the information is hidden in the Comet Backup client GUI, the customer could still discover it by network analysis or with a software debugger.

The access key and secret key are part of the Storage Vault settings inside the user profile, and are directly accessible by both server and client.

When using Storage Templates for direct-to-cloud storage, sComet will manage the data within the cloud provider's bucket, and generate unique sub-directory level access keys per Storage Vault to minimize the impact this could have if the credentials are acquired.

Direct-to-cloud also applies for custom Storage Vaults; ensure credentials are exclusive to the user, i.e. generated per-user credentials with separate buckets.

Utilizing cloud storage with Comet Storage Gateway

The backup data will pass through the Comet Self-Hosted Storage Gateway first and then the server will upload the data to the cloud storage provider (in memory). Cloud storage credentials won't be exposed to the client using this method. If your Comet Self-Hosted Storage Gateway is hosted in the same region as the cloud storage account, this may or may not have zero-rated bandwidth cost.

This method can be configured at the Storage Gateway location on your Comet Self-Hosted Storage Gateway's Settings page. When setting up a Storage Vault for an account, simply assign it a Storage Template. Your Comet Self-Hosted Storage Gateway will automatically create subdirectories to store the data for each new Storage Vault.

When using cloud storage as the storage location for Storage Gateway, the upload stream is proxied through to the cloud storage account (if possible), or buffered in memory. No data touches the local disk (no IOPS penalty). However, under memory pressure, the operating system may write to the swap file in proportion to the transfer load.