Available storage locations

Comet supports S3-Compatible as well as on-prem storage.

This document lists all the storage locations that Comet supports and provides instruction on required fields to configure each storage type.

Local path

Data will be stored on the local filesystem.

Configuration

The following configuration options are available:

Option	Requirement	Description
Local path	Mandatory	The path to store data

Simultaneous connections

In this mode, Comet does not place a limit on the number of simultaneous accesses to the storage.

Windows

Some versions of Windows have a limit of 255 characters in a path name. As of Comet 2.8.6, Comet works around this issue, so there is no restriction on the path length that you use for local storage.

If the storage path is a mapped network drive, then ensure that you consider any path restrictions on both the mapped drive and its source drive.

Storage Gateway Bucket

In the Comet Backup client application, this option appears under a rebranded name.

Comet includes the "Storage Gateway" microservice. The Storage Gateway is an object storage system; you can create any number of "buckets", each with a secret authorization key, which can be used to store and retrieve structured data from the bucket.

A Storage Gateway Bucket is a supported data storage location for a Storage Vault. In the interest of preventing confusion, Comet does not accept using another Storage Gateway Bucket as a data storage location; if you want to store data from one Comet Storage Gateway on another physical Comet Storage Gateway, consider enabling Storage Gateway replication instead.

Configuration

Option	Requirement	Description
Hostname	Mandatory	The URL of the Storage Gateway hosting this bucket. The URL should include the protocol (http/https) and a trailing slash.
Bucket ID	Mandatory	The name of the bucket.
Bucket Key	Mandatory	The secret key, to authorize bucket access.

Simultaneous connections

Comet Backup makes up to 10 simultaneous connections to the Storage Gateway

If multiple operations are running simultaneously in Comet Backup, the number of connections may be higher.

SFTP

Data will be stored on an SFTP server, such as OpenSSH.

Configuration

The following configuration options are available:

Option	Requirement	Description
Address	Mandatory	The hostname for the SFTP server. You may specify a port by adding a :22 suffix. The default port is 22.
Username	Mandatory	The username to log in to the SFTP server.
Remote path	Optional	A remote path or subdirectory where data will be stored on the SFTP server. If the path starts with /, it is an absolute path. If the path does not start with /, the path is relative to the SFTP user's home directory.
Authentication	Mandatory	The authentication type to log in to the SFTP server. Must be one of Native, Password, or Private key. In Native mode, the system OpenSSH configuration is used to log in to the server.
Password	Required when Authentication is Password	The account password.
Private key	Required only when Authentication is Private key	The account private key, unencrypted, in OpenSSH format.
Verification	Required only when Authentication is not Native	Whether to verify host keys for the SFTP server. Must be one of Native, Allow any host key, or Custom path to known_hosts file. In Native mode, the system OpenSSH configuration is used to check known hosts.
known_hosts file	Required only when Verification is Custom path only	A local file path, used to keep track of SFTP server host keys.

Simultaneous connections

During a job, Comet Backup performs up to 10 simultaneous actions upon an SFTP destination.

If multiple operations are running simultaneously to an SFTP Storage Vault in Comet Backup, the number of network connections may be higher.

FTP

Data will be stored on an FTP server, such as FileZilla Server or ProFTPd.

WARNING: The FTP protocol has a number of inherent limitations, making this storage type a "last resort" for accessing certain remote data locations. If possible, you may achieve better performance and connectivity by using an alternative server, such as SFTP or the "Local Path" type over a mounted SMB or NFS share.

Configuration

The following configuration options are available:

Option	Requirement	Description
Address	Mandatory	The hostname for the FTP server. You may specify a port by adding a :21 suffix. The default port is 21.
Security	Mandatory	The FTP over TLS (FTPS) mode.
Username	Mandatory	The username to log in to the FTP server.
Password	Mandatory	The account password.
Use default home directory	Optional	Whether to store data in the FTP user's home directory.
Custom directory	Required only when Use default home directory is disabled	A remote path to store data in. Paths are relative unless anchored with a leading /.
Connections	Mandatory	The number of simultaneous FTP connections. The default connection limit is 5.
Accept invalid SSL	Optional	Whether to accept invalid SSL Certificate from the FTP server.

Simultaneous connections

The number of simultaneous connections can be configured for the FTP server (default 5).

If multiple operations are running simultaneously to an FTP Storage Vault in Comet Backup, the number of network connections may be higher.

SMB

Data will be stored on SMB shared storage such as a Windows network using the SMB protocol or on Linux via smbd.

The following configuration options are available:

Option	Requirement	Description
Server	Mandatory	The hostname or IP address of the SMB server. You may specify a port by adding a :445 suffix. The default port is :445
Share	Mandatory	The name of the SMB share.
Directory	Optional	The directory in the share to store data in. If left blank, data will be stored at the top level of the share.
Username	Mandatory	The username to log into the SMB server.
Password	Mandatory	The account password

Amazon S3

Data will be stored in an Amazon S3 bucket.

With Amazon S3, bucket names are globally unique across all of the standard regions, so it's not necessary to specify which region the bucket is contained in (e.g. us-east-1, eu-central-1) as this can be determined automatically.

If you specify a bucket name that does not exist, Comet will attempt to create it automatically in the default US East (N. Virginia) region.

It's possible to connect to Amazon S3 using the "S3-compatible" storage type, however, a dedicated option is available owing to the popularity of this service.

Comet supports Amazon S3 since the very first release.

Configuration

The following configuration options are available:

Option	Requirement	Description
Access key	Mandatory	The access key for the Amazon S3 account
Secret key	Mandatory	The secret key for the Amazon S3 account
Bucket	Mandatory	The globally unique name of the storage bucket
Subdirectory	Optional	A prefix to add to all stored blobs. This may be useful to isolate Comet's data within the bucket if the bucket is being shared with another application.

Simultaneous connections

Comet makes up to 10 network connections to the Amazon S3 server.

If multiple operations are running simultaneously to an Amazon S3 Storage Vault in Comet Backup, the number of network connections may be higher.

Amazon S3 (China)

Data will be stored in an Amazon China S3 bucket.

Most people have learned that an S3 bucket name must be unique across all regions in AWS, but this does not include the AWS China Region. Due to its special status you can have one bucket in the China region and one in a global region which both use the same name. The current avaiable Amazon China regions are China (Beijing) Isolated Region and China (Ningxia) Isolated Region.

It's possible to connect to Amazon China S3 using the "S3-compatible" storage type, however, a dedicated option is available owing to the popularity of this service.

Configuration

The following configuration options are available:

Option	Requirement	Description
Region	Mandatory	The endpoint for accessing Amazon China S3
Access key	Mandatory	The access key for the Amazon China S3 account
Secret key	Mandatory	The secret key for the Amazon China S3 account
Bucket	Mandatory	The globally unique name of the storage bucket
Subdirectory	Optional	A prefix to add to all stored blobs. This may be useful to isolate Comet's data within the bucket if the bucket is being shared with another application.

Simultaneous connections

Comet makes up to 10 network connections to the Amazon China S3 server.

If multiple operations are running simultaneously to an Amazon China S3 Storage Vault in Comet Backup, the number of network connections may be higher.

Google Cloud Storage

Google Cloud Storage is an S3-compatible storage product within the Google Cloud Platform.

It's possible to connect to Google Cloud Storage using the "S3-compatible" storage type, however, a dedicated option is available owing to the popularity of this service.

You can retrieve credentials under the "Interoperability" section of the Google Cloud Platform web interface.

Configuration

Option	Requirement	Description
Access key	Mandatory	The access key for the Google Cloud Platform account
Secret key	Mandatory	The secret key for the Google Cloud Platform account
Bucket	Mandatory	The globally unique name of the storage bucket
Subdirectory	Optional	A prefix to add to all stored blobs. This may be useful to isolate Comet's data within the bucket if the bucket is being shared with another application.

Simultaneous connections

Comet makes up to 10 network connections to the Google Cloud Storage server.

If multiple operations are running simultaneously to a Google Cloud Storage Storage Vault in Comet Backup, the number of network connections may be higher

S3-compatible

The protocol for object storage on Amazon S3 eventually became widespread enough to be called a pseudo-standard. A number of alternative storage providers offer S3-compatible object storage, in order to interoperate with the large body of available software and services surrounding this pseudo-standard.

A number of commercial storage providers offer an S3-compatible API. For more information about S3-compatible cloud storage providers, please see the "Cloud Storage Providers" document.

Alternatively, you can run your own S3-compatible storage server as a replacement for Storage Gateway. A number of server software packages are available for this purpose, such as

• Minio
• OpenStack Swift
• Ceph Object Store
• OpenIO
• LeoFS
• Pithos on Apache Cassandra
• Scality Zenko CloudServer (formerly Scality S3 Server)
• SeaweedFS
• Read more on Wikipedia

API Compatibility

Comet uses S3's streaming APIs to improve performance. You should ensure that any S3-compatible server implements streaming APIs in order to maintain compatibility.

In late August 2017, the S3-compatible storage provider Wasabi added support for S3's streaming APIs, and is now compatible with all versions of Comet Backup.

Configuration

Option	Requirement	Description
Hostname	Mandatory	The hostname for the S3-compatible server.
Region	Optional	Custom region for the S3 endpoint.
Access key	Mandatory	The access key for your user account
Secret key	Mandatory	The secret key for your user account
Bucket	Mandatory	The bucket in which to store all data
Subdirectory	Optional	A prefix to apply to all objects stored in the bucket
Use encrypted transmission	Optional	Whether to access the S3-compatible server using the HTTPS protocol. Data is encrypted and authenticated regardless of whether HTTPS is in use.

Simultaneous connections

Comet makes up to 10 network connections to the S3-compatible Storage server.

If multiple operations are running simultaneously to an S3-compatible Storage Vault in Comet Backup, the number of network connections may be higher.

• Custom IAM-Compatible

  IAM is a protocol used to provision new storage account credentials. It was created by Amazon AWS but is now widely used by other providers to manage authentication and authorization. If you have a storage provider that supports the IAM API, you can use this option to create a Storage Template in Comet.

  Configuration

  Option	Requirement	Description
  S3 endpoint	Mandatory	The base URL for the custom S3-compatible API. If unsure, please reach out to your storage provider for assistance.
  IAM enpoint	Mandatory	The base URL for the custom IAM-compatible API. If unsure, please reach out to your storage provider for assistance.
  Region	Optional	Custom region for the S3 endpoint.
  Bucket Name	Optional	The bucket in which to store all data. If present, Comet will create private subdirectories within this bucket otherwise Comet will create separate buckets.
  Access Key	Mandatory	The access key for your user account.
  Secret Key	Mandatory	The secret key for your user account.
  Prefix	Optional	A prefix to apply to all objects stored in the bucket.

Wasabi

Wasabi Cloud Storage is an S3-compatible storage product.

It's possible to connect to Wasabi Cloud Storage using the "S3-compatible" storage type, however, a dedicated option is available owing to the popularity of this service.

You can retrieve your access keys under the hamburger button on the top left section and then clicking on Access Keys

Configuration

Option	Requirement	Description
Access key	Mandatory	The access key for the Wasabi Cloud Storage account
Secret key	Mandatory	The secret key for the Wasabi Cloud Storage account
Bucket	Mandatory	The globally unique name of the storage bucket
Subdirectory	Optional	A prefix to add to all stored blobs. This may be useful to isolate Comet's data within the bucket if the bucket is being shared with another application.

Simultaneous connections

Comet makes up to 10 network connections to the Wasabi Cloud Storage server.

If multiple operations are running simultaneously to a Wasabi Cloud Storage Storage Vault in Comet Backup, the number of network connections may be higher.

Immutability / Anti-Ransomware

If malicious software gains access to the access key and secret key used by Comet, it would be able to delete files within the Wasabi storage location. Comet is not able to defend against misuse of these credentials - however, the Wasabi storage platform does offers multiple features for defending against this:

Object Locking

Comet supports Object Locking on Wasabi using Compliance mode. Refer to our guide on S3 Object Locking.

Bucket Versioning

Comet does support Bucket Versioning. It is safe to enable this feature in Wasabi for use with Comet. No additional configuration is required within Comet itself.

When you enable Bucket Versioning for a Wasabi bucket, each file stored in that bucket is redefined to comprise of a stack of versions. In this mode, when Comet (or any malicious software using Comet's access key / secret key pair) goes to delete a file, then instead of it being actually deleted, a "delete marker" or "tombstone version" is placed on top of the file's version stack.

A "tombstoned" file generally seems to have been deleted; it is not available via regular use of the access key. The file will appear to be missing when viewing a subdirectory and will not be available for download. However, such files can later be undeleted by removing the tombstone from on top of the version stack.

When using Bucket Versioning there are some further key points to be aware of:

• Cost
  • Wasabi will continue to charge you for the ongoing storage of deleted files, until the entire version-stack is deleted.
• Regaining access to deleted files
  • You can regain access to deleted files by removing the "tombstone" from each of the version stacks.
  • Because Comet never modifies a file after creation, one way to achieve this is to delete the top-most version from all files that have multiple versions in their version stack. This may cause some excess files to reappear, but a single retention pass and/or Unlock operation should resolve any issues.
  • Another way to achieve this is to use timestamp metadata on each files' version stack to roll back the versioned-bucket to a specific point in time.
  • This operation can be performed in bulk from the command-line via an S3 command-line tool such as Minio Client, awscli, or s3cmd: e.g.
    • mc rm --recursive --rewind "2021.11.29" (Read more)
• Lifecycle policy
  • To avoid ever-increasing costs, it is important to periodically delete tombstoned files in order to reduce costs. For instance, you may want to fully delete a file from Wasabi 90 days after it has been tombstoned.
    • This extra time period is in addition to the retention policy configured within Comet itself.
  • (As of November 2021) Wasabi does not currently have built-in retention settings for the Bucket Versioning feature. It does have built-in retention settings for the Compliance Mode feature and we do expect this to become available for Bucket Versioning in the near future.
  • Until that time, you can periodically prune tombstoned files either
    • Manually, by following Wasabi's instructions at https://wasabi-support.zendesk.com/hc/en-us/articles/360042534431-How-do-I-delete-old-object-versions- ; or
    • In bulk, by running Wasabi's example python script at https://wasabi-support.zendesk.com/hc/en-us/articles/360045760851-How-do-I-mass-delete-current-and-non-current-objects-inside-a-bucket-
• Permissions
  • The additional security of this model is predicated on Comet's access key / secret key not having permission to delete entire version stacks (s3:DeleteObjectVersion), and only being able to tombstone a file (s3:DeleteObject).
  • If you use the Storage Template feature in Comet Management Console to automatically provision new access key / secret key pairs for each customer, they only have the more restricted s3:DeleteObject API available. For a versioning-enabled bucket, this means that deletions will only tombstone a file instead of deleting the entire version stack.
  • If you use the root Wasabi key, it has permission to delete entire version stacks. Although Comet will still only call the s3:DeleteObject API, to tombstone a file instead of deleting it, any malicious user who gains access to these keys would still be able to delete the entire version stack via the broader s3:DeleteObjectVersion API, meaning that you do not gain any additional security from enabling this feature.
  • If you are declaring a custom IAM policy document, please consider taking the above detail into account in the Action array.

Backblaze B2 Cloud Storage

B2 is a low-cost storage offering from Backblaze.

BackBlaze B2 supports multiple Application Keys for different access permission to B2 Buckets. You can configure Comet to use either the Master Application Key, or a per-bucket Application Key.

Configuration

Option	Requirement	Description
Application key ID / Account ID	Mandatory	If you are using the Master Application Key, enter the Account ID for your Backblaze B2 account here. For a per-bucket Application Key, enter the Application Key ID.
Application key	Mandatory	Enter your Master Application Key or for per-bucket, enter the Application key (this is shown when creating the Application key).
Bucket name	Mandatory	The bucket in which to store all data
Subdirectory	Optional	A prefix to apply to all objects stored in the bucket
Hide files rather than delete them	Optional	When enabled, Comet will hide files rather than deleting them from the bucket when deleting old backups. This setting should only be enabled alongside bucket lifecycle rules which remove hidden files after a certain length of time; see Hiding deleted files for more information.

Alternative connectivity

As an alternative, you can also access Backblaze B2 Cloud Storage using the "S3-compatible" storage type if you run a S3-to-B2 gateway server, such as Minio.

Hiding deleted files

Comet supports hiding files which would be deleted from a B2 bucket rather than deleting them. This setting provides improved protection against ransomware; when this setting is enabled, Comet can use keys which do not have the B2 deleteFiles capability (see Application Keys) and thus cannot delete files from the bucket, preventing a compromised key from being used to delete backed up data. While this setting can be enabled and used with a key that does have the deleteFiles capability, it does not offer improved protection against ransomware in this configuration, as if compromised the key can still be used to delete files.

This setting should only be enabled alongside bucket lifecycle rules which remove hidden files after a certain length of time. If no such rule is enabled, Comet will not delete old files and the bucket size can grow indefinitely. See the B2 Lifecycle Rules documentation for more information on setting up appropriate rules.

When this setting is enabled, Storage Templates will automatically generate keys which do not have the deleteFiles capability. It is safe to toggle this setting on or off for an existing Storage Template; future keys generated by the Storage Template will reflect the changed setting, while pre-existing provisioned Storage Vaults and keys will be unaffected and will continue to use the setting value from the time at which the Storage Vault was provisioned.

Comet does not automatically detect the capabilities of the key that is used for a storage vault. If a storage vault that uses a key without the deleteFiles capability has this option toggled off, it will attempt and fail to delete files in future operations. To remedy this, when this option is toggled off the vault should also be updated to use a key with the deleteFiles capability.

Backblaze B2 Fireball

To perform a backup to the Backblaze Fireball, connect your Backblaze Fireball to your network or computer and ensure the SFTP is enabled.

On the Comet client in the Account tab, select + Add Storage Vault and click Custom.

Create a name and in the Storage tab, select the SFTP option. Enter the Fireball’s address, path, username and password (change Authentication to Password).

Now you can backup to Backblaze Fireball.

After you have backed up the data to the Fireball, follow Backblaze’s instructions to courier the Fireball to their datacentre. You will then need to update the storage vault details so that it points to the Backblaze B2 bucket.

Azure Blob Storage

Azure Blob Storage is a public-cloud object storage service operated by Microsoft under the Azure branding.

Compatible implementations

Comet's support for Azure Blob Storage can also be used to connect to third-party storage services that are compatible with the Azure Blob Storage API.

You can enter a custom "Realm Address" in the configuration to use a third-party storage service.

Comet is compatible with storage providers based on

• Microsoft Azure Stack
• Azurite
• Microsoft Azure Storage Emulator (not recommended for production data storage)

Alternative connectivity

You can access Azure Blob Storage using the "S3-compatible" storage type if you run a S3-to-Azure gateway server, such as Minio.

Configuration

Option	Requirement	Description
Realm address	Mandatory	The hostname or URL for the Azure Blob Storage realm (usually core.windows.net). Use a scheme-qualified URI (e.g. http://) in order to toggle TLS.
Account name	Mandatory	The name of your user account
Account key	Mandatory	The key for your user account
Container	Mandatory	The container in which to store all data
Subdirectory	Optional	A prefix to apply to all objects stored in the container

Simultaneous connections

During a job, Comet Backup performs up to 10 simultaneous actions upon an Azure Blob Storage destination.

If multiple operations are running simultaneously to a Azure Blob Storage -based Storage Vault in Comet Backup, the number of network connections may be higher.

Filebase

Filebase Cloud Storage is an S3-compatible storage product powered by Blockchain.

It's possible to connect to Filebase using the "S3-compatible" storage type, however, a dedicated option is available owing to the popularity of this service.

To perform a backup to the Filebase, on the Comet client in the Account tab, select + Add Storage Vault and click Custom. Create a name and in the Storage tab, click the + button.

Select Filebase option and click OK button.

Enter the Filebase Access key, Secret key and Bucket. You can test the connectivity to Filebase storage by clicking the Test Connection button.

You can retrieve your access keys by clicking on Access Keys on the left section

Configuration

Option	Requirement	Description
Access key	Mandatory	The key for the Filebase Cloud Storage account
Secret key	Mandatory	The secret for the Filebase Cloud Storage account
Bucket	Mandatory	The globally unique name of the storage bucket
Subdirectory	Optional	A prefix to add to all stored blobs. This may be useful to isolate Comet's data within the bucket if the bucket is being shared with another application.

Simultaneous connections

Comet makes up to 10 network connections to the Filebase Cloud Storage server.

If multiple operations are running simultaneously to a Filebase Cloud Storage Storage Vault in Comet Backup, the number of network connections may be higher.

IDrive e2

IDrive e2 Cloud Storage is a S3-compatible storage product.

It's possible to connect to IDrive e2 using the "S3-compatible" storage type, however, a dedicated option is available owing to the popularity of this service.

Access keys can be retrieved by navigating Access Keys on the left menu and then clicking on Create Access Key

Configuration

Option	Requirement	Description
Access key	Mandatory	The access key id for the IDrive e2 Cloud Storage account
Secret key	Mandatory	The secret access key for the IDrive e2 Cloud Storage account
Bucket	Mandatory	The globally unique name of the storage bucket
Subdirectory	Optional	A prefix to add to all stored blobs. This may be useful to isolate Comet's data within the bucket if the bucket is being shared with another application.

As Storage template

IDrive e2 can be configured as a Storage Template. Storage Vaults created from an IDrive e2 Storage Template will create private, separate credentials and a private bucket within the same IDrive e2 account.

However, IDrive with object lock enabled cannot be configured as a Storage Template and will have to be configured manually as a Storage Vault instead.

Option	Requirement	Description
Access key	Mandatory	The access key id with at least write access across all buckets in account
Secret key	Mandatory	The secret access key associated with the access key

For Cluster Manager

The IDrive e2 credentials entered for Cluster Manager will be used to clean up any unreferenced buckets and account keys previously used in a user's storage vault. This does not recognize IDrive e2 buckets behind a Storage Gateway. It is advised to have a separate account for user buckets and another for the Storage Gateway to avoid Cluster Manager deleting buckets used by the Storage Gateway.

Impossible Cloud

Impossible Cloud Storage is the decentralized, enterprise-grade cloud storage solution based on nextgeneration web3 technology for hyperscale edge performance, industry-leading ransomware protection, and up to 80% savings on total cloud spend.

With full S3 compatibility, Impossible Cloud Storage is very easy to set up with Comet Backup. To access Impossible Cloud Storage, you utilize the S3-compatible storage type.

Obtain your access keys

Navigate to the Impossible Cloud console's 'keys' section, as shown below, to obtain your access keys. You will find the relevant section of the Impossible Cloud documentation for further details regarding keys by following this link.

Configuration

Option	Requirement	Description
Access key	Mandatory	The access key for the Impossible Cloud Storage user
Secret Key	Mandatory	The secret key for the Impossible Cloud Storage user
Bucket	Mandatory	The unique name of the storage bucket
Folder	Optional	A prefix to add to all stored backups. This may be useful to isolate Comet's data within the bucket if the bucket is being shared with another application.

Simultaneous connections

With Comet Backup, it is possible to establish up to 10 network connections to the backend of Impossible Cloud Storage. In cases where multiple operations are executed simultaneously on a Storage Vault in Comet Backup, the number of network connections may increase.

Immutability / Anti-Ransomware

Impossible Cloud Storage provides Object Lock to safeguard against ransomware. You can customize the retention policy according to your specific requirements, as shown below.

Further recommendation on features

Enabling Object Lock

Comet Backup supports Object Lock for Impossible Cloud Storage, and the guide on S3 Object Lock provides you with further assistance.

Enabling Versioning

The Versioning of Impossible Cloud Storage is supported by Comet Backup. You have to set one click on Impossible Cloud Storage to enable it. No further configuration is necessary within Comet Backup. With Versioning, each file stored in that bucket is transformed into a stack of versions.

Compliance Mode

Comet Backup is incompatible with the Impossible Cloud Compliance Mode due to its need for read/write access to backups. Therefore, enabling Compliance Mode, even if later disabled, may result in undesired outcomes and is not recommended.

Storj DCS

Storj DCS is a decentralized cloud storage which provides a low-cost object storage service. Comet could use Storj as an S3-compatible service, but this option uses the native Storj API to transfer data from a client device to satellites.

Access keys could be generated by navigating Access on the left hand side menu, and then clicking on Create Keys for CLI on the right hand side.

Configuration

Option	Requirement	Description
Satellite	Mandatory	Choose a satellite node. A satellite region is shown at the corner of the login promt of the Storj website.
API key	Mandatory	Use API key to access bucket
Password	Mandatory	Bucket password
Bucket	Mandatory	A bucket to store data
Prefix	Optional	A bucket prefix. Multiple users’ data could be saved in a bucket with different prefixes.

As Storage template

Storj DCS could be configured with storage template. Each storage vault is created under Bucket with a system-generated bucket prefix.

Option	Requirement	Description
Satellite	Mandatory	Choose a satellite node. A satellite region is shown at the corner of the login promt of the Storj website.
API key	Mandatory	Use API key to access bucket
Bucket	Mandatory	A bucket name
Password	Mandatory	A password - used for all bucket prefix

Storadera

Storadera Cloud Storage is an S3-compatible storage product.

It's possible to connect to Storadera using the "S3-compatible" storage type, however, a dedicated option is available owing to the popularity of this service.

You can retrieve your access keys by navigating Keys on the top section and then clicking on Create.

Configuration

Option	Requirement	Description
Access key	Mandatory	The access key id for the Storadera Cloud Storage account
Secret key	Mandatory	The secret access key for the Storadera Cloud Storage account
Bucket	Mandatory	The globally unique name of the storage bucket
Subdirectory	Optional	A prefix to add to all stored blobs. This may be useful to isolate Comet's data within the bucket if the bucket is being shared with another application.

Simultaneous connections

Comet makes up to 10 network connections to the Storadera Cloud Storage server.

If multiple operations are running simultaneously to a Storadera Cloud Storage Storage Vault in Comet Backup, the number of network connections may be higher.

Digital Ocean Spaces

Digital Ocean Spaces is an S3-compatible storage product with a built-in CDN. The current avaiable Digital Ocean Spaces regions are San Francisco Region, Frankfurt Region, Amsterdam Region, New York Region and Singapore Region.

It's possible to connect to Digital Ocean Spaces using the "S3-compatible" storage type, however, a dedicated option is available owing to the popularity of this service.

You can retrieve your access keys by navigating API on the left section and then clicking on Generate New Key on the Space access keys section.

Configuration

Option	Requirement	Description
Access key	Mandatory	The key for the Digital Ocean Spaces
Secret key	Mandatory	The secret for the Digital Ocean Spaces
Bucket	Mandatory	The globally unique name of the storage bucket
Subdirectory	Optional	A prefix to add to all stored blobs. This may be useful to isolate Comet's data within the bucket if the bucket is being shared with another application.

Simultaneous connections

Comet makes up to 10 network connections to the Digital Ocean Spaces server.

If multiple operations are running simultaneously to a Digital Ocean Storage Vault in Comet Backup, the number of network connections may be higher.

Catalyst Cloud Object Storage

Catalyst Cloud Object Storage is an S3-compatible storage product based in New Zealand. The Catalyst Cloud has three regions (separate geographical locations) that can operate independently of each other. Each region provides local redundancy and high availability for increased service levels. The current avaiable regions are New Zealand Hamilton Region, New Zealand Porirua Region and New Zealand Wellington Region.

It's possible to connect to Catalyst Cloud Object Storage using the "S3-compatible" storage type, however, a dedicated option is also available.

You can retrieve your access keys by navigating API Access on the left section and then clicking on Create EC2 Credentials.

Configuration

Option	Requirement	Description
Access key	Mandatory	The EC2 Access Key for the Catalyst Cloud account
Secret key	Mandatory	The EC2 Secret Key for the Catalyst Cloud account
Bucket	Mandatory	The globally unique name of the storage bucket
Subdirectory	Optional	A prefix to add to all stored blobs. This may be useful to isolate Comet's data within the bucket if the bucket is being shared with another application.

Simultaneous connections

Comet makes up to 10 network connections to the Catalyst Cloud server.

If multiple operations are running simultaneously to a Catalyst Cloud Storage Vault in Comet Backup, the number of network connections may be higher.

OpenStack Swift

OpenStack Swift is an open-source data storage system that can be implemented by a storage provider. A number of commercial storage providers using this system are listed on the "Cloud Storage Providers" document.

Comet's integration with OpenStack Swift also supports compatible implementations of the Swift protocol, such as Ceph Object Store.

Configuration

Your connection to OpenStack is authenticated using one of the following methods:

• Identity API v1 is a basic system available from all OpenStack providers.
• Identity API v2 introduced "Tenants", to isolate multiple users.
• Identity API v3 introduced "domains", as another layer of isolation. Additionally, Tenants were renamed to "Projects".

Comet automatically determines the API version to use based on the URL set in the "Auth Endpoint" field. For security reasons, we recommend using the highest possible API version that is supported by both Comet and your OpenStack provider.

Option	Requirement	Description
Auth Endpoint	Mandatory	The URL for the OpenStack Auth server, including protocol (http:// or https://). These URLs commonly end in v1.0 or v2.0.
Username	Mandatory	The username for your account.
API Key	Mandatory	The API key for your user account.
Container	Mandatory	The container in which to store all data. It will be created if it does not yet exist.
Subdirectory	Optional	A prefix to apply to all objects stored in the container
Domain Name	Optional	The domain name. Applies to v3 only
Region	Optional	The region to use (e.g. LON or ORD). If not specified, a region may be automatically selected. Applies to v2 / v3
Tenant name	Optional	Name of the tenant/project. Applies to v2 / v3
Tenant ID	Optional	The ID of the tenant/project. Applies to v2 / v3
Tenant Domain	Optional	Name of the tenant/project's domain, if it differs from the user domain. Applies to v3 only
Trust ID	Optional	The ID of the trust Applies to v3 only
Container Policy	Optional	The policy to use when creating the container, if it does not yet exist.

Simultaneous connections

Comet makes up to 10 network connections to an OpenStack Swift server.

If multiple operations are running simultaneously to an OpenStack Swift Storage Vault in Comet Backup, the number of network connections may be higher.

WebDAV

Data will be stored on a WebDAV server, such as Nextcloud or Apache.

Configuration

The following configuration options are available:

Option	Requirement	Description
Hostname	Mandatory	The URL of the WebDAV server, including protocol (http:// or https://) and any custom port.
Username	Mandatory	The username for logging in to the WebDAV server.
Password	Mandatory	The password for logging in to the WebDAV server.
Path	Optional	The target directory path within the WebDAV server.

Simultaneous connections

Comet makes up to 10 network connections to a WebDAV server.

If multiple operations are running simultaneously to an WebDAV Storage Vault in Comet Backup, the number of network connections may be higher.

Spanned

The 'Spanned' storage location is a virtual location that pools disk space from multiple other storage locations. It is implemented similarly to a union filesystem, where all locations are viewed as one. Data files can be spread between all locations.

File placement strategy

Multiple file placement strategies are available in Comet.

Default (Automatic)

If multiple storage volumes are configured, then new data will be automatically written in a balanced way between applicable drives.

Comet must determine which volume to write to. New files are placed on one of the constituent targets in a way that is intended to balance the priorities of (A) storage space, (B) keeping disks busy, and (C) minimizing overheads associated with measuring disk free space and disk IOPS.

At first, when all volumes have a lot of free space, writes are randomly distributed evenly between all storage volumes. This strategy means that disk space is used roughly efficiently across all volumes, and all drives will simultaneously serve the writing load on the server, providing the maximum benefit.

Once any volume has less than a certain amount of free space, Comet gradually switches to a different strategy, in which it writes solely to a particular volumes with the most free space available. This means that the remaining disk space will be used efficiently, although only one drive at a time will service the writing load (all drives will still service the reading load).

It is not generally feasible to determine the free space available on a cloud storage provider. When spanning between local disks and cloud storage providers, cloud storage providers are preferred.

When using the Default file placement method, you can add a new spanned drive location at any time. This has the effect of increasing the space available for storage.

There is zero overhead when using the Spanned type with one single location. There is a small overhead when using multiple locations, as Comet must check each location to find any requested file.

Static locations

The default automatic Spanned file placement strategy puts files in any location that has free space. That means when Comet needs to get a file back, it has to check in all locations to find it, reducing performance.

"Static locations" is an alternative file placement strategy where files are consistently placed in one of the span targets based on the file name. Comet will immediately know where to look for any given filename.

The placement algorithm hashes the requested filename and dividing its integer hash between the number of total targets.

This has several limitations:

• This is only feasible for brand-new Span targets, since if you have an existing one, the files will be in any of the constituent targets, instead of exactly where Comet expects them
• It is not safe to add additional targets
• It is not safe to reorder the targets
• The targets should also all have roughly the same total free space, for the same reasons

If you are creating a brand new Spanned storage location, and you can abide by the above restrictions, this may provide higher performance.

If at any time you are no longer able to abide by the above restrictions (e.g. need to expand storage to an additional location) it is always safe to convert from "Static locations" into "Default" file placement, but not vice versa.

Using "Spanned" with Comet Storaage Gatewy

When using the Spanned type as a storage location for a Storage Gateway, this allows your Storage Gateway to virtually offer data that is hosted on a remote server (e.g. Amazon S3).

Cloud Failover

When using the Spanned type with Comet Storage Gateway, you can achieve a best-performance scenario by configuring a Spanned storage with only a Local Path location, and then expanding to a cloud storage provider once local resources are exhausted. (This requires the "Default" file placement strategy.)

Simultaneous connections

Each constituent data location that comprises the Spanned data location is subject to its individual connection limit. There is no additional connection limit for Spanned data locations.

Rebalancing

The "Spanned" virtual storage location does not actively rebalance data between the underlying locations.

However, some rebalancing will happen passively, as retention passes will delete files from the old unbalanced layout, and new data is always written in a balanced way.

If you are using the "Default" file placement strategy, you may manually rebalance the constituent storage locations by moving files between storage locations. This operation is safe to perform live while Comet Storage Gateway is running.

If you are using the "Static locations" file placement strategy, you may not rebalance the constituent storage locations.

Expanding available storage space by changing Storage Type to Span

Whilst the principle of adding a new Span Target to increase storage space already applies to all storage types, this guide will address the straight-forward addition of a single new hard drive to expand the storage of a Comet Storage Gateway.

In the following example, Storage Gateway is enabled, and currently stores data on an almost-full hard drive, which shows as "D:". A new hard drive has been attached to the device running Storage Gateway, and this new drive shows as "E:".

The steps to add this additional storage to the Comet Storage Gateway so as to expand its storage space are as follows:

• Open the Settings page to the Storage Gateway > Locations section
• Choose where data will be stored will show the current type of storage, as well as the physical storage location.
• In this example, the type is Local Path and the path is "D:".
• Change the type from Local Path to Spanned.
• Add a Span Target, using the existing location used previously, i.e.: Local Path and "D:".
• Add a new Span Target, using the new drive and location, e.g: Local Path and "E:"
• Leave the File Placement Strategy as Default, as "D:" already has data stored on it.
• Click 'Save' at the top of the Settings page, and the Comet Storage Gateway will restart.

If, after applying the changes, you experience issues with accessing your Comet Storage Gateway, check your system logs for any errors related to storage, and edit the storage settings through the Comet Core Service Manager, or config file. It is possible to change from Spanned to a different storage type, such as Local Path, however, this is a potentially lengthy and complex process. Contact Support for more details.

Unsupported storage locations

Amazon GovCloud

It is possible to configure Amazon GovCloud as a storage location but Comet is not able to provide support for Amazon GovCloud at this time.