Skip to main content

Troubleshooting

Error "local error: tls: record overflow"

This message means the connection was corrupted over the network, and Comet's network library closed the connection.

This can happen because of random network conditions. Retrying the operation should fix the issue.

If the issue keeps happening repeatedly, this message indicates that something is interfering with packets in your network.

  • Failing NIC
  • Bad NIC driver or driver configuration
  • Failing RAM, on either the endpoint machine or any of the intermediate routers
  • Outdated firewall or proxy, performing incorrect SSL interception

For more information, please see the record_overflow section in IETF RFC 5246.

"Paused" state on Windows service

Comet Server on Windows consists of two parts: cometd.exe and cometd-service.exe. The latter binary is registered as a service with Windows, and is responsible for ensuring that the former binary stays running.

If Comet Server encounters an error and closes, the service binary will restart it. If the Comet Server is repeatedly unable to start - if it closes immediately when launched, several hundred times consecutively - then the service manager will assume the error is permanent and abandon restarting the process. This condition is displayed as the "Paused" state.

You can resolve this issue by resolving the underlying issue with the service. The error message should be recorded in Comet Server's log file.

You can view Comet Server's log files

  • in Comet Server Service Manager, use the Service menu > "Browse log files" option, or
  • by browsing the C:\ProgramData\Comet\logs directory.

Comet Server makes one log file per day. The error should be recorded in the most recent file (highest number in name / latest modification date).

Forgotten administrator password

If you are locked out of the Comet Server web interface, you can change your administrator password by editing the cometd.cfg file.

  1. Stop the server, and edit the cometd.cfg file
  2. Find the AdminUsers section for the administrator user in question
  3. Set "PasswordFormat" to 0
  4. Set "Password" to e.g. "admin"
  5. Save the file and restart the server.

You should now be able to log in with the reset password. The password will be hashed and/or encrypted after first use.

Storage server "127.0.0.1" in use by accounts but not managed by Constellation

A user account has a bucket on a Storage Role Comet Server at the address 127.0.0.1, but this address was not selected for management by Constellation. Aside from the standard warnings about managing Constellation, the 127.0.0.1 is a special case.

If Comet Server is configured to listen on all network interfaces, the server will be accessible on both 127.0.0.1 as well as LAN, WAN, or DNS addresses. If you log in to either the Comet Backup application or the Comet Server web interface at 127.0.0.1, and request a new Storage Vault from a Comet Server configured as "Local" (or $self$ in cometd.cfg), the resulting Storage Vault will be configured using 127.0.0.1 as the remote network address.

However, 127.0.0.1 has a different meaning depending on where it is found. The connection will fail if the Comet Backup client is not running on the same machine. There may also be unintended consequences if the account is replicated to another server.

To avoid this problem, either

  • always use the external DNS name for your Comet Server when requesting new Storage Vaults, or
  • change the Request destination to use "Remote" with an external DNS name instead of "Local" (or $self$ in cometd.cfg).

A similar caveat applies to the software downloads, which can include embedded server address details.

Microsoft SQL Server backup encountered a VDI error

You should ensure that the necessary VDI .dll files are registered and are the correct version for your SQL Server installation. You can use Microsoft SQL Server Backup Simulator to check the status of the VDI .dll files.

Error "Access is denied" when backing up files and folders on Windows

An "Access Denied" error message means that the Windows user account running the backup job does not have access to read the file content.

Current versions of Comet automatically creates a service account with all necessary permissions to read local files. If you are experiencing "Access Denied" errors, you may be trying to back up a network path that has been mounted as a directory. Please see the Accessing Windows network shares and UNC paths section below for more information. If you are experiencing "Access Denied" errors and you are certain that you are not backing up a mounted network path, please contact support. You may also be trying to backup a local Protected Item to which the backup software does not have read-access, such as a file within a user's home directory. In this case, you will need to alter the backup delegate service, and change it to 'logon as' an account which does have access to the Protected Item.

Antivirus detects Comet Backup as a virus or malware

Comet Backup is a safe application. Any such detection is a "false positive".

When Comet Backup is rebranded, it might seem like a new, unknown program. An unknown program that installs system services, accesses files on the disk and uploads them to the network, might be considered to be malware if it was installed without consent. Unfortunately it's understandable for an Antivirus product to detect this.

In this situation, there are some actions you can take:

  • Please ensure your Antivirus product is fully up-to-date.
  • Please contact Comet Support with a screenshot of the error message. In some situations, it may be possible for our developers to resolve the issue.
  • Choose to "allow" or "white-list" the file in the Antivirus software. This may send a signal to the Antivirus software vendor that the software is safe (e.g. ESET LiveGrid, Windows Defender Automatic Sample Submission, Kaspersky KSN, etc).
  • Enable Authenticode signing on Windows. This may provide additional "reputation" to the software installer.

The client software internal binaries are signed by Comet Backup (Software NZ Limited). The installer is signed by the same certificate if the Pre-built software client option is used, by the certificate configured in the Comet Server's client branding settings, or unsigned if custom branding is used with no certificate. Any digital signature-based exclusion should be include the internal Software NZ Limited certificate and any added in a custom configuration.

Error "backup-tool.exe couldn't be launched. CreateProcess() failed: Access is denied" message

This error message indicates that something on the PC is blocking Comet's main backup-tool.exe program from running. It's likely this is the antivirus. Please follow the above steps to allowlist Comet in your antivirus application.

Avast "FileRepMalware"

You will receive the "FileRepMalware" error message for any file that was

  • downloaded from the internet; and
  • does not have an Authenticode certificate; and
  • has not been seen yet by many Avast users.

Many custom-branded Comet Backup installers do fall into this category.

You can resolve this issue by using the pre-built client software (signed by Comet Backup) or by purchasing and installing an Authenticode certificate.

Bitdefender

Bitdefender may flag the desktop client software, you can create an exclusion policy on Bitdefender to prevent this from happening.

Installation files exclusions

For the client installation files, you may or may not be required to install from the root directory. You can simply create a folder on each machine to put the installation files into.

Exclusions should should be configured based on the digital signature or certificate hash of the software. https://www.bitdefender.com/business/support/en/77209-342987-exclusions.html#UUID-3ae96d1a-424c-6eec-0c2d-85aa0b1966c3

To find the certificate hash, open the Comet Backup installer properties -> Digital Signatures tab -> Details -> View Certificate -> Details tab -> Thumbprint.

If your installer does not have a Digital Signatures tab, then the software is branded and unsigned. Use the pre-built client or configure codesigning for compatibility with BitDefender.

Excluding by file path

BitDefender also allows path and file extension exclusions. They are not recommended as part of best practices.

Product name compatibility

The use of an exclamation mark in your product name can cause Comet to be blocked by BitDefender. The exclamation mark must be removed to resolve this issue.

SentinelOne

Comet is highly distrusted by SentinelOne by default. Exclusions can be made based on the certificate.

Create a new exclusion, set the exclusion type to "certificate" and enter your Authenticode certificate or the digital signature thumbprint of the installer with default branding.

To find the certificate hash, open the Comet Backup installer properties -> Digital Signatures tab -> Details -> View Certificate -> Details tab -> Thumbprint.

If your installer does not have a Digital Signatures tab, then the software is branded and unsigned. Use the pre-built client or configure codesigning for compatibility with SentinelOne.

Watchguard EPDR

Comet is reportedly blocked by Watchguard EPDR's code injection detection. If Comet's services are regularly disabled, check if this feature is enabled. Try adding an exclusion for backup-tool.exe, or disabling the feature entirely if the issues persist.

Network connectivity errors

Comet Backup uploads files to Comet Server (or to a cloud storage provider) over the internet. Occasionally, you may see errors such as the following:

  • Couldn't save data chunk:
  • HTTP/1.x transport connection broken
  • net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  • wsarecv
  • wsasend
  • An existing connection was forcibly closed by the remote host
  • dial tcp: lookup [...]: no such host
  • connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

Comet Backup retries the upload several times, but eventually gives up. After a failed data chunk upload, you may see several more messages of the form Couldn't save data chunk: context canceled while Comet terminates the other upload threads.

Network errors have many possible causes:

  • Customer's PC
  • Customer's network
  • Customer's ISP
  • Internet-wide outages between customer's ISP and your ISP
  • Your ISP
  • Your network
  • Your Comet Server hardware
  • Comet Server software

To troubleshoot these issues, please check:

  • Does the backup succeed if it is retried?

    • Many network errors are temporary and will only occur rarely. In addition, a repeated second backup job will often run faster because many of the existing data chunks have already been uploaded. (Any unused data chunks in the Storage Vault will be automatically cleaned up by the next retention pass.)

  • Does the error message always happen at a certain time of day?

    • It may be possible to reschedule the backup to avoid times of heavy internet congestion.

  • Are there any corresponding messages for around the same time in your Comet Server logs?

    • This is important to determine the cause of some failures.
    • Some relevant Comet Server log messages take the form Error saving upload stream or Blocking re-upload of preexisting file

Accessing Windows network shares and UNC paths

This section applies to both Comet Server and Comet Backup.

Comet can back up Windows network paths, and back up to Windows network storage (SMB / CIFS). However, because Comet runs as a service user, there are some issues with authentication to be aware of.

Please note that if you are using Comet Backup to back up data from a network device, you should prefer to install Comet Backup directly on the device instead of backing it up over the network. This will also significantly improve performance, as less data needs to be transferred over the LAN.

Mapped network drives

On Windows, each logged-on user session has its own set of mapped network drives. The service user account is unlikely to have any mapped drives. If you see error messages like WARNING Missing: 'Z:\', this is probably the reason. You can work around this by using a UNC path instead.

The Comet Backup desktop application will automatically convert mapped network drives to their UNC path equivalents.

Authentication

If the UNC share requires authentication, the service user account is probably not logged-in to the UNC share. If you see error messages like WARNING Lstat: CreateFile \\?\UNC\...: Access is denied., this is probably the reason.

Comet has a built-in option for setting Windows network authentication credentials.

Error "A specified logon session does not exist...

Error "A specified logon session does not exist. It may already have been terminated" when accessing a Windows network share (SMB)

Some SMB servers don't seem to accept multiple SMB login sessions, if they use the same SMB credentials from the same host from different Windows user accounts.

On Windows, each user session has its own set of network login sessions. Comet performs the backup using a service user account. Therefore, this issue could occur with an affected SMB server if the interactive Windows user was logged in to the network share simultaneously.

Some backup types make multiple connections to run at a higher privilege level (Disk Image, SQL Server, Hyper-V), and may trigger this error. The workaround for these backup types is to use an alternative protocol like S3, SFTP or WebDAV.

Incorrect credentials may have been entered, or may be required in another format by the SMB server. Try entering the username as hostname\username. This requirement is common on Synology NAS devices.

Verifying the issue

The "Run as Administrator" session also has its own separate network login sessions.

  • Are you able to browse the network share from both an elevated and unelevated application? (e.g. "File > Open" from notepad and from a notepad launched with "Run as Administrator")

The interactive user account may have an open network session to the affected device.

  • From a command prompt, can you run net use to see if the interactive user is logged in to the network share?

  • From a command prompt, if you run net use \\server_name\share_name /DELETE to log the interactive user out of the network share, does this allow the backup to proceed?

    • Note that this may cause the same error to affect the interactive user. You may have to run this same command as an After command in the backup job, to log the background service account out of the network share, so that the interactive user can log back in.

Workarounds

If you are backing up from the SMB server, you could work around this issue by

  • installing Comet Backup directly on the network device (e.g. for Synology, use the Synology SPK installer), to back up the files directly. This may have significantly better performance, as less data needs to travel across the LAN and the disk access latency is significantly improved.

Alternatively, by

  • logging the interactive user account out of Windows entirely before the job starts. This will close their network sessions. Comet will successfully log in to the network share; you should add net use /DELETE as an After command in Comet to ensure the service user logs out again after the backup

Alternatively, by

  • logging the interactive user account out of the network share before the job starts, and logging them back in afterward. You may be able to use Windows Task Scheduler for this with the "Run only when user is logged on" option to ensure that the commands run inside the correct logon session:
    • net use /DELETE in Task Scheduler before the backup window, to log the interactive user out
    • Enter network credentials in Comet, so the service user logs in
    • net use /DELETE as an After command in Comet, so the service user logs out
    • net use in Task Scheduler after the backup window, to log the interactive user back in again

If you are backing up to the SMB server, you could work around this issue by

  • enabling the SFTP system, or the Minio S3-compatible app, and configuring your Storage Vault to use that instead. These protocols support explicitly entering credentials, that should avoid this issue.

Error "Couldn't take snapshot: The specified object was not found" using DESlock+

Some individual software, while it is launched from the within the users session, it may elevate itself when running or run under the System user account. If this happens, the encryption keys will not be available to that software's process and access will be denied to the containers.

Example scenarios where the above behavior may be experienced are when software is running under a different user context within the users session. Comet runs the backup as a service user account (usually "NT SERVICE\backup.delegate" or "LOCAL SYSTEM" in some cases). The DESlock virtual drive is unavailable for other user accounts on the system.

It's possible to mount a virtual disk globally so all users on the system will be able to access its contents. This is done using the [DESlock+] command line tool

Follow DESlock's own instructions to mount the drive for all users.

Source: https://support.deslock.com/index.php?/Default/Knowledgebase/Article/View/244

Error "VSS Error ... The shadow copy provider had an unexpected error"

Error "VSS Error: Couldn't take snapshot. The shadow copy provider had an unexpected error while trying to process the specified operation"

A VSS Provider is the driver in your system that is responsible for taking VSS snapshots. A VSS snapshot is a consistent, single point-in-time view of your harddisks. If your installed VSS provider driver is unable to create new snapshots, the backup may- or may not- be able to continue with reduced consistency.

There are multiple possible causes for this error. Some possible causes could include:

  • You have an incompatible Provider installed for your hardware. For instance, if the PC was originally installed on Hyper-V and then migrated to VMware or Azure, it may still have the "Hyper-V IC Provider" installed. Check the list of installed providers by running vssadmin list providers and ensure there are no unexpected results for your hardware.

  • When you run vssadmin list providers, if you have any Provider drivers installed other than solely the default "Microsoft Software Shadow Copy provider", it's possible that switching Comet to use the "Microsoft Software Shadow Copy provider" will help resolve the issue. You can do this by

    • In a Registry editor, in the following two keys:
      • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\backup.delegate\Parameters\
      • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\backup.elevator\Parameters\
    • edit the existing AppEnvironmentExtra entry to add an extra line containing the additional content:
      • COMET_FORCE_VSS_PROVIDER=system
    • and then restart both the backup.delegate and backup.elevator services, and see if this resolves the issue for you,

  • The issue may occur when trying to take a VSS snapshot backup of mapped network drives. You cannot use VSS on network share.

  • Shadow storage on the source drives is not configured or not large enough. The shadow storage size can be checked and manually changed through command prompt:

    • To check the current limit set: vssadmin list shadowstorage
    • To change the limit: vssadmin Resize ShadowStorage /For=X: /On=X: /Maxsize=XX%

  • Microsoft's native snapshot manager is only able to perform one snapshot at a time. If a snapshot process is already running when the backup job starts, then the backup job could fail. Stopping and restarting the Volume Shadow Copy service can resolve this problem. To do this, open an elevated command prompt window and run the following commands:

    • net stop vss
    • net start vss

A reboot of the server has also been known to clean up the snapshot manager correctly, should a service restart not resolve the issue.

  • Multiple backup products installed could cause this error. Many backup solutions have their own proprietary snapshot manager which can cause conflicts with other backup solutions installed on the system.

  • VSS snapshots have been known to fail because an Advanced Format (AF / 4Kn) drive is connected to the machine

  • Check you have all the VSS writers installed and working. On an elevated command prompt: vssadmin list writers

If you're still having issues, please open a support ticket with your results on the above troubleshooting.

Email delivery failed when using MX Direct

When using the "MX Direct" option to send email, your Comet Server is acting as an MTA (Mail Transfer Agent) to deliver the email directly to the recipient. There are strict requirements on MTAs to not have your outgoing emails discarded as spam.

You may find more information online about running an email server and not having your messages discarded as spam. The advice following are only general guidelines:

IP block reputation

Many email servers will maintain a list of known-bad IP ranges that have served spam emails in the past. If you are hosting Comet Server on a cloud VPS or cloud server, it's likely that the IP address has been used by someone else in the past. This is especially likely with the very small size of the IPv4 internet.

Some IP reputation services allow you to submit your IP address for removal.

At the time of writing, the following web services allow you to check if your IP address is on an email spam denylist:

Forward-confirmed reverse DNS

Reverse DNS is a system where, for an IP address 1.2.3.4, looking up the special domain name 4.3.2.1.in-addr.arpa produces a TXT record explaining the owner of the 1.2.3.4 IP address.

One anti-spam measure in email is Forward Confirmed Reverse DNS (FCrDNS). Looking up the DNS of the reverse DNS should produce the original domain name again. This is a method of ensuring that the domain name owner is also the IP address owner, an important authenticity check for anti-spam.

Many cloud server providers will automatically configure the RDNS record to a suitable value. However, by changing it to match your domain name exactly, you may help ensure the email delivery is successful. If you do not have the access necessary to configure reverse DNS then you will need to find out who owns your IP block (usually this is your ISP or hosting provider) and contact them to configure it.

Read more about FCrDNS:

SPF

SPF (Sender Policy Framework) is a system that allows your domain name to restrict which servers are allowed to send email messages on behalf of that domain to a set of allowed mail servers. An SPF record is added to your domain's DNS zone file as a TXT record and it identifies authorized SMTP servers for your domain.

Generally speaking, and SPF policy will only block emails; lack of an SPF policy does not reduce deliverability rates. However, you may have an existing SPF policy added for your existing domain email provider (e.g Office 365, G Suite, Zoho Mail) that prevents other servers from sending email for your domain.

Example: include:yourserver.example.com

DKIM signing

DKIM signing is a method of cryptographically ensuring that the mail server is authorized to send email. At the time of writing, Comet Server's built-in MTA does not support DKIM signing.

Jobs left in Running state, Cancel a running job

Comet Backup is responsible for closing-off a job log with Comet Server. If the PC is shut down unexpectedly, a job would be left in "Running" state indefinitely.

The old, inactive "Running" jobs will be cleaned up automatically if Comet sees an opportunity to prove that they are no longer running.

As of Comet 21.9.4, the following actions will clean up old, inactive "Running" jobs:

  • Run an exclusive action against the Storage Vault
    • For safety reasons, some operations require Comet to temporarily take exclusive control over a Storage Vault. Comet makes a number of checks to verify this exclusivity, but the practical benefit is that when such an operation runs, all past backup jobs must no longer be running by definition.
      • A reindex operation (n.b. may take a considerable amount of time), or (prior to Comet 21.9.4) a retention pass
  • Run a new backup job on the same device
    • Comet can compare lockfiles in the Storage Vault with the running process IDs in Task Manager. If a lockfile was created by a process that is no longer running, that job must have stopped.
  • Run a software update
    • If a software update job completes successfully including a changed version number, the software update process must have terminated all prior jobs. All past backup jobs on this device are no longer running

Note: If a backup job has before or after commands, then the Cancel command may not appear to work. If the commands are stalled, then the cancellation request may have no apparent effect until the commands either finish, or are stopped. The commands are started by the backup.delegate service, and as such, are not readily visible in a normally-logged-in user-view of Processes in the Windows Task manager. The processes may be found in the 'details' view, and can usually be stopped from there. Once stopped, the cancellation should work normally.

Out of memory

Comet Backup needs RAM to run. The main cause for this is to hold deduplication indexes; therefore the amount of RAM used is proportional to the size of the Storage Vault.

You might see these error messages:

  • runtime: VirtualAlloc of 1048576 bytes failed with errno=1455 on Windows
  • 0x5AF ERROR_COMMITMENT_LIMIT: The paging file is too small for this operation to complete. on Windows
  • fatal error: out of memory on all platforms

On Linux, when the system is out of memory (OOM), the kernel "OOM Killer" subsystem will immediately terminate a process of its choosing, to free up memory. If you see an error message like signal: killed in Comet on Linux, this means the process was terminated by a user or a subsystem, that might possibly be the OOM Killer. You can check for this in dmesg or kern.log.

You can reduce Comet Backup's RAM usage by trying to limit how much data is in each Storage Vault. For instance, instead of having multiple devices backing up into a single Storage Vault, create multiple Storage Vaults for each device. This will reduce the deduplication efficiency, but it will also reduce the necessary memory usage.

Trade-offs

Some trade-offs are possible, that can reduce Comet's memory usage at the expense of other system resource types:

Rescan unchanged files

This option causes Comet to read more data from the source disk, reading less data from the Storage Vault into in-memory indexes. This can have a varied impact on RAM usage, and may be positive or negative depending on the shape of your directories.

Prefer temporary files instead of RAM

The "Prefer temporary files instead of RAM" option on a backup job schedule will cause Comet to keep indexes in an on-disk database file, instead of a pure in-memory index. The on-disk database file is mapped into pageable memory, that can more easily be reclaimed by the OS when the system is under memory pressure.

Depending on how you measure Comet's memory usage, this option may not immediately appear to have lower memory usage if your measurement includes mmap disk sections. However, the resident working set is reduced.

There is a major performance penalty for using this option (approximately 5x or worse) and it is not generally recommended.

Error "HTTP 500" in Comet Backup logs

If you see an HTTP 500 error message in the Comet Backup logs, this means the server encountered an unspecified error.

If you see this while performing an operation to

  • Comet Server storage, then you should check the Comet Server log file for around the same timestamp, to see if there was a corresponding error on the server.
  • Cloud storage, then the cloud storage provider experienced an error at their end.
    • The error message may contain more detail; or
    • The operation may succeed if you retry it a short time later.
    • Open the cloud-provider web interface, and attempt to download a file from the backup in question. If it is not accessible, this points to a cloud-provider problem (such as hard-drive failure). Contact the cloud provider directly for more information.

Change of hardware causes registration dialog to appear

Comet detects the current device based on a hardware ID.

The hardware ID may be changed in some situations:

  • if you replace the motherboard or CPU; or
  • if you upgrade the BIOS / UEFI, without preserving hardware IDs; or
  • if you virtualise a physical server; or
  • if you migrate a VM guest to a different VM host, without preserving hardware IDs; or
  • if you install "sandboxing" software, or install certain PC security software that includes a "sandboxing" feature (e.g. Comodo Containment); or
  • if you make certain specific modifications to the operating system.

In these situations, the device's hardware ID will change, and Comet will recognise the PC as a new device.

Handling a changed device ID

If your device is recognised as a new device, you should register it again.

The original backup data is still preserved in the Storage Vault, and will be deduplicated against any future backups from this device.

You can move the Protected Item settings from one device to another, by using the Copy/Paste buttons in the Comet Server web interface on the user's page > Protected Items tab.

The old device should be revoked once the new device has been properly set up to prevent it from incurring a license charge, alternatively, removing all Protected Item's from the old device will also prevent the device from being charged.

The backup job log history will be preserved in the customer's account, but, it will be associated with the old device.

  • Once you de-register the original device, it would show as "Unknown device (XXXXX...)" in the job history.

  • The customer can still see these old jobs in the Comet Server web interface.

  • The customer can still see these old jobs in Comet Backup if they use the filter option > "All devices".

Because the device is detected as a new device, the billing period for this device will be restarted.

Storage Vault Locks

(This section applies to Comet prior to 21.9.4.)

Lock files are an important part of Comet's safety design. Comet uses lock files to ensure data consistency during concurrent operations.

Problem statement

Comet Backup supports multiple devices backing up into a shared Storage Vault simultaneously.

A retention pass first looks at what data chunks exist in the Vault, then secondly deletes the unused ones.

A backup job first looks at what data chunks exist in the Vault, then secondly uploads new chunks from the local data, and uploads a backup snapshot that relies on both pre-existing and newly-uploaded chunks.

It is perfectly safe for multiple backup jobs to run simultaneously, even from multiple devices.

In Comet 21.9.4 and later, it is also perfectly safe to run backup and retention jobs simultaneously. A mark-and-sweep algorithm is used to safely delete data chunks while other backup operations are proceeding concurrently.

Lock files

In order to check whether a retention pass is currently running, Comet must communicate between all devices that could potentially be using the Storage Vault.

In order to determine whether any other device is actively using a Storage Vault, Comet writes a temporary text file into the Storage Vault, and deletes it when the job is completed. This is the only mechanism supported across all Storage Vault types (i.e. local disk / SFTP / S3 / etc). Then, any other job can look for these files to see what other operations are taking place concurrently.

Downsides of lock file design

If Comet is stopped suddenly (e.g. PC crash), the lock file would not be removed. All other Comet processes would not realize that the job had stopped. This could prevent proper functioning of backup jobs and/or retention passes.

Comet will alert you to this issue by failing the job. The error message should explain which device and/or job was responsible for originating the now-stale lock file.

You may see error messages of the form:

  • Locked by user '...' on this device (PID #...) since ... (... days ago)
  • Locked by user '...' on computer '...' (PID #...) since ... (... days ago)
  • However, the responsible process might have stopped.
  • If you investigate this process, and are absolutely certain it won't resume, then it's safe to ignore it and continue.

It is possible to delete lock files to recover from this situation. However, you MUST investigate the issue to ensure that the responsible process really has stopped. Consider that a PC may go to sleep at any time, and wake up days or weeks later, and immediately resume from the middle of a backup or retention operation; if the lock files were removed incorrectly, data loss is highly likely.

If you are sure that the responsible process is stopped, you can delete the lock files.

You can initiate this either

Backup strategies to avoid lock file issues

  • If you are experiencing lock file errors where the duration is 5 minutes or less, consider rescheduling the backups, moving a backup ahead or behind by 30 minutes.
  • If you are experiencing lock file issues of more than 5 minutes (anywhere from several hours to multiple days of the storage vault being locked), consider moving one or more of the devices to a separate storage vault where the backups will not be affected by the other devices.

Automatic unlock

Comet Backup will automatically delete stale lock files when it determines that it is safe to do so.

  • When Comet is running on the same PC as a potentially-stale lock file, it can check the running processes to see if the originator process is still running.

Recovering from unsafe unlock operations

If you encounter a Packindex '...' for snapshot '...' refers to unknown pack '...', shouldn't happen error, a data file has been erroneously deleted inside the Storage Vault. Data has been lost. This can happen if the "Unlock" feature is used without proper caution as advised above.

In this situation, you can recover the remaining data in the Storage Vault by following the instructions in the "Recovering from file corruption" section above.

Error "Found X packs in index but not appearing on disk. Reindex needed!"

The message indicates that X number of index files refer(s) to data chunks that cannot currently be found inside the Storage Vault.

Encrypted data chunks in the Storage Vaults are carefully curated and indexed. It is still possible for the index of data chunks to become mismatched with the actual chunks stored. This can happen:

  • if a file was manually deleted from the Storage Vault, or
  • if your storage platform experienced data loss for any reason (e.g. RAID failure), or
  • if the Unlock action was used in an unsafe way, or
  • if a retention pass went back in time (e.g. via VM snapshot resumption)

Solution

  • Toggle on Admin -> Advanced Options
  • Accounts -> Users -> name -> Devices tab -> click 'Online' to open dialogue -> Storage Vault tab -> Reindex
  • When done, Toggle off Admin -> Advanced Options
  • Accounts -> Users -> name -> Devices tab -> click 'Online' to open dialogue -> Storage Vault tab -> Apply Retention Rules Now
  • When done, Accounts -> Users -> name -> Actions -> View Job History -> 'Report' to view last job
  • An absence of errors indicates the optimization pass found the missing data chunk(s) somewhere in the Storage Vault, and created a new index to reference them. The issue is solved.
  • If errors, check documentation for solutions to the error message

Error "<packindex/pppppppp> says <snapshot/ssssssss> depends on missing pack <data/dddddddd>, non-restorable!"

The message is similar to the 'Reindex needed' issue. It indicates that a specific backup snapshot, 'ssssssss', refers to data chunks, 'dddddddd', that cannot currently be found inside the Storage Vault. The specified backup snapshot is not currently restorable.

The error message may show more than one snapshot.

The index of encrypted data chunks in the Storage Vaults is carefully curated. It is still possible for the index of data chunks to become mismatched with the actual chunks stored. This can happen:

  • if a file was manually deleted from the Storage Vault, or
  • if your storage platform experienced data loss for any reason (e.g. RAID failure), or
  • if the Unlock action was used in an unsafe way, or
  • if a retention pass went back in time (e.g. via VM snapshot resumption)

Solution

We need to remove the packindex 'pppppppp' from the Storage Vault, then run a Retention Pass into the Vault to recalculate and regenerate the packindex.

Steps:

  • Identify which bucket is used by the user-profile via: Accounts -> Users -> name -> Storage Vaults tab -> click the Vault link to open dialogue -> Storage section -> Bucket ID.
  • Record the bucket ID, in the form of "969c047a-bb90-4a29-b6f8-07a2d56bee8f" (example ID only)
  • Identify where the bucket is located via: Cluster -> About This Cluster -> Bucket Users.
  • Record the 'Bucket Instance Location' for the bucket with the ID "969c047a-bb90-4a29-b6f8-07a2d56bee8f". The 'Bucket Instance Location' will be the 'friendly' name for one of your storage locations.
  • Access the storage location using your file explorer program, web browser, or however you normally access the storage location, to view and edit files.
  • In the storage location, find the folder which is the bucket with ID "969c047a...", and open the folder for editing.
  • Open the 'packindex' sub-folder, and delete the file with ID which starts with 'pppppppp'. There may be more than one packindex files to delete.
  • Accounts -> Users -> name -> Devices tab -> click 'Online' to open dialogue -> Storage Vault tab -> Apply Retention Rules Now. Check the log for errors.

Note: We recommend that you delete only the packindex files indicated by the error log. You could delete all packindex files, but, whilst technically a safe operation, the regeneration time for the files may be very long, especially for large data sizes.

  • An absence of log errors indicates the optimization pass found the missing data chunk(s) somewhere in the Storage Vault, and created a new index to reference it. The issue is solved.

  • If the optimization pass does NOT find the missing data chunk(s), then snapshot will never be recoverable, and we must remove it from the list of available snapshots. The following steps detail this procedure.

CAUTION: A deletion cannot be undone! Ensure you have followed all instructions above before proceeding!

  • Open the 'snapshots' sub-folder, and delete the file with ID which starts with 'ssssssss'.
  • Accounts -> Users -> name -> Devices tab -> click 'Online' to open dialogue -> Storage Vault tab -> Apply Retention Rules Now. Check the log for errors.
  • An absence of log errors indicates the optimization pass removed any remaining unreferenced data chunks, preparing the Vault for the next backup. The issue is solved.
  • If errors, check documentation for solutions to the error message

Backup process stalled on "Preparing Storage Vault for first use"

The first step on accessing a new, uninitalised Storage Vault is to generate and store some encryption material.

If a backup to a new Storage Vault seems to hang at this initial step, it's likely that Comet Backup is failing to access the storage location, and repeatedly retrying- and timing-out. An error message may appear after some extended duration.

Some possible causes of this issue are

  • Storage Vault misconfiguration
    • For Storage Vaults located in a Comet Server bucket: check the Storage Vault properties > "Hostname" field, that it points to a valid URL and not e.g. 127.0.0.1
    • For Storage Vaults using cloud bucket credentials: double-check the credentials, and ensure there are no extra spaces pasted around the field values
  • Outdated CA certificates
    • This would prevent Comet Backup from making an HTTPS / SSL connection to the storage location
    • On Windows, run Windows Update
    • On Linux, update the ca-certificates package

Errors accessing virtual cloud storage files

Notes about virtual cloud storage

OneDrive, Google Drive, and similar cloud document editors/storage all operate alike in that user files are stored as on-demand files in the respective cloud server repositories. These locations which are not easily accessible by Comet, regardless of whether a device can view and access them. Backing up these files is ONLY possible if you make a local copy of these files available to the Comet client software. This may have unwanted consequences on network loading and local disk space.

OneDrive

To save on disk space, OneDrive (and some other cloud storage providers) use a system where some files are only "virtually" stored on the local disk, and are materialized from the cloud storage on-demand.

Error "Media is write protected" backing up OneDrive with VSS

When you use the "Take filesystem snapshot" option in Comet, Comet takes a VSS snapshot of the disk. This is a read-only snapshot.

When you back up the OneDrive directory with VSS enabled, OneDrive is not able to download files into the snapshot, because the snapshot is read-only. This causes the "Media is write protected" error message.

In this situation, your OneDrive data is not being protected by Comet and is not available for restore.

You can workaround this issue by creating two Protected Items: one with VSS enabled, that excludes the OneDrive directory; and a second one with VSS disabled, that only includes the OneDrive directory.

Note that if OneDrive needs to materialize a lot of data from the cloud, then backing up the OneDrive directory may cause a lot of data to be downloaded.

Warning "Access to the cloud file is denied" backing up OneDrive

If you encounter the "Access to the cloud file is denied" warning message, this means that file in question does not exist on the local PC, and the OneDrive virtual filesystem driver is refusing to download this file on-demand for Comet to read it.

At the time of writing, the only available workaround is to disable the "Files-On-Demand" feature in OneDrive. However, this may cause an unacceptable increase in local disk usage for some customers.

To disable the "Files-On-Demand" feature in OneDrive:

  1. Right-click OneDrive in the System Tray
  2. Click the menu icon -> Settings -> Settings tab -> "Files-On-Demand" section -> disable the "Save space and download files as you use them" option

Google Drive

Warning "G:\ not found" backing up Google File Stream items

The Google File Stream app creates a virtual drive, e.g. G:\, that streams-in data from a business Google Drive account, over the internet, on-demand, without using any local disk storage, other than a small cache.

The virtual drive only exists for the interactive logged-in user session.

Comet Backup runs all backups (both interactive and scheduled backups) using a service user account. Different logon sessions have their own set of mapped drive letters, and the service session simply does not see that the virtual drive, G:\, exists.

It's still visible in the Comet Backup GUI file picker, which runs as the interactive logged-on user for performance reasons.

Ideally it would be possible to mount the virtual G:\ drive "For all users", to resolve this issue.

Currently, this does not appear to be possible in File Stream: https://productforums.google.com/forum/#!topic/apps/iRH9lD9NM90

Note: If the virtual G:\ drive can be made available to all users, the process of backing up this data requires that it be downloaded first from the Google Drive, and this may cause large amounts of inbound network traffic.

There are third-party programs for mounting Google Drive as a drive letter that may support "All Users". StableBit Cloud Drive (commercial) may do this, as well as 'rclone mount'.

Warning "EFS-encrypted files may be unusable once restored..."

You may see a warning of this form in the backup job logs:

EFS-encrypted files may be unusable once restored, unless you also backup the EFS encryption keys from this PC.
To disable this warning, please ensure you have backed up the EFS encryption keys, and then tick the 'Dismiss the EFS warning' option in the Protected Item settings.

EFS is a Windows feature that allows you to encrypt individual files on disk. The backup job was successful, but if you restore the data to a new PC, the files might not be readable because the EFS encryption keys are tied to the Windows user account. In effect, the backup might not be restorable in a practical sense.

For more information, please see the full article on EFS in the "Protected Items" section,

powershell -command "if (Test-Path -Path 'X:\WindowsImageBackup') { Remove-Item -Recurse -Force 'X:\WindowsImageBackup'} else { exit 0 }"

You should then remove this command from the job settings after the command has run, because this command would cause problems if two Comet backup jobs ever run simultaneously in the future.

Error "An attempt was made to access a socket...

Error "An attempt was made to access a socket in a way forbidden by its access permissions" in Comet Server

This error message occurs when configuring Comet Server on Windows to listen on an IP/port combination that is either (A) forbidden by the firewall, or (B) in-use by another application. Usually this affects port 80 or 443.

The following troubleshooting guidance is available:

Firewall

One possibility is that the system firewall is preventing Comet Server from binding to this IP/Port. Comet Server automatically makes a firewall exception for itself in Windows Firewall.

  • Is a third-party firewall installed, that could be preventing Comet Server from binding to this IP/port?

In use by other application

  • Is there any website visible when you browse to the IP/port in a web browser?

  • Is IIS installed?

If you open Command Prompt as Administrator and run netstat -o, it shoudl list all the open network connections.

  • Can you check if any process is using the affected port (e.g. :80), and compare the PID in Task Manager to find the process?

If the affected process is "SYSTEM" (PID 4), then the http.sys driver is responsible for binding the port. The network system from IIS (http.sys) is also a necessary component of some other Windows features and may have been installed automatically.

You can run net stop http to stop the driver and all the services that are using it.

Some common services that use this driver are

  • World Wide Web Publishing Service (IIS)
  • "Web Deployment Agent Service" (MsDepSvc service)
  • "SQL Server Reporting Services" services
  • Windows Remote Management (WS-Management service)

If you discover the services are essential, you could work around this issue by

  • configuring Comet Server to listen on a different port; or
  • configuring Comet Server to listen on a different port, and use NAT translation to make it seem like the in-use port to the outside world; or
  • configuring Comet Server to listen on a specific interface or a different IP address.

Error "too many open files"

A file handle is an abstract concept that includes network connections, temporary files, and disk files.

If you experience the too many open files error message, this means Comet is either

  • (A) running at the same time as another process with high file-handle usage; or
  • (B) has been restricted to use a very low available amount of file handles; or
  • (C) is itself using an excessive number of file handles

During a backup job, Comet uses approximately

  • approximately 10-20 handles for files being read from the disk; and
  • approximately 10-20 handles for open network connections; and
  • an unknown number of temporary cache files created during the "Building cache" phase

You may be able to work around this issue by

  • raising the kernel file handle limit (described below); or
  • ensuring no file-intensive processes are running at the same time as the Comet Backup job; or
  • ensuring multiple Comet backup jobs are not running simultaneously; or
  • for "File and Folder" protected items, by enabling the "Rescan unchanged files" option. This reduces the number of temporary files that Comet uses for local caching. However, it may reduce the backup performance.

If you discover your device has a different limit than the Operating System default, you should find the configuration file where the limit has been altered.

On macOS

macOS supports system and per-process limits on the number of open file handles.

The default limits are fixed at a quite low value (at the time of writing: 12288 system-wide, 10240 per-process).

You can check the current system-wide limits by running: sysctl kern.maxfiles

You can check the current per-process limit by running: sysctl kern.maxfilesperproc

On macOS 10.12.x and later, you can raise the system-wide limits by creating a .plist file in the /Library/LaunchDaemons/ directory. Please see these instructions for more information.

On macOS 10.11.x and earlier, you can raise the system-wide limits by updating the kern.* settings in /etc/sysctl.conf. Please see these instructions for more information.

On Linux

Linux supports system limits, per-process soft limits and per-process hard-limits on the number of open file handles.

The default limits are fixed at a quite high value (1048576 on Debian 10 "Buster"). However, the limits may have been lowered by the system administrator, especially if the Linux PC is a multi-tenant server, web server, container server or OpenVZ server, in order to provide a limited but consistent experience to the system tenants. Any installed Linux Security Module (LSM) such as SELinux or AppArmor may also impact the value.

Any new child process will inherit the parent process's limit values.

Per-process limits

You can check the current limits for new processes spawned by the current user account, by running: ulimit -n

You can check the available hard and soft limits for any process by running (e.g.):

  • Find the PID of each Comet Backup process: pidof backup-tool
  • Check the soft/hard limits for the PID: grep files "/proc/PID/limits"

You can update the current limit for new child processes, by running: ulimit -n 10485760 (for a 10x raise from the default) or ulimit -n unlimited.

  • Only the root user has permission to raise their own ulimit.
  • This will only affect newly spawned child processes. Existing processes will retain the previous limit. You should restart Comet Backup for the changes to take affect; or, you should restart the whole PC for the changes to affect all running processes, however, any changes may not survive a reboot.

You can set the per-process file handle limit for processes spawns by systemd, by adding a LimitNOFILE=... stanza to the systemd unit file. The infinity value is supported.

System-wide limits

You can view the current system-wide number of open file handles by running: cat /proc/sys/fs/file-max.

You can view the current system-wide number of open file handles by running: cat /proc/sys/fs/file-nr, or, by running lsof | wc -l. The values may differ slightly.

The default system-wide limit might be currently set

  • in the /etc/security/limits.conf file, or
  • by any file in the /etc/security/limits.d/ subdirectory, or
  • in the /etc/sysctl.conf file.

You can update the system-wide limit by running:

echo "fs.file-max = 10485760" >> /etc/sysctl.conf
/sbin/sysctl -p

This raises the system file handle limit 10x higher from the default, and then reloads the sysctl variables. On Debian, the sysctl program is in the procps package that also provides the pidof and ps programs.

On Windows

Each process has a limit on the total number of open handle objects. The maximum number of open handles is 16711680 (16 million) on x86_64 versions of Windows. However, it may be lower on other CPU architectures, or it might lowered by a system administrator via Group Policy.

There is also a per-session limit on the number of opened files over the network. The default value is 16384; you can see this by running: net config server.

You can see the current per-process handle count from Task Manager, on the Details tab by enabling the optional column "Handles".

Error "Couldn't decrypt Vault contents" message in job reports

Comet tried to access the Storage Vault, but it contained data using an unknown encryption key. Probably this Storage Vault is using the same data location as another Storage Vault (from the same- or a different- user account).

Each Storage Vault in a user's profile is automatically encrypted on first use, with a randomly generated key. If you reuse the data storage location that was already used by another user's Storage Vault, Comet would not know the encryption key for the Storage Vault, and would be unable to access it.

If you intended to share the same Storage Vault between multiple users, you should log their devices into the same account. Otherwise, you should use a different physical location for each Storage Vault.

Error "0xc0000142" in KERNELBASE.dll starting services on Windows Server 2012 R2

Windows was unable to launch Comet's process because of an internal error.

In our experience, this issue can be resolved by running Windows Update. Please ensure Windows Update is fully up-to-date on this PC.

If this does not resolve the issue, please contact support for further assistance,

Multiple devices are detected as being the same device

Comet tells machines apart by their "device ID". This is automatically determined from a mix of hardware and software identifiers.

One possible cause of this issue is if the two VMs were originally clones of each other. If you have cloned a VM in the past, it might have the same hardware and software identifiers, and so appear to Comet as the same device.

If multiple devices appear to Comet as the same device, they will share the same Protected Items and job scheduling. This causes follow-on issues for logging and reporting.

You can resolve this issue by changing the hardware or software ID for the affected VM. This will influence Comet's device ID to force the devices to be detected as different devices.

On both Linux and MacOS, the SSH host keys are one signal that influences the generated device IDs. Installing SSH, or regenerating the SSH host keys, will cause the device ID to change.

On Windows, you can add extra data to influence the generated device ID by creating a registry key.

  1. Open Registry Editor (regedit.exe)
  2. Browse to the HKEY_LOCAL_MACHINE\Software\backup-tool folder key, creating it if it does not already exist
    • The HKEY_LOCAL_MACHINE\Software\cometbackup folder key is also recognized for backwards compatibility
  3. Create a "String Value" with name DeviceIdentificationEntropy
  4. Set any random text as the Data value. This value will influence the generated device ID.
  5. Restart all Comet services (e.g. backup.delegate and backup.elevator)

Can't use %USERPROFILE% in selected backup paths on Windows

Comet does not expand Windows environment variables in the path selection.

The Comet Backup app runs backup jobs as a dedicated service account. If the %userprofile% environment variable was expanded, it would refer to the "wrong" user account.

You can back up the %USERPROFILE%\Documents directory for all users, by

  • including the entire C:\Users directory, and
  • excluding other parts of it (e.g. pattern C:\Users\**\Downloads or regex ^C:\\Users\\[^\\+]\\Downloads).

Error "ciphertext verification failed" when using a Storage Vault

This error message can occur either immediately, when running any backup or restore operation; or, it can occur part-way through a job.

Error occurs immediately

If this error message occurs immediately, it means Comet was unable to connect to the Storage Vault at all, because the encryption key in the user's Storage Vault settings does not match the files in the /keys/ subdirectory in the data storage location.

When a Storage Vault is used for the first time, Comet generates a random encryption key, and stores it in an encrypted form in both (A) the user's profile, and (B) in the /keys/ subdirectory in the data storage location. It's important that these match at all times. If they do not match, Comet will be unable to use the data inside the Storage Vault.

You may potentially encounter this issue in the following situations:

  • If the first time this Storage Vault was used, multiple backup jobs ran simultaneously

    • The first-time initialization make take a few seconds. If multiple initialization jobs were running simultaneously, this may have caused a conflict when saving the encryption key into the user profile

  • If you were performing a Seed Load, but...

    • created a new Storage Vault in the client instead of reusing the existing one; or
    • did not copy the /keys/ subdirectory or the top-level config file; or
    • misconfigured the "subdirectory" or "path" option in the Storage Vault settings

  • If you change the Storage Vault location to point to another user account's Storage Vault

    • Data locations cannot be simply reused by multiple user accounts - the other user account would have a different encryption key. If you want to share a single data storage location between multiple customers, you should have both customers log in to the same account as devices, so that they can share the Storage Vault settings including the encryption material.

Error occurs part-way through a running job

This indicates that a file inside the Storage Vault is corrupted. Please run a "Deep Verify" action on the Storage Vault, and see the "Data validation" section in the Appendix for more information.

Error "not a supported backup storage location" backing up System State to a USB flash drive

The "Windows Server System State" Protected Item type in Comet inherit some restrictions from the underlying technology (wbadmin).

It's not officially possible to spool the backup job to a USB flash drive:

"You cannot store backups on USB flash drives or pen drives." - https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc753528(v=ws.11)

This is a preventative measure in case the drive is removed mid-backup.

The following workarounds are available:

  1. Modify the flash drive to appear as a non-removable disk

For more information, please see https://social.technet.microsoft.com/Forums/windowsserver/en-US/c39c050f-d579-4222-8ad1-44d2ff53882b/windows-backup-cannot-see-usb-flash-drive?forum=windowsbackup#03e336a4-bc68-46f6-9a4c-be6907903da6

  1. Create a shared network directory on the flash drive, and tell Comet to use the UNC network path as the spool directory instead

For more information, please see https://social.technet.microsoft.com/Forums/lync/en-US/e1f0fa4e-5fb0-4749-82d6-16b1bd427495/when-will-windows-server-backup-allow-usb-flash-drives-as-a-target?forum=windowsbackup#fc8affd9-378a-447e-a36e-c5af0bb1e40b

Error "The service did not start due to a logon failure"

This error may affect Comet Backup or Comet Server. Both products make use of a background Windows service.

Please check in Windows Event Viewer for more detail about the error message. There may be an error report in the "System" log category, from the "Service Control Manager" source.

One possible reason is that an account password was specified incorrectly.

Another possible reason for this issue is if the right to log on as a service is denied for this user account. Normally the Comet installer asserts this policy during installation; however, on a domain-joined machine, it might have been overwritten by Active Directory policy.

  • If the machine is domain-joined, please check the Active Directory policy.
  • If the machine is not domain-joined, you can check this policy inside gpedit.msc; on the left-hand tree, expand "Computer Configuration" > "Windows Settings" > "Security Settings" > "Local Policies" > "User Rights Assignment"; then open the "Log on as a service" item. The target user should appear in this list, or, the target user should be a member of a Windows group that appears in this list.

Slow backup jobs

There are many possible reasons why a backup job might be slow.

Recent changes

Did the issue suddenly start happening, on a certain time?

  • New software
    • Any recently-installed software might change the performance profile of the customer's PC.
    • On Windows, check in "Programs And Features" and sort by Date to see any recently-installed software
    • Does the issue coincide with a Comet software update?

Customer PC performance

Are multiple customers experiencing the issue, or just a single customer? This helps determine whether the issue is related to your general/server-side infrastructure or whether the issue is related to the customer's environment.

  • Antivirus

    • Many antivirus programs will scan each file as Comet reads them, including but not limited to ESET NOD32 and Windows Defender.
    • Does it help to exclude Comet's backup-tool.exe program in the antivirus software?
      • Comet automatically does this for Windows Defender.
    • Does the antivirus process show as having high usage in Task Manager when the backup is running?

  • Use of slow settings

    • Ensure the "Limit backup to use only 1 disk thread" option is not enabled
    • Ensure the "speed limit" option is not enabled
    • Ensure the "Prefer temporary files instead of RAM (slower)" option is not enabled
    • Toggle the "Rescan unchanged files" option, to see if it increases- or decreases- performance

  • RAM usage

    • With large (multi-TB) Storage Vaults, there are many different data chunks that could be deduplicated against. Comet will start to use a few GB of RAM to hold all the indexes for deduplication. If the local PC is low on RAM, it may use the swapfile / pagefile, that can significantly reduce performance.

  • CPU usage

    • Comet compresses and encrypts all data before upload. On weak CPUs this may cause high CPU usage. The CPU usage may become a bottleneck.

Storage performance

Check what kind of disks the customer is backing up.

Check what kind of storage the customer is using.

Check where the temporary directory is for the backup service user account.

  • Avoid backing up files from a network share

    • If you are backing up files from a network location, Comet must make many network roundtrips to access the data. It may be substantially faster to install Comet on the network device instead.

  • Backup storage on the same volume as the backup source

    • Using a mechanical harddrive for multiple tasks simultaneously may reduce its performance from the sequential-level down to the random-level, even for sequential tasks.

  • Backup source is a single-queue block device

    • Comet issues many requests to the source disk in parallel. To avoid negatively affecting other programs on the PC, Comet tries to access the source disk at a low OS priority, but this may be ineffectual if your disk only supports a single queue. You can toggle the "Limit backup to use only 1 disk thread" option to force Comet to make only a disk request to the source disk at a time. This may have a positive effect on other programs on the PC, at the expense of backup job performance.

  • Use of external harddrives

    • Is it USB 2 or USB 3?
    • Some disk drives may experience slow performance. You can use a benchmarking tool to determine the expected performance of the USB drive both in sequential reads, and in small random reads) independently of Comet, as a baseline to compare against Comet's performance.
      • At the time of writing, CrystalDiskMark is a popular freeware software for measuring disk performance on Windows.
    • Performance Mode
      • There is an option in Windows to control whether USB drives are configured for "Quick removal" (default) or "Better performance". Switching to the latter mode can significantly improve performance, but requires you to safely eject the drive. To change this setting:
        1. Open Device Manager > Disk drives > Properties > Policies tab
        2. If the "Quick removal" / "Better performance" radio option is available, ensure it is set to "Better performance"
        3. If the "Enable write caching" checkbox option is available, ensure that it is enabled

  • Backing up direct to cloud storage

    • Check the customer's internet connection
    • Check the service provider's status page, to ensure they are not currently experiencing any errors

  • Backing up to Comet Server Storage Role bucket

    • Check the customer's internet connection
    • Check the end-to-end latency of the storage, from the customer's PC through to the final storage location. High latency can reduce Comet performance
    • Ensure the Comet Server is not experiencing high CPU or RAM usage.

Error "ERR_SSL_VERSION_INTERFERENCE" connecting to Comet Server

The latest 1.3 version of TLS reduces connection latency and improves connection security. Comet Server continues to support genuine TLS 1.2 connections.

If you see the ERR_SSL_VERSION_INTERFERENCE error, this message means that your web browser and the web server tried to use the latest TLS 1.3 standard, but, something inbetween them does not support TLS 1.3. Your web browser chose to abandon the connection rather than downgrading the connection security to TLS 1.2.

If there are middleboxes or software on the network path, that expect to be able to intercept SSL traffic but do not support TLS 1.3, then all TLS 1.3 connections will fail. These middleboxes or software on the network path may need a software update to support TLS 1.3.

  • Are you using any "web security" software that intercepts SSL certificates? (e.g. ESET Internet Security, Symantec Web Security Service)

  • Are you behind a corporate or education network proxy that intercepts SSL certificates? (e.g. BlueCoat / Microsoft Forefront TMG)

  • Is the web browser up-to-date? Some web browsers use an early draft of the TLS 1.3 standard that might be incompatible with the final TLS 1.3 used by Comet Server

  • A last-resort is to disable TLS 1.3 in your web browser, so that the web browser connects with TLS 1.2.

Error "mysqldump: Couldn't execute 'SHOW PACKAGE STATUS..."

Error "mysqldump: Couldn't execute 'SHOW PACKAGE STATUS WHERE Db = '[...]'': You have an error in your SQL syntax [...] (1064)" when backing up MySQL

This error can occur if you are using a version of mysqldump from MariaDB 10.3 prior to July 2019, connecting to an older MySQL database.

This version of mysqldump does not correctly limit itself to the remote server's capabilities.

For instance, this issue can occur with the defalt mysqldump in Debian Buster 10.0.

You can read more about this issue on the MariaDB bug tracker:

Workaround

You can work around this issue by disabling backup of stored procedures.

If this is acceptable, you can perform this workaround by stripping the --routines parameter that Comet passes to mysqldump. To do so on Linux,

  1. Create a file with the following content:
    #!/bin/bash
    # This program is a wrapper for mysqldump that removes the --routines argument,
    # to work around issue MDEV-17429 with older MySQL servers
    args=("$@")
    for ((i=0; i<"${#args[@]}"; ++i)); do
        case ${args[i]} in
            --routines)
                unset args[i];
            break;;
        esac
    done
    /usr/bin/mysqldump "${args[@]}"
  1. Save this file as /opt/CometBackup/mysqldump-no-routines
  2. Mark the file as executable: chmod +x /opt/CometBackup/mysqldump-no-routines
  3. In the Protected Item settings, set "custom mysqldump path" to this file

Error "Incorrect function"

This error message indicates the application tried to do something not supported by the disk, but all Comet is doing is reading those files and directories. If this error happens for a normal local disk, then that is certainly supported functionality.

One possibility is that the disk driver is reporting this error message as a symptom of disk corruption when it fails to read sectors for those files.

  • Check if it is possible to open the affected files in any normal app.
  • Use the disk health tools on the device. In Windows Explorer > "This PC" > right-click the drive affected (shown in the error) > "Tools" tab > "Check for errors".

Error "operation not permitted" macOS

Since macOS 10.14, Apple has introduced a new privacy flow. The user is now asked for permission when an app requires access to certain features or functions. The user will need to explicitly grant "Full Disk Access" to the Comet Backup application.

  1. Open the System Preferences (Apple menu)
  2. Select "Security and Privacy" > "Privacy" tab
  3. Select "Full Disk Access"
  4. Add Comet Backup