Data Integrity doesn't detect deleted files

General Discussion in English
Post Reply
daiquiri
Posts: 13
Joined: Sun Sep 20, 2020 11:17 am

Data Integrity doesn't detect deleted files

Post by daiquiri »

Hello,
I'm testing Syncovery 9.13b windows 64bit,

Data integrity fails to detect deleted files in the right size.
  • Create a new exact mirror from left to right
  • Enable use compressed packages
  • Enable all encryption options
  • Run the profile
  • Everything is OK at this point
  • Delete all files inside right side folder
  • Do an integrity test
  • No errors detected despite all files are deleted!
Is this expected behavior?

tobias
Posts: 1603
Joined: Tue Mar 31, 2020 7:37 pm

Re: Data Integrity doesn't detect deleted files

Post by tobias »

Hello,
yes, the only purpose of the integrity check is to find out if Synthetic Backup files can be reassembled.

If you don't use Synthetic Backup, then don't use the integrity check please.

daiquiri
Posts: 13
Joined: Sun Sep 20, 2020 11:17 am

Re: Data Integrity doesn't detect deleted files

Post by daiquiri »

If it not intended to be used without Synthetic why do I have the choice of "test all files" and "test only files that need synthetic backup reassembling?"
This is confusing. If no Synthetic options are enabled in the profile the option should be disabled?
It works for other usecases, just not for the use case where all files in the right side are deleted

So is there anyway of checking right side data integrity?

tobias
Posts: 1603
Joined: Tue Mar 31, 2020 7:37 pm

Re: Data Integrity doesn't detect deleted files

Post by tobias »

Hello,
it depends what you are trying to achieve exactly. In general, if you run the job and the Sync Preview is empty, then nothing needs to be backed up and the backup is complete. Especially when you use "Exact Mirror". If that is the case, you can then use "Verify integrity" to make sure that all files can actually be downloaded and unpacked.

To verify a backup completeness and contents in a single step, run the same job with "Binary comparison of existing files" under Comparison->More in the profile. But not if the files are zipped. The verification works only with the new Sz compression format, as well as uncompressed files.

Choose "Remember Results" so that the same files won't be compared over and over when you run the job. If all files compare OK, the Sync Preview will come up empty (or show Nothing to do). You can check verification details in the log file.

You need to do the Binary Comparison only once, and then you can remove the "Binary Comparison" checkmark.

For new files added in the future, you can use "Verify copied files" on the tab sheet Files. This verification works with zipped files too, but it only verifies the zip file itself, not the contents. As opposed to the Sz format, where the contents are compared.

daiquiri
Posts: 13
Joined: Sun Sep 20, 2020 11:17 am

Re: Data Integrity doesn't detect deleted files

Post by daiquiri »

Thank you for explaining.

I'll try to exercise these settings combinations.

These are a lot of undocumented conditions/restrictions/limitations :-)

daiquiri
Posts: 13
Joined: Sun Sep 20, 2020 11:17 am

Re: Data Integrity doesn't detect deleted files

Post by daiquiri »

Ok I continued to test this combinations but without any success.
I've done the following:

- choose "SmartTracking"
- Use SZ compression, multiple files per package
- enable "Binary comparison of existing files"
- Delete all files after first backup

If you Test Backup integrity: no errors
If you try to backup again, Syncovery thinks all files are backed up

However, the destination is empty

Am I doing something wrong?

tobias
Posts: 1603
Joined: Tue Mar 31, 2020 7:37 pm

Re: Data Integrity doesn't detect deleted files

Post by tobias »

SmartTracking is only for two-way syncs.

In one-way syncs, the only reason to use SmartTracking is if you want to ignore deletions. Which is exactly what it did.

You really need to use Standard Copying or Exact Mirror.

daiquiri
Posts: 13
Joined: Sun Sep 20, 2020 11:17 am

Re: Data Integrity doesn't detect deleted files

Post by daiquiri »

I'm so sorry. I'm not finding a solution.

I want:
  • Copy from source to destionation
  • Use compression (SZ)
  • Use compressed packages
  • All encryption options
Then
  • Run the copy once: copy ~1.5TB
  • The source starts to have more files, if I run it a second time it copies ONLY the new files to destination
  • If a file/package is deleted from destination, it copies the deleted files again
Is this possible?

tobias
Posts: 1603
Joined: Tue Mar 31, 2020 7:37 pm

Re: Data Integrity doesn't detect deleted files

Post by tobias »

Hello,
Syncovery works best with the option "Compress Each File Individually". That way, Syncovery (and you) can always easily see which files are on the destination. This is the native mode of operation for Syncovery, because it's heart is a "file synchronizer". It works with individual files. Not packages.

Working with individual files has many advantages, such as being able get a complete listing of all files, being able to retain a specific number of older versions for each file, being able to delete files individually, and having direct access to any specific file you want to restore.

Multiple files per package was added for specific, limited use cases only, such as one-time archiving, or regular archiving of a (smaller) set of files where each archive should contain all files.

If you really want to use compressed packages (multiple files per package), it becomes very difficult. If you delete a package, Syncovery cannot know which files it contained and cannot re-copy them. If you want to restore a single file, it will be extremely hard, if not impossible, to find which package contains it. This is just not the use case that Syncovery was designed for.

When you use packages with multiple files per package, Syncovery cannot see which files are already on the destination side. Therefore, files have to be marked as archived on the source side. On Windows, this is done with the Archive flag. On Linux or Mac, it can be done with Extended Attributes. The archive flags / extended attribute marking options are under Masks & Filters -> General Filters.

Either way, please make sure you have the latest Syncovery version.

Post Reply