How does Syncovery compare left and right files if you use compression and encryption?

English Support for Syncovery on Windows.
IMTheNachoMan
Posts: 86
Joined: Sun Nov 20, 2022 5:11 am

How does Syncovery compare left and right files if you use compression and encryption?

Post by IMTheNachoMan »

I am backing up ~2 TB to B2. I am using max/ultra compression and encryption.

Some of my source/left files are huge (10 GB).

So how will Syncovery compare the source/left file with the compressed and encrypted right/destination file?

Will Syncovery have to first compress/encrypt the source/left file to compare or does it keep a DB of hashes?

tobias
Posts: 1603
Joined: Tue Mar 31, 2020 7:37 pm

Re: How does Syncovery compare left and right files if you use compression and encryption?

Post by tobias »

Hello,
files are compared by file names, last modification timestamp, and file size.

Because the original file size isn't known for compressed files, Syncovery adds it to the filename like this:

file.s1234.zip

where 1234 is the original file size.

A database is not needed, and hashes aren't needed either.

IMTheNachoMan
Posts: 86
Joined: Sun Nov 20, 2022 5:11 am

Re: How does Syncovery compare left and right files if you use compression and encryption?

Post by IMTheNachoMan »

Ah, that is pretty slick. In your example, is the 1234 in bytes, kilobytes, ??

Does Syncovery have an option to use a DB that stores hashes? I would be interested in using that if it will improve performance.

tobias
Posts: 1603
Joined: Tue Mar 31, 2020 7:37 pm

Re: How does Syncovery compare left and right files if you use compression and encryption?

Post by tobias »

Hello,
1234 is the size in Bytes. For larger files, this string will be longer ;=)

When building the list of files to copy, Syncovery does not look at the file contents by default, only the metadata: file names, timestamps, and file sizes. Hashes would mean looking at the file content, which is normally not necessary.

Syncovery can store hashes for some purposes but it will slow things down, not make them faster. A use case for hashes is Synthetic Backup:
https://www.syncovery.com/synthetic-backup/

If you feel the need to verify the contents of files (which will take a long time), you can use "Binary Comparison of existing files" on the tab seet Comparison->More in the profile. This should rarely be necessary. Uploads to B2 are verified using hashes on multiple protocol levels during upload.

IMTheNachoMan
Posts: 86
Joined: Sun Nov 20, 2022 5:11 am

Re: How does Syncovery compare left and right files if you use compression and encryption?

Post by IMTheNachoMan »

If I use "Binary Comparison of existing files" on the right side, and I also have compression/encryption on the right side, how does it work?

The files on the left are not encrypted/compressed, and the files on the right are encrypted/compressed. So they won't have the same hashs.

Or is it doing some cool magic?

tobias
Posts: 1603
Joined: Tue Mar 31, 2020 7:37 pm

Re: How does Syncovery compare left and right files if you use compression and encryption?

Post by tobias »

Hello,
it will download the files, extract them and compare then content.

IMTheNachoMan
Posts: 86
Joined: Sun Nov 20, 2022 5:11 am

Re: How does Syncovery compare left and right files if you use compression and encryption?

Post by IMTheNachoMan »

Even for a one way backup?

I see why you were saying to avoid that because if the file is huge then it'll be a big deal to download.

If you're taking suggestions for features, I have some:
  • Local cache list of destination files and their hashes. This assumes nothing else is modifying the files in the destination but that's true for many use-case.
  • Store the hash data in in a text file that is added to the ZIP file of the compressed/encrypted file. Then, for future syncs, you only have to download enough bytes to get this text file to do the compare. So, if you have some file.docx, when you encrypt/compress it you get some file.docx.zip and this zip file would have both the some file.docx file and a some file.docx.txt file.
Just a thought for how to add efficiences.

Thank you!

tobias
Posts: 1603
Joined: Tue Mar 31, 2020 7:37 pm

Re: How does Syncovery compare left and right files if you use compression and encryption?

Post by tobias »

Hello,
Syncovery can keep a local cache if you choose "Cache Destination File List" on the "Special" tab sheet.

But it doesn't normally calculate and store any hashes because they would be useless. All that is needed to sync files is the modification timestamp and file size.

There's one feature that causes it to store hashes, and that is Block Level Copying. The hashes are NOT used to compare source and destination. They are only used to find out which blocks have changed and need to be uploaded. This is only useful for files such as database files, Outlook PST files, and VM images, which are modified frequently and you don't want to upload the complete file every time.

IMTheNachoMan
Posts: 86
Joined: Sun Nov 20, 2022 5:11 am

Re: How does Syncovery compare left and right files if you use compression and encryption?

Post by IMTheNachoMan »

Got it.

Thanks.

I know that mod time and file size are all you need 99% of the time but there are cases where it isn't perfect and the super paranoid (like me) might want 100% verification. For example, I've seen situations where a file gets corrupted but the mod time and size stays the same. Some ransomware can lock a file in a way that keeps the file size and mod time. I'm just a very paranoid person and I'm looking for super paranoid level verification. :)

Another idea I had was to compress/encrypt the file locally, then take the hash of that and compare that with the hash of the compressed/encrypted file on the right. Many cloud providers (like B2) give you the file checksum when you list files so it would be super-fast to compare that with the local file.

Anyway, I got what I needed. Thank you!

IMTheNachoMan
Posts: 86
Joined: Sun Nov 20, 2022 5:11 am

Re: How does Syncovery compare left and right files if you use compression and encryption?

Post by IMTheNachoMan »

tobias wrote:
Thu Nov 24, 2022 10:08 pm
where 1234 is the original file size.
I was thinking about this some more. That means that I can change the compression level after initial upload and it will still work -- since it is comparing the original file size. Right?

So do a backup with compression set to low. Then if I change compression level, it'll still know if the file changed and if it needs to upload a new file, the new file will be on the new compression level.

Right?

Post Reply