Is there a way to specify an external compression tool?

[KolbyG]

Hi there,

I see that there's an external file copying tool option, but is there way to specify an external tool for the compression/encryption portion, and use Syncovery to copy the resulting file to the destination (Azure in this case)? I'd like to invoke WinRAR to take advantage of recovery records for some of my files, but not sure if this is possible.

Thanks!

[KolbyG]

Alternatively, does the .sz format support any kind of parity system or recovery records?

[tobias]

Recovery records are not necessary or useful when uploading to Azure. Corruption does not happen on Azure, or if it does, files are gone completely and not partially.

There could be hacks to invoke WinRAR but I do not believe it will be useful. You could possibly use two Syncovery jobs, one to make a local compressed copy and the other one to upload the files.

Or if you become a PascalScript artist or order premium support to have a script be written, it might be possible.

Please see
https://www.syncovery.com/pascalscript/

[KolbyG]

Recovery records are not necessary or useful when uploading to Azure. Corruption does not happen on Azure, or if it does, files are gone completely and not partially.

There could be hacks to invoke WinRAR but I do not believe it will be useful. You could possibly use two Syncovery jobs, one to make a local compressed copy and the other one to upload the files.

Or if you become a PascalScript artist or order premium support to have a script be written, it might be possible.

Please see
https://www.syncovery.com/pascalscript/

I suppose that's a good point about files while sitting in Azure, though I believe there is still a possibility of corruption while the file is being uploaded to Azure.
Are you aware of a way to have Syncovery validate each file uploaded to Azure via hash? I know Azure can create MD5's under some conditions, but I believe its depends on the file size or requires special upload arguments.
I tried the verification system but it seems to download every uploaded file instead of comparing any sort of azure-calculated hash, resulting in huge bandwidth costs that make that type of verification completely unusable.
I did look at writing a script but would really rather not if I can avoid it. I had a fairly complex script with my last backup solution to handle compression including creating packages which Syncovery thankfully supports natively, so I'd like to simplify this.

I'm probably making this into a bigger deal than it is, but with millions of files and close to 100TB to upload, I just want to make sure to do this in the best way from the start.

[tobias]

Hello,
each uploaded file or block is verified with an MD5 checksums that Syncovery calculates locally, and Azure calculates on their side. If there is an MD5 mismatch, Syncovery will raise an exception and re-upload the file until it matches, or if the number of retries is exceeded, it will error out and consider the file not uploaded. But I've never seen MD5 mismatches.

In the past 15 years or so, I've never heard of corrupted uploads to Azure BLOB storage. I think it is extremely trustworthy. Transfers are secured by TLS and verified with MD5. I don't see any way that a corruption could occur in transit that would not be detected.

[KolbyG]

Hello,
each uploaded file or block is verified with an MD5 checksums that Syncovery calculates locally, and Azure calculates on their side. If there is an MD5 mismatch, Syncovery will raise an exception and re-upload the file until it matches, or if the number of retries is exceeded, it will error out and consider the file not uploaded. But I've never seen MD5 mismatches.

In the past 15 years or so, I've never heard of corrupted uploads to Azure BLOB storage. I think it is extremely trustworthy. Transfers are secured by TLS and verified with MD5. I don't see any way that a corruption could occur in transit that would not be detected.

Thank you for that reassurance, and for confirming that there is verification involved. Just to confirm, this happens by default? There's nothing in Syncovery that I have to toggle for this checksum verification to occur?

[tobias]

Yes it is always done.