Cache destination does not work as intended when used in combination with PGP encryption

[alexmaurer]

When using "Cache destination File List" and ZIP compression with password encryption all is working as excepted.
- On first profile run, all files are copied (for example left to right on Google Cloud Storage)
- On next profile run, only the modified/moved/deleted files are processed.

But when using "Cache destination File List" and "Encrypt PGP", the next profile run would like to delete all files on right (Google Cloud Storage) and copy them all again.

Here is my setup :
- I'm using Linux version 10.15.9.
- I only selected "Encrypt PGP".
- Filenames are not encrypted.
- Resulting filename on right side ends with ".pgp" extension

Has someone experienced the same behavior ? I'm missing something ?

[tobias]

Hello,
yes that is correct. Even when not using the cache, Syncovery cannot match source files against pgp files. Eventually this will be implemented.

In the meantime, you need to adjust the profile settings so that it will work better.

The primary use case for PGP encryption in Syncovery is the exchange of encrypted files with third parties. For example, some institutions use the feature to send transaction data. These files are usually removed at the destination, so Syncovery never needs to match existing files agains the pgp files.

If you want to use PGP for a mirrored backup, you need to use Standard Copying mode and cannot work with deletions at all. Files that have already been copied need to be marked off so that they aren't copied again. You will find two checkmarks for marking copied files under "Masks & Filters" -> "General Filters" in the profile. On Windows, you would use the Archive attributes.

[alexmaurer]

Thank you Tobias for the quick answer !

Unfortunately my use case is to get an exact mirror involving 365 days delayed deletion on target (as per the ZIP-compression-encryption functionality).

I thought it was great to switch to pgp but I should have tried before sending 500Gb over to GCS Archive class

The "workaround" with marking files when copied is not suitable for me, I hope one day the cache database could be use the same manner as the ZIP.


Es ist jetzt Mittag und ich wünsche Ihnen einen guten Appetit !

[tobias]

Hello,
sorry if this doesn't work for you as it is now. Maybe I can implement a solution relatively quickly, if you can wait for a few days?

The main difference so far is that pgp files don't get the original file size added to the file name, as Syncovery does with zip, sz and 7z. Therefore the synchronization needs to be done based on timestamp only. Which might be good enough actually.

How did you connect to GCS? Using the native Google Cloud Storage protocol, or S3?

Did you use the new GCS authentication method that is documented on this page?
https://www.syncovery.com/google-cloud-storage-sync/

[alexmaurer]

sorry if this doesn't work for you as it is now. Maybe I can implement a solution relatively quickly, if you can wait for a few days?

Ho that would be great !! Of course I can wait.

....Therefore the synchronization needs to be done based on timestamp only. Which might be good enough actually.

Timestamp only will be largely good enough.

I use the native Google Cloud Storage and I switched a moment ago with the new GCS authentication method you provided.

[tobias]

Hello,
thanks, I will start working on this now.

[tobias]

Hello,
the new version 10.15.10 for Linux can now match up files against .pgp files.

I have also fixed a bug where the pgp files would get the modification date from the upload rather than the original date.

I believe that your previously uploaded files did not retain the original modification timestamp. You can ask Syncovery to fix the timestamps without re-uploading the files by going to the "Comparison" tab sheet in the profile and choosing "Never Copy Files With Identical Sizes ... Adjust Timestamp Only". You could let Syncovery perform one run to fix the timestamps, and then remove the checkmark again.

Or you could leave the timestamps as they are, and remove the Exact Mirror checkmark:
- Overwrite Newer Files

Then Syncovery will only re-upload a file if it has been modified and if it has a newer timestamp than the one from the previous upload.

I hope this helps and you won't have to re-upload the data.

[alexmaurer]

Thank you tobias for the 10.15.10 release.

I managed to not send my data all over again thank you.
I disabled "Overwrite newer files" under "Exact mirror" settings.
And I had to enable"Double-check the actual destination for those folders where new or updated files seem to exist" in "Destination caching" settings for more consistency.

One odd behavior is that Syncovery would like to always copy one file to GCS on every run.
The filename on left is "main~1.HEX" and this file exist on right (GCS) under the filename "main~1.HEX.pgp"

[tobias]

Hello,
can you send the list of all existing files in the folder which are alphabetically near this one? Maybe Syncovery can't match them up due to a sorting problem.

I believe there are additional "main" files in that folder.

It could also be a case sensitivity problem.

[alexmaurer]

Files in the directory are :

Code: Select all

2007-05-20 15:20:02.000000000 +0200 main~1.HEX
2007-05-20 15:20:02.000000000 +0200 main.BAK
2007-05-20 15:20:02.000000000 +0200 main.c
2007-05-20 15:20:02.000000000 +0200 main.cof
2007-05-20 15:20:02.000000000 +0200 main.err
2007-05-20 15:20:02.000000000 +0200 main.HEX
2007-05-20 15:20:02.000000000 +0200 main.LST
2007-05-20 15:20:02.000000000 +0200 main.PJT
2007-05-20 15:20:02.000000000 +0200 main.sta
2007-05-20 15:20:02.000000000 +0200 main.SYM
2007-05-20 15:20:02.000000000 +0200 main.tre