SFTP Mirroring w/ Version + Block-Level

[tobias]

Hello,
I have implemented this now, but it requires SSH shell access. There is an SFTP copy command but none of the SFTP servers I tried allow it (Operation not supported).

Can you tell me more about the type of SFTP server you are using? So I can double-check it before uploading the next build.

[DrkMstr310]

Using a Synology NAS.

As a proof of concept I tried a remote copy w/ WinSCP, & it seemed to work, so maybe thats the easiest thing to test/replicate?

[tobias]

Hi,
good if it works for you - unfortunately my Synology shows different paths via SFTP, they are shortened. Not even WinSCP can duplicate a file. Because the shell command would need the complete path, which is not exposed via SFTP.

Maybe this can be configured ...

If it works for you I will upload the new development version tonight so you can test it.

[DrkMstr310]

Hmm...well I originally did my test a very long time ago, when I first mentioned this issue, so I'll do a more careful reconfirmation after work today & let you know, just to be certain. It's possible various things have been updated since back then.

[tobias]

In the meantime, I will add a setting where you can specify the missing part of the path, I think it's simply /volume1 ...

[tobias]

Hello,
you can try version 10.dev.15 now. It will try to make a versioned copy before doing block level copying, if both versioning and block level copying are chosen. For SFTP uploads only. Compression/Encryption must not be chosen.

For Synology, I added a new checkmark on the second tab sheet of the FTP/Internet dialog:

X Path Prefix for Shell Commands...

I had to specify /volume1 as a path prefix, because the SFTP paths were all relative to /volume1.

[DrkMstr310]

Hi,

OK, I gave it a try as follows in 10.dev.15:
1) Starting with an existing profile that only had versioning enabled
2) I enabled Internet->Advanced->Path Prefix For Shell Commands=/volume1
3) I enabled Special->Block Level Copying (using checksums)
4) I did a first run, and saw it calculating the initial checksums as expected
5) I made a small edit to a sqlite file that's within the profile folder
6) I ran the profile again. However, it still re-uploaded the full size of the sqlite.

In the log, I find:

Code: Select all

Executing SSH shell command: cp -p "/volume1/AppData/CherryTree.ctb" "/volume1/AppData/$VERSION$/CherryTree.d20220907-u153156.ctb"
Rename OK: /AppData/CherryTree.ctb to /AppData/$VERSION$/CherryTree.d20220907-u153156.ctb
MOVE /AppData/CherryTree.ctb
     -> ../$VERSION$/CherryTree.d20220907-u153156.ctb
Exception checking the file size for /AppData/CherryTree.ctb: tgsftp_getstat: Error 2, no such file or directory
RIGHT SIDE: Not using partial updating because the destination file size cannot be confirmed for /AppData/CherryTree.ctb

However, I also just re-tested remotely duplicating the file in WinSCP, and it doesn't seem to be working there either now. When I previously tested it Synology was still on DSM 6, a previous major release - they since released DSM7 that I think increased security with a lot of things, so it's possible this changed.

I also have a thought. When we originally talked about this in 2020, you mentioned the option of just using Syncovery Remote Service to do the copy on the sever side. If this "ssh shell approach" is finicky and prone to break, it seems like that might be the easier option, as that would work on any platform, for all your users, regardless of shell access and regardless of how the server is implemented. Thoughts?

[tobias]

Hi,
I think you just need a different path prefix, not /volume1 like on my NAS.

Please try to find out what your complete path is in shell, and see what's different from the SFTP paths.

It could even be /volume2 or something else.

I will eventually consider enabling the remote service for this too, but in this case, using a shell command is much simpler and straightforward.

[tobias]

You can also see the full path in the Synology File Station, by right-clicking a share or folder and choosing "Properties". In my case, the SFTP path is /MainData, but the full path is /volume1/MainData.

synology_full_path_smaller.png (149.13 KiB) Viewed 4691 times

[DrkMstr310]

The prefix does indeed seem to be /volume1 for me too