Block-level copying keeps writing the whole file

English Support for Syncovery on Linux etc.
vance
Posts: 16
Joined: Tue Aug 24, 2021 10:28 pm

Block-level copying keeps writing the whole file

Post by vance »

I have a profile consisting mostly of a handful of large VirtualBox and Veracrypt drive container files. The destination is a USB drive attached to another Linux machine, over SMB/wifi. Because the files are so large, yet will have small portions within them changing every day, I'm using block-level copying.

But every time I run this profile, Syncovery copies the files in their entirety. The log says: "Not using partial file updating for <filename> (dest file does not match database)".

How is it making this determination? Does it use the same rules as for ordinary file comparison? In case the timestamps are somehow slightly off, I tried "ignoring seconds" -- it didn't help.

No one has touched the PC to which the destination drive is attached, and no app is running on it that I'm aware of.

To simplify the setup, I've now masked the profile to a single 4GB Veracrypt volume, allowing it to complete in 1-2 hours. I've then run it in succession a few times, with only brief periods in between. This minimizes the possibility that some external factor is messing with the destination file. It still happens.

What am I missing? Could it matter if the drives are using different file systems? The source drive is ext4, but I believe the destination is NTFS.

tobias
Posts: 670
Joined: Tue Mar 31, 2020 7:37 pm

Re: Block-level copying keeps writing the whole file

Post by tobias »

Hello,
yes this means that the database "remembers" a different version of the file in terms of file size and modification timestamp.

The log file should contain the details which are deviating, such as
DstSize:
last mod:
DB: Size:
last mod:

Please post these details. It would also help if you make a screenshot of the Sync Preview every time before letting Syncovery copy, so we can compare those timestamps and sizes against the logs.

It could be that there is a problem on Linux that causes this mechanism to fail if the source file is being written to while the copy takes place.

vance
Posts: 16
Joined: Tue Aug 24, 2021 10:28 pm

Re: Block-level copying keeps writing the whole file

Post by vance »

Now that you point out the sync view, I realize it's been staring me in the face: the destination file is somehow getting written with a modified date of 2/18/2026.

* The system clocks on the two machines are both set accurately
* I see the 2026 date not only here in Syncovery, but also when I look up the file on the destination machine (although Syncovery seems to read it one hour off)
* The "created date" is accurate; 2026 is only shown in the "modified date"
* The 2026 timestamp advances with real time. So for example, a test from yesterday reported the modified date as 2/17/2026
* File system isn't a factor, because I'm now writing to another ext4 volume

So for example, the test I ran today... screenshot attached. This was in the log, afterward:
DstSize: 4294967296, last mod: 02/18/2026 16:11:05
DB: Size: 4294967296, last mod: 09/14/2021 12:23:02
Not using partial file updating for syncovery-test (dest file does not match database)

And on the destination machine, the file shows the creation date as this morning (when I first switched the profile to an ext4 destination), but a modified ate of 2/18/2026, 15:06:46. (The test began around 16:22, so the time appears equally random.)

The backup took <90 minutes, so the time gap above doesn't reflect the write time.

During these latest attempts, I'm not writing to the source file during the transfer. The Veracrypt container remains dismounted from beginning to end.

Let me know if you need more info? I've ruled out everything I can think of... I'm hoping you have some ideas.
Attachments
Syncovery preview.png
Syncovery preview.png (71.09 KiB) Viewed 181 times

vance
Posts: 16
Joined: Tue Aug 24, 2021 10:28 pm

Re: Block-level copying keeps writing the whole file

Post by vance »

Also, I did a manual copy of this same file (using the file system), to the same destination drive. That file has a correct modification date.

tobias
Posts: 670
Joined: Tue Mar 31, 2020 7:37 pm

Re: Block-level copying keeps writing the whole file

Post by tobias »

Hello,
thanks. I will make a similar test here.

In the meantime, some thoughts:

- do you have the latest Linux version 9.38g?
- can you go to the Program Settings dialog, tab sheet "Logs", and choose: Destination Timestamp Setting Details
- can you test copying small files without block level copying, to the same destination? Does that work?
- if not, can you send the log file to support@syncovery.com or post it here?
- if the small files work, can you make another test of a large file with the timestamp settings details in the log and provide that log file?

Thanks.

vance
Posts: 16
Joined: Tue Aug 24, 2021 10:28 pm

Re: Block-level copying keeps writing the whole file

Post by vance »

The version is actually v9.38e. (I've been waiting to buy until I feel more sure this setup is going to work.)

For the first test today, I did a backup of this same 4GB file, but with block-level copying disabled; it still used a 2026 mod date.

Then I did the backup with a new, zero-byte file. The backup failed "incomplete", with some stack/thread errors logged -- I'll email this to you.

And then it seems that whatever went wrong, it couldn't recover. I tried to run this same backup again, but this time it hung on "Starting...". I reloaded the webpage, and the view remained hung on an hourglass with the word "Loading...". I then tried to use the commands you gave me in another thread, to kill the background thread (SyncoveryCL stop, sudo killall SyncoveryCL, sudo killall -9 SyncoveryCL), and tried to restart the thread within a terminal window. I'll send you the output, but it basically complains that the old thread was still running. So, not seeing another way out of this, I rebooted.

After rebooting, I added two bytes to the file (just in case zero bytes was part of the problem) and re-ran it. This backup completed, but the destination file still has the 2026 modified date.

Then I ran this same two-byte file to a local USB drive. This time, the modified date was today's!

Similarly, I have another PC with essentially the same profile, with block-level copying of a ~50GB Veracrypt volume, on the same versions of Linux and Syncovery, but to a local USB drive. That one works really well; there's no 2026 date.

So I'm thinking this might be related to SMB vs. local? In case it matters, this SMB is implemented with Samba, installed on the destination machine. Like everything else in this setup, the Samba version is really recent. The Samba configuration is really basic.

tobias
Posts: 670
Joined: Tue Mar 31, 2020 7:37 pm

Re: Block-level copying keeps writing the whole file

Post by tobias »

Hello,
thanks for the logs! I will analyze these.

I noticed that you are using Syncovery's internal SMB feature, which is based on libsmbclient. Which is usually less stable and slower than if you use a mounted smb folder. Could you actually mount the network drive to a local folder and use that in Syncovery?

vance
Posts: 16
Joined: Tue Aug 24, 2021 10:28 pm

Re: Block-level copying keeps writing the whole file

Post by vance »

That seems to have worked! In limited testing so far, anyway. Tomorrow I'll work with it more, apply it to my actual profile, and close back here.

vance
Posts: 16
Joined: Tue Aug 24, 2021 10:28 pm

Re: Block-level copying keeps writing the whole file

Post by vance »

I think I finally have it working. I mounted the SMB volume locally using cifs-utils, which got rid of the 2026 mod date problem.

I then ran into a separate timestamp problem that was probably contributing as well. A few times, I temporarily moved the destination drives to a local USB connection to speed things up. Somehow the timestamp on those files was being read differently by the two PCs, such that the destination PC saw them seven hours later. When I moved the drive back to that machine and mounted on the source machine via SMB, Syncovery saw them as future dates and wanted to re-write them all.

I never did figure out the time gap. The two PCs are identical in their date/timezone configuration. My timezone happens to be seven hours behind GMT, so it's almost like something was assuming GMT instead of accounting for my timezone. I finally just used "touch" to modify the timestamps on the large files, to bring them back in line with the source. Syncovery then saw them as equivalent.

One last question... how much risk is there in backing up a VM file while the VM is still running? Even if the backup is limited to night-time (no user), the guest OS is probably flipping bytes fluidly (Windows Update, indexing, etc) as Syncovery sweeps sequentially across the file. It seems like a good way to introduce corruptions. Is it a best practice to pause the VM during the backup?

tobias
Posts: 670
Joined: Tue Mar 31, 2020 7:37 pm

Re: Block-level copying keeps writing the whole file

Post by tobias »

Hello,
thanks for the update! Depending on the file system, sometimes the timestamps might be interpreted as UTC and sometimes as local timezone. The issue is rare nowadays, under Windows, but might still happen sometimes when Linux mounts non-native file systems like NTFS or exFAT. Maybe even with ext4, I'm not sure.

Yes, if possible, I highly recommend stopping or pausing VMs before doing a block level copy on Linux. It is different on Windows, where Volume Shadow Copying is used for active VM images. The volume shadow ensures that a snapshot is captured in one moment, so that the contents will be consistent. But on Linux, Syncovery accesses the live files without using any snapshotting technology. So it's much more likely that the copy will not be valid.

Post Reply