Block Level Copying with Syncovery
Syncovery can detect which parts of a larger file have changed, and copy only the changed blocks rather than the complete file. This feature works similar to Rsync (but it’s not the same).
Copying only the changed blocks can save bandwidth and time, especially over a slow connection. When copying between local disks or in a LAN environment, it can save bandwidth too, but may not always save much copying time, because the source file has to be read in its entirety every time in order to determine the changed blocks (except when using the File System Monitoring Service).
Only block-oriented file types are eligible for block-level copying. These include database files such as SQL or Outlook PST, as well as drive images and virtual hard disk images (VMs). Stream-based files, on the other hand, will usually cause all blocks to be changed whenever they are modified (for example text documents, spreadsheets, zip files, and photos). Thus block-level copying won’t be able to save much bandwidth with stream-based files.
In Syncovery, block-level copying is sometimes also called “Partial File Updating”. In many cases, you need to choose only the checkmark “Block Level Copying”, which is on the Special settings category in the profiles (in Advanced Mode).
The program needs to have fast access to at least one of the sides of the synchronization (except for Mode 0). The other side may be a low-bandwidth connection. If you are using an Internet Protocol, please note that only SSH/SFTP supports block-level updating directly. The other protocols can only be used with Synthetic Backup (see further down below).
Please note: Block level copying with SSH/SFTP has only been implemented for uploads, not downloads.
Block Level Copying can work in four different ways:
Mode 0: With File System Monitoring Service (new in Syncovery 8):
– the Syncovery File System Monitoring Service must be running and monitoring the source folders
– if the source is on a local hard disk, the File System Monitoring service is automatically configured when you save the profile.
– extremely fast, no full scanning of the source file necessary and no MD5 checksums needed
– Destination files must not be modified by any other profile, person, or tool
– Destination must be accessed via LAN, VPN, or SSH/SFTP
– if the source is a UNC path, a special set-up is necessary (see at end of page)
– cannot be combined with Mode 3 (Synthetic Backup) at this time
Mode 1: With Database:
– Source access must be fast
– Destination may be slow
– MD5 checksums are stored in database
– Destination files must not be modified by any other profile, person, or tool
– Destination must be accessed via LAN, VPN, or SSH/SFTP
Mode 2: With Remote Service:
– Syncovery Remote Service computes MD5 checksums on remote computer
– The “slow” side can be both source or destination
– MD5 checksums are newly calculated each time
– Files on both sides can be modified by other profiles, persons, or tools
– One side must be local or LAN/VPN, the other can be LAN, VPN, or SSH/SFTP
Mode 3: Synthetic Backup:
– similar to Mode 1, plus:
– adds Zip compression, versioning, and filename encoding
– can work locally or with any Internet Protocol for the destination side
– the changed blocks are uploaded in a new, separate zip file every time
– all older zip files must stay on the backup storage, but can be thinned out
– all connection types and Internet Protocols are supported
Mode 1: Slow Destination
In this mode, the speed-up is available when you copy files from a location to which you have fast access (preferably your own hard disk). The destination can be a slow connection, but it must be a normal file system (either LAN or VPN) or SSH/SFTP. For other connections, you can use Synthetic Backup.
Instructions for Mode 1
In your profile, make the following checkmark: Block Level Copying, which is on the Special tab sheet when editing the profile in Advanced Mode. The next time you run the profile, a database is created on your hard drive where information needed for the speed-up is stored. The second time you run the profile, you should notice the speed-up.
Mode 2: With Remote Service
This mode can speed up updating large files in both directions. The remote computer can be both source and/or destination. This is achieved by running a small service application on the remote computer, which will create the necessary checksums on the fly, when requested by the main application running on a different machine. See more information on the Syncovery Remote Service here.
The other (local) computer, where the main Syncovery program is running, needs to have normal file system access to the remote computer (LAN or VPN), or it can use SSH/SFTP. It needs to have write access to the remote computer so that it can save the checksum request file there. The MD5 checksums are created when needed, so that no database is being used.
Instructions for Mode 2
On the remote system, run the Setup program and install the Syncovery Remote Service along with its control panel. Start the control panel from the Syncovery group in the Start menu. On the tab sheet Configure Checksummer, enter the base folders that will be used for synchronization. Click Apply. On the tab sheet Service Configuration, click on Install Service and Start. The service will be using the Windows System account by default. If this account doesn’t have sufficient access privileges, you may have to change the account in Windows Control Panel -> Administrative Tools -> Services.
On the local system, you are running the main Syncovery program. In your profile, the right-hand side must be the remote system. Specify one of the folders which you have specified for the remote service to monitor. The left side should be your local folders, or a network drive with relatively fast access. On the Specials tab sheet in Advanced Mode, make the following checkmarks: Block Level Copying and Right side uses Remote Service.
Mode 3: Synthetic Backup
This feature is intended for backing up from a local storage to any type of backup storage. The backup can be local or online.
Choose “Synthetic Backup” on the tab sheet Versioning->Synthetic Backup. This will automatically place these additional checkmarks:
– Block Level Copying (under Special)
– Filename Encoding (under Versioning)
– Compress Each File Individually (under Compression/Encryption)
Find out more on Synthetic Backup on this page.
Setting up the Mode 0 (File System Monitoring Service) with a UNC path
The file system monitoring service can be used to enable block level copying from a UNC path, in addition to copying from local drives.
To make this work, the monitoring service must be installed and configured on all machines that write to the UNC path. The Auxiliary Services Control Panel must be used to configure the File System Monitoring Service. You need to type the UNC path that is the source path for the Syncovery job into the configuration field.
In addition, the File System Monitoring Service must be given a user account that has access to the UNC path. This is done via Windows Control Panel->Admin Tools->Services by editing its properties. Then you can start the service. You should see a hidden .Syncovery folder in the UNC path, and a file such as SyncoveryFSMonitor.MACHINENAME.active in it. If you have problems setting this up, please contact support@syncovery.com.