Page 1 of 1

Feature suggestion: Exact mirror + Reverse Incremental Backup

Posted: Mon Mar 01, 2021 10:49 pm
by Shaipetit
As Syncovery has moved to a new forum, I'm copy pasting this old feature suggestion that I have made in 2017:

https://www.backupcreations.com/forum/p ... html#p5485

I'd like to suggest a new feature for Syncovery.

Syncovery could implement Exact Mirror + Reverse Incremental sync operation mode.

An incremental backup of the changes made between two instances of a mirror is called a reverse incremental. By applying a reverse incremental to a mirror, the result will be a previous version of the mirror. In other words, after the initial full backup, each successive incremental backup applies the changes to the previous full, creating a new synthetic full backup every time, while maintaining the ability to revert to previous versions. The main advantage of this type of backup is a more efficient recovery process, since the most recent version of the data (which is the most frequently restored version) is a (synthetic) full backup, and no incrementals need to be applied to it during its restoration.

At first a full backup is made. But when the following backup is created, first all data from the full backup is moved to this new backup and the previous one is replaced with a small reverse incremental version. The old backup then contains only the differences to the following backup (in contrast to normal incremental systems, where each backup contains the differences to the preceding backup).

One effect of reverse incremental storage is that the newest backup is always a full backup.

Advantages

Faster restore:
With normal incremental systems the data of the newest backup is spread out over many backup versions. This slows down the restore process considerably. With reverse incremental storage this data is stored in one block in the latest backup version and fast access is possible.

Higher data safety:
The most valuable data of a backup is always the latest data. With normal incremental storage the latest backup depends on all preceding backups. If any one of these is damaged then all following backups will be broken, including the most recent one. In a reverse incremental system the newest backup is completely independent of all other backups. If an old backup is damaged then only the preceding backups are affected, but not the more valuable newer backups.

Simpler storage management:
With reverse incremental backups the oldest backup can safely be deleted at any point in time. The following backups are not affected by that at all. This enables fine-grained storage management that can also be automated easily. Even if the destination drive is completely full the solution is very simply: simply delete the oldest backup.

Here's a nice short animation highlighting the advantages and benefits of reverse incremental (decremental) backup algorithm:

https://www.youtube.com/watch?v=9PqZkM6RerU

Veeam Backup, which is a popular Enterprise backup solution, has this mode:

https://www.veeam.com/kb1933

Would it be possible for Syncovery to implement Exact Mirror + Reverse Incremental option?

Thank you.

Re: Feature suggestion: Exact mirror + Reverse Incremental Backup

Posted: Tue Mar 09, 2021 11:44 pm
by tobias
Hello,
many thanks for the reminder, I will look into this!

Re: Feature suggestion: Exact mirror + Reverse Incremental Backup

Posted: Tue Jan 04, 2022 9:57 pm
by Shaipetit
Dear Tobias,

Have you had a chance to look into this feature?

Re: Feature suggestion: Exact mirror + Reverse Incremental Backup

Posted: Thu Jan 13, 2022 4:20 pm
by tobias
Hello,
many thanks for the reminder. I hope to start working on Syncovery 10 in a few weeks, and depending on how difficult the feature it is, I may be able to add it. I do have at least 100 other requests and ideas though. We will see.

Re: Feature suggestion: Exact mirror + Reverse Incremental Backup

Posted: Tue Mar 12, 2024 11:12 pm
by Shaipetit
Dear Tobias,

Have you had a chance to look into this feature?

Here's a quick summary of this feature:
https://www.youtube.com/watch?v=9PqZkM6RerU

Re: Feature suggestion: Exact mirror + Reverse Incremental Backup

Posted: Wed Mar 13, 2024 11:08 am
by tobias
Hello,
thanks for posting the link!

Syncovery does exactly that, it keeps a 1:1 mirror and moves older files or deleted files to separate folders, depending on the Versioning settings and the settings for Deleted Files.

So you can always restore the mirror immediately, without any reassembly needed.

The Reverse Incremental option would only be a relevant addition for large files, where you want to do a block level backup. Interestingly, the video doesn't differentiate between large and small files at all.

Reverse Incremental for large files could only be used if the backup is not compressed or encrypted ... unless you are fine with the whole file being backed up every time ... so you don't care about bandwidth but only about storage space ... this is complicated, as it adds different operating modes within the new Reverse Incremental feature.

Which mode would you be using?

- a 1:1 mirror where only changed blocks are copied and bandwidth usage is reduced (no encryption, no compression)
this would work only if you copy to local drives, network drives, or via SFTP, as it needs block level access to the destination file

- or a full copy every time (if the file has been modified), which would allow for encryption and compression,
but use much more CPU and bandwidth;
this would actually require keeping a separate unencrypted local copy of the backup to create the delta against,
or it would require to restore and unpack the previous backup to a temporary file and build the delta against that;
both of these options are not great and I think I want to avoid implementing that ...

Syncovery's current options for large files and block level backup are documented here:
https://www.syncovery.com/block-level-copying/

I will consider this for Syncovery 11, which I am already working on, but the use case is very limited due to the considerations I mentioned.