How to improve performance with sync of large number of very small files?

English Support for Syncovery on Windows.
Post Reply
james_c_stein@fd.org
Posts: 14
Joined: Wed Oct 28, 2020 1:01 pm

How to improve performance with sync of large number of very small files?

Post by james_c_stein@fd.org »

I am seeing very poor performance while copying large numbers of very small files. I am talking hundreds of thousands of files that are less than 10k in size. Large files copy quickly but the small ones seem to take ages.

By comparison, normal/large files I can average 1GB per hour at my current throughput limit. These super small files I had a profile running 12 hours and it only managed to copy 128MB.

Is it maybe a buffer or caching setting that I am just not seeing?

tobias
Posts: 482
Joined: Tue Mar 31, 2020 7:37 pm

Re: How to improve performance with sync of large number of very small files?

Post by tobias »

Hello,
no, there is no buffer or caching that could make this faster. The only option is to copy more files in parallel (use more threads). The number is specified on the Files tab sheet in the profile.

If problems remain, please mention more details, such as where you are copying from and to, and which protocols you are using (for example, UNC paths or FTP or which cloud etc.).

To speed up building the file list (before the copying phase), please see
https://www.syncovery.com/documentation/faq/fastlist/

Post Reply