Page 1 of 1

How to improve performance with sync of large number of very small files?

Posted: Tue Apr 13, 2021 2:15 pm
by james_c_stein@fd.org
I am seeing very poor performance while copying large numbers of very small files. I am talking hundreds of thousands of files that are less than 10k in size. Large files copy quickly but the small ones seem to take ages.

By comparison, normal/large files I can average 1GB per hour at my current throughput limit. These super small files I had a profile running 12 hours and it only managed to copy 128MB.

Is it maybe a buffer or caching setting that I am just not seeing?

Re: How to improve performance with sync of large number of very small files?

Posted: Tue Apr 13, 2021 7:06 pm
by tobias
Hello,
no, there is no buffer or caching that could make this faster. The only option is to copy more files in parallel (use more threads). The number is specified on the Files tab sheet in the profile.

If problems remain, please mention more details, such as where you are copying from and to, and which protocols you are using (for example, UNC paths or FTP or which cloud etc.).

To speed up building the file list (before the copying phase), please see
https://www.syncovery.com/documentation/faq/fastlist/