How to improve performance with sync of large number of very small files?
Posted: Tue Apr 13, 2021 2:15 pm
I am seeing very poor performance while copying large numbers of very small files. I am talking hundreds of thousands of files that are less than 10k in size. Large files copy quickly but the small ones seem to take ages.
By comparison, normal/large files I can average 1GB per hour at my current throughput limit. These super small files I had a profile running 12 hours and it only managed to copy 128MB.
Is it maybe a buffer or caching setting that I am just not seeing?
By comparison, normal/large files I can average 1GB per hour at my current throughput limit. These super small files I had a profile running 12 hours and it only managed to copy 128MB.
Is it maybe a buffer or caching setting that I am just not seeing?