"Number of files to copy in parallel" has no effect on upload speed to OneDrive

English Support for Syncovery on Windows.
Redwood
Posts: 46
Joined: Wed Jun 02, 2021 12:44 pm

"Number of files to copy in parallel" has no effect on upload speed to OneDrive

Post by Redwood »

I'm testing Syncovery trial version x64 9.35e.

1) Upload speed to OneDrive for Business is the same for any value of "Number of files to copy in parallel" field from 3 up to 10
I have 100 Mbit/sec internet connection. Upload speed is about 5 Mb/sec.
But Air Explorer and Cyberduck can do about 8.5 Mb/sec with 10 threads and 11 Mb/sec with 15 threads.
Why Syncovery does not react to changes in "Number of files to copy in parallel" field?

2) When I try to increase the number above 10 I get an error message "The maximum number of files to copy in parallel with this cloud storage is 10"
Why is that?

tobias
Posts: 1603
Joined: Tue Mar 31, 2020 7:37 pm

Re: "Number of files to copy in parallel" has no effect on upload speed to OneDrive

Post by tobias »

The number is actually limited to 3 internally because we have experienced that anything above that will cause penalty delays from the cloud servers, especially for small files. If you upload large files, I guess the number could be increased. I will think about that. What’s your average file size?

Redwood
Posts: 46
Joined: Wed Jun 02, 2021 12:44 pm

Re: "Number of files to copy in parallel" has no effect on upload speed to OneDrive

Post by Redwood »

tobias wrote:
Wed Jun 02, 2021 1:35 pm
The number is actually limited to 3 internally
So why does Syncovery say "The maximum number of files to copy in parallel with this cloud storage is 10"?
Could you remove that limitation? (To leave a user to decide himself)
tobias wrote:
Wed Jun 02, 2021 1:35 pm
anything above that will cause penalty delays from the cloud servers
Which cloud servers?
I use OneDrive for Business.
I did not have any penalty delays while using Air Explorer or Cyberduck with 10-15 threads.
tobias wrote:
Wed Jun 02, 2021 1:35 pm
What’s your average file size?
5-30 Мб

I need to decide which cloud backup software to buy to backup my 5 Tb data on the clouds.
Syncovery is great software and has more features. But Air Explorer is significantly faster because of number of threads.
As for me upload speed is more important than numerous features.
It seems Air Explorer is better for the moment.

tobias
Posts: 1603
Joined: Tue Mar 31, 2020 7:37 pm

Re: "Number of files to copy in parallel" has no effect on upload speed to OneDrive

Post by tobias »

Hello,
thanks for the info, I will remove the limitation in a Syncovery update tomorrow.

Redwood
Posts: 46
Joined: Wed Jun 02, 2021 12:44 pm

Re: "Number of files to copy in parallel" has no effect on upload speed to OneDrive

Post by Redwood »

Great news!

tobias
Posts: 1603
Joined: Tue Mar 31, 2020 7:37 pm

Re: "Number of files to copy in parallel" has no effect on upload speed to OneDrive

Post by tobias »

Hello,
I'm working on it now; I am planning two changes:
a) more generous default limits
b) a global checkbox to remove all limits

Redwood
Posts: 46
Joined: Wed Jun 02, 2021 12:44 pm

Re: "Number of files to copy in parallel" has no effect on upload speed to OneDrive

Post by Redwood »

There is "Maximum Upload Threads per File" option in "Program Settings".
How does it work with "Number of files to copy in parallel" option?
Does it work for clouds only or for local/mapped/network drives too?

tobias
Posts: 1603
Joined: Tue Mar 31, 2020 7:37 pm

Re: "Number of files to copy in parallel" has no effect on upload speed to OneDrive

Post by tobias »

Hello,
this option can split huge files in parts and upload several parts simultaneously. It applies only for very large files being uploaded to Amazon S3, Microsoft Azure, Amazon Glacier, and Backblaze B2.

All other large transfers are single-threaded per file.

The option has nothing to do with "Number of files to copy in parallel".

tobias
Posts: 1603
Joined: Tue Mar 31, 2020 7:37 pm

Re: "Number of files to copy in parallel" has no effect on upload speed to OneDrive

Post by tobias »

Hello,
the new Syncovery version 9.36 includes these improvements:
- allows up to 12 files being uploaded in parallel to Sharepoint and OneDrive by default
- adds a new checkbox to the Performance tab sheet of the Program Settings dialog: “Remove default threading limits and other program limits”

Redwood
Posts: 46
Joined: Wed Jun 02, 2021 12:44 pm

Re: "Number of files to copy in parallel" has no effect on upload speed to OneDrive

Post by Redwood »

Another problem with the "Number of files to copy in parallel" option.
For some reasons I use CloudMounter to mount one of my OneDrive cloud storage as local drive.
In Syncovery I've created new profile with default settings.
"Left-Hand Side" - local folder
"Right-Hand Side" - folder on cloud mapped drive
Local folder contains 1 folder with several files inside.
After I started backup task I get a window with error message - Unable create existing folder. Error creating folder on the destination volume.
There is three options below with "Don't retry this file" option selected.
After I press OK button I get the same windows again.
Again press OK button and the backup task continues.
The problem is gone after I set "Number of files to copy in parallel" to 1.
It seems when first thread starts it creates a folder but other threads don't know about that.

Log:

Code: Select all

Log for profile Test 1, started at 0:31:01 on 04.06.2021

LEFT: D:\Temp\test
RIGHT: Z:\Temp\test

The profile is run in attended mode.

Previous run was at 04.06.2021 0:25:06: 12 copied (44,2MB)

Left Side Volume Name:                Data
Right Side Volume Name:               OneDrive For Business OneDrive

Copying Direction : Left To Right

Scanning Folders Started At 04.06.2021 0:31:01

Free Space on left side :       454 GB, needed:    0 Bytes
Free Space on right side:    20 467 GB, needed:  45 211 kB

Created folder: Z:\Temp\test\1
Unable to create folder Z:\Temp\test\1, error code 183: Невозможно создать файл, так как он уже существует.
Using Internal Copying Function
CreateFile Z:\Temp\test\1\1.tif, failed with error 31: Присоединенное к системе устройство не работает.
Unable to create folder Z:\Temp\test\1, error code 183: Невозможно создать файл, так как он уже существует.
Copy L->R D:\Temp\test\1\1.tif  (3,7MB)
Copy L->R D:\Temp\test\1\11.tif  (3,7MB)
CreateFile Z:\Temp\test\1\10.tif, failed with error 31: Присоединенное к системе устройство не работает.
Unable to create folder Z:\Temp\test\1, error code 183: Невозможно создать файл, так как он уже существует.
Copy L->R D:\Temp\test\1\10.tif  (3,7MB)
Copy L->R D:\Temp\test\1\2.tif  (3,7MB)
Copy L->R D:\Temp\test\1\3.tif  (3,7MB)
Copy L->R D:\Temp\test\1\4.tif  (3,7MB)
Copy L->R D:\Temp\test\1\12.tif  (3,7MB)
Copy L->R D:\Temp\test\1\5.tif  (3,7MB)
Copy L->R D:\Temp\test\1\7.tif  (3,7MB)
Copy L->R D:\Temp\test\1\8.tif  (3,7MB)
Copy L->R D:\Temp\test\1\9.tif  (3,7MB)
Copy L->R D:\Temp\test\1\6.tif  (3,7MB)

SUMMARY
------------------------------------------------------------------
Short Results: 12 copied (44,2MB)
Operation completed at 0:31:26 on 04.06.2021
Total duration: 00:00:25

Post Reply