Hello
We are running a Syncovery instance on a Windows server EC2 in AWS which is likely way overkill.
We would like to optimize our costing and move to a docker instance running in ECS Fargate instance but we would need advices on sizing the instance in terms of vCPU, RAM and local storage
Our typical usage includes transferring everyday between 100 to 500 large video files from 1 GB to 50GB (occasionally up to 100GB)
Do you have some guidelines to help us with that ?
When moving files, are the files temporarily stored locally ? so should we account for local storage able to cope with the potentially large video files tat we are transferring ?
Let us know if you have any question to help with that
Best regards
Syncovery on AWS ECS Fargate - instance sizing
-
Jean-Michel
- Posts: 15
- Joined: Fri Jun 10, 2022 2:12 pm
Re: Syncovery on AWS ECS Fargate - instance sizing
Hello,
it depends very much on settings and on which source and destination you are copying from/to, and if you are compressing and/or encrypting the files.
To get a rough idea, it would be good to simply watch the system usage in Task Manager when Syncovery runs. Does it use a lot of RAM and CPU or not? It should give you an idea of what you need. In some cases, Syncovery can perform well with very little RAM and CPU, but in other cases, it may need some more.
Can you share more details on your settings and the source and destination paths or protocols?
For example, if you are copying from a local path or UNC path to Amazon S3 or SFTP etc.?
it depends very much on settings and on which source and destination you are copying from/to, and if you are compressing and/or encrypting the files.
To get a rough idea, it would be good to simply watch the system usage in Task Manager when Syncovery runs. Does it use a lot of RAM and CPU or not? It should give you an idea of what you need. In some cases, Syncovery can perform well with very little RAM and CPU, but in other cases, it may need some more.
Can you share more details on your settings and the source and destination paths or protocols?
For example, if you are copying from a local path or UNC path to Amazon S3 or SFTP etc.?
-
Jean-Michel
- Posts: 15
- Joined: Fri Jun 10, 2022 2:12 pm
Re: Syncovery on AWS ECS Fargate - instance sizing
Hi Tobias
The transfers would mainly occurs between our side and customer's side, in one or the other direction
- Our side would be AWS S3 in the same region. The instance would run in our VPC so we would need to be sure that it uses our existing S3 Gateway endpoints in order to avoid egress
- Customer's side can be
- Other S3 buckets in the same region
- Other cloud storage in GCP or Azure anywhere
- On-prem using SFTP
There would be no local path
This is why it is critical to understand if when a file is moved it is first downloaded locally and then re-uploaded, or if this is always directly piped through.
The transfers would mainly occurs between our side and customer's side, in one or the other direction
- Our side would be AWS S3 in the same region. The instance would run in our VPC so we would need to be sure that it uses our existing S3 Gateway endpoints in order to avoid egress
- Customer's side can be
- Other S3 buckets in the same region
- Other cloud storage in GCP or Azure anywhere
- On-prem using SFTP
There would be no local path
This is why it is critical to understand if when a file is moved it is first downloaded locally and then re-uploaded, or if this is always directly piped through.
Re: Syncovery on AWS ECS Fargate - instance sizing
Hello,
in these cases, Syncovery has to use temporary files on the machine where it's running. It will download the files to the TEMP folder and then upload them to the other side.
So you will need some generous hard disk space.
You should also from time to time check the TEMP folder to make sure there aren't any large temporary files left over from broken transfers. It should not happen, but you should check anyway.
Depending on the number of jobs running simultaneously, and the number of simultaneous transfers within a job, you may also need a bit more RAM than for minimal tasks. Please check memory usage in Task Manager.
in these cases, Syncovery has to use temporary files on the machine where it's running. It will download the files to the TEMP folder and then upload them to the other side.
So you will need some generous hard disk space.
You should also from time to time check the TEMP folder to make sure there aren't any large temporary files left over from broken transfers. It should not happen, but you should check anyway.
Depending on the number of jobs running simultaneously, and the number of simultaneous transfers within a job, you may also need a bit more RAM than for minimal tasks. Please check memory usage in Task Manager.