Log file

English Support for Syncovery on Windows.
Post Reply
jfbradfo
Posts: 4
Joined: Fri May 27, 2022 5:58 pm

Log file

Post by jfbradfo »

We are using Syncovery for a share synchronization need involving file shares that may be TBs in size and may contain millions of files. We have a need to capture some additional information from the logs that is collected during the file list building process (specifically catching files that are older on the left side) for inspection and manual remediation, if necessary.

Unfortunately, we discovered what look like two issues that impact the usage:
  • The information generated from the start of list building to the end of the file copy/move phase appears to be maintained in a circular buffer in memory, so we’re losing the list building part of the process, as well as at least some of the copy information
  • The log files seem to have a ~2GB limit, so we’re losing any copy information past that limit, as well as the sync summary data.
I found a reference to the AbsMaxMemory setting in the the INI file, but I'm not sure this will help. I also see a setting called LogCheckpoints, but I don't know what it really does.

What am I missing? How can I ensure I get all of the list building and copy/move data in the log for these large shares?

tobias
Posts: 1047
Joined: Tue Mar 31, 2020 7:37 pm

Re: Log file

Post by tobias »

Hello,
thanks for your post in our forum!

There isn't a 2GB limit, that just depends on your text editor. There isn't any circular buffer either. You just need a good editor like Notepad++ for large files. However indeed we do not guarantee that everything is logged if you turn on excessive logging (see further below).

You can split jobs into smaller parts on the Memory tab sheet of the Program Settings dialog. For example, you can let it split the job (and log file) after 1 million files or less.

But I don't think that such detailed and large log files are very useful. Please remove any checkmarks in the "Additional Logging (for troubleshooting)" section from the Logs tab sheet of the Program Settings dialog. Newer Syncovery versions remove most of these checkmarks automatically after 24 hours anyway (but that can be disabled with a special line in Syncovery.ini).

If you use Exact Mirror, but disallow replacing newer files under Safety->Unattended Mode, then the log file will contain an error message for each file that is newer on the wrong side of the sync.

In addition, you can catch some information with a PascalScript. You can log it to the main profile log file, or to separate files like csv files.

Please see
https://www.syncovery.com/pascalscript/

However, I believe that we may have to extend the hooks to fulfill your requirements exactly. Custom script related development is available as part of our Premium Support offering.

jfbradfo
Posts: 4
Joined: Fri May 27, 2022 5:58 pm

Re: Log file

Post by jfbradfo »

What we are seeing is log files that are almost exactly 2GB, and where the file list building detail is missing, the file action begins in the middle of a file name (so we can't see what came before), and the log ends in the middle of an action entry with no summary data. The only additional Log setting we enable is File List Building Detail, so I wouldn't expect that to create "excessive" logging. However, with 400000+ files with actions, I guess it could be.

Here are a couple of examples of what we see:
-a---- 5/27/2022 3:40 PM 2147540449 3_CTE - 2022-05-27 13.21.30 - 435384 copied (621.5GB), 3477 renamed.log
-a---- 5/26/2022 7:00 PM 2149779767 3_CTE_Debug - 2022-05-26 16.24.39 - INCOMPLETE. 529135 copied (707.7GB) of 529136, 9974 renamed, Err.log

This is not an editor issue, this is a file size issue. And we don't intend to use an editor at all with these, but rather to parse out and re-format the data in the logs with Powershell.

I will give the Exact Mirror with the safety options a try, but I have a couple of questions:
  • If this doesn't work, we'll need to keep doing what we are doing. Please provide the INI setting that will allow us to keep the Log settings permanent.
  • Since I don't have an easy way to test with the production data, how can I run a "what if" that generates the list of actions that would be taken, but without taking them?
Thanks for the help!

tobias
Posts: 1047
Joined: Tue Mar 31, 2020 7:37 pm

Re: Log file

Post by tobias »

Yes, "File List Building Details" is the most excessive logging that Syncovery offers. It is intended only for debugging and not for any real use. It is not intended for large sync jobs.

To force Syncovery to keep it on, add this to the [Main] section of Syncovery.ini:
KeepDebugLoggingOptions=1

You should run tests against test data.

If you use "Exact Mirror" but disallow replacing newer files with older ones under Safety->Unattended Mode, then newer files will not be replaced.

What you might need is an extended PascalScript hook that gives you the information you need and you can log it using the PascalScript Log function.

jfbradfo
Posts: 4
Joined: Fri May 27, 2022 5:58 pm

Re: Log file

Post by jfbradfo »

Thanks again for the information. I have setup the "Exact Mirror" as described below, but now what I see are confirmation errors and the sync is always marked incomplete, even though the intent is being met. We have all newer files copied (and older renamed if necessary), new files are copied, and older files are NOT replaced.

How do I avoid the incomplete sync error?

tobias
Posts: 1047
Joined: Tue Mar 31, 2020 7:37 pm

Re: Log file

Post by tobias »

Hello,
I think you need to live with this INCOMPLETE result. It tells you that there are older files on the wrong side, which you will probably want to fix anyway?

However if you do not want to see this, then the only way is to tell Exact Mirror not to replace newer files with older ones. Or use Standard Copying. But then you won't see anything in the log file about these files.

jfbradfo
Posts: 4
Joined: Fri May 27, 2022 5:58 pm

Re: Log file

Post by jfbradfo »

I agree that we'll have to live with that for now. This is a short-term need and so long as there aren't actual errors in the result (access, etc), this should be fine.

We are generating a clean list from the log file in a CSV format for the users/data owners to review and manually remediate those that they require.

I don't know why I didn't ask (and this should probably be a new thread), but is there a way natively (without PascalScript) to version with older files on the left? That is, if the file is older on the left, to copy it to a new name (timestamped) similarly to versioning of older on the right.

Thanks again for the help!

tobias
Posts: 1047
Joined: Tue Mar 31, 2020 7:37 pm

Re: Log file

Post by tobias »

Hello,
versioned files can be older or newer, but the point is that the versioning is always done on the destination side. Can be both left or right. But it's on the destination side of the copy.

tobias
Posts: 1047
Joined: Tue Mar 31, 2020 7:37 pm

Re: Log file

Post by tobias »

Hello,
FYI, I am enhancing the PascalScript language in the next update so that your requirements could be implemented exactly, should you choose to go that route.

Post Reply