Can I generate cleaner log files?

[IMTheNachoMan]

I have a Left to Right job backing up files from my computer to B2.

Right now the log file in /etc/.Syncovery/logs are very detailed but very difficult to read.

I'm wondering if I can somehow generate some individual log files:

1. A text file with just a list of files that will be copied from Left to Right
2. A text file with just a list of files that will be deleted from the right
3. A text file with transactions that errored -- and just the path to the file

If #1 and #2 need to be in the same file but are prefixed with some string like "transfer" and "delete", that would be okay too.

[tobias]

Hello,
there is a feature to create action oriented logs in csv format.

To activate it, create a small INI style text file like this:

[Main]
ActionsCSVLogFile=/home/tobias/logs/$PROFILENAME.txt

You can also use variables like $DATE and $TIME if you want separate logs for each day or profile run.
See
https://www.syncovery.com/variables/

Then this text file needs to be imported via the Program Settings dialog -> Import Config Lines (INI Style).

As an alternative, a completely custom log file can be programmed via PascalScript using the OnActionComplete hook and the function AtomicAppendTextFileLine.

https://www.syncovery.com/pascalscript/

Finally, I might be going to add a cleaner log file mode, maybe you can tell me exactly which lines in the log are disturbing you.

[IMTheNachoMan]

there is a feature to create action oriented logs in csv format.

Thanks. Let me try this.

Finally, I might be going to add a cleaner log file mode, maybe you can tell me exactly which lines in the log are disturbing you.

For me, seeing multiple lines for one file is clunky and cumbersome to look at. For example, here is an excerpt from my last run:

Code: Select all

Copy L->R /home/myHome/Downloads/openSUSE-Tumbleweed-NET-x86_64-Snapshot20230613-Media.iso  (223.0MB -> 209.1MB)
Still Working At 2023-06-17 23:33:29
Still Working At 2023-06-17 23:33:59
Still Working At 2023-06-17 23:34:29
Still Working At 2023-06-17 23:34:59
Still Working At 2023-06-17 23:35:29
Still Working At 2023-06-17 23:35:59
Still Working At 2023-06-17 23:36:29
Still Working At 2023-06-17 23:36:59
Still Working At 2023-06-17 23:37:29
Encrypting (AES256) /home/myHome/Pictures/misc/2023-01-09 - stuff/Movie/2023_0109_085312_PF.MP4 to /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140573262472896-361391-935229730.zip
Zipping error for /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140573262472896-361391-935229730.zip, OS error code: 0, VolumeNumber: 0, VolumeFileName: /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140573262472896-361391-935229730.zip, disk full (A).
Writing error during zipping
Zipping error for /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140573262472896-361391-935229730.zip, OS error code: 28, VolumeNumber: 0, VolumeFileName: /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140573262472896-361391-935229730.zip, disk full (A).
Zipping result false, InUpdate=FALSE, ProcessedFileCount=2, AddedFileCount=2, TimeSkippedCount=0, OnlyIfNewer=FALSE, SourceFileSpecificErrorOccurred=FALSE, GeneralZIPErrorOccurred=TRUE, JustCanceled=TRUE
Unknown Unix Error Code 28
Unknown Unix Error Code 28
23:37:55.105 Error adding to zip: /home/myHome/Pictures/misc/2023-01-09 - stuff/Movie/2023_0109_085312_PF.MP4
Exception closing compressed archive file /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140573262472896-361391-935229730.zip: Operation Canceled
Unknown Unix Error Code 28
Error while zipping /home/myHome/Pictures/misc/2023-01-09 - stuff/Movie/2023_0109_085312_PF.MP4, Result: 217. Operation Canceled, Source File Exists: Unknown Unix Error Code 28
TRUE
Encrypting (AES256) /home/myHome/Pictures/misc/narrative.tar.gz to /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140572996400832-312968-769800975.zip
Zipping error for /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140572996400832-312968-769800975.zip, OS error code: 0, VolumeNumber: 0, VolumeFileName: /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140572996400832-312968-769800975.zip, disk full (A).
Writing error during zipping
Zipping error for /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140572996400832-312968-769800975.zip, OS error code: 28, VolumeNumber: 0, VolumeFileName: /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140572996400832-312968-769800975.zip, disk full (A).
Zipping result false, InUpdate=FALSE, ProcessedFileCount=2, AddedFileCount=2, TimeSkippedCount=0, OnlyIfNewer=FALSE, SourceFileSpecificErrorOccurred=FALSE, GeneralZIPErrorOccurred=TRUE, JustCanceled=TRUE
Unknown Unix Error Code 28
Unknown Unix Error Code 28
23:37:55.128 Error adding to zip: /home/myHome/Pictures/misc/narrative.tar.gz
Exception closing compressed archive file /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140572996400832-312968-769800975.zip: Operation Canceled
Unknown Unix Error Code 28
Error while zipping /home/myHome/Pictures/misc/narrative.tar.gz, Result: 217. Operation Canceled, Source File Exists: Unknown Unix Error Code 28
TRUE
Encrypting (AES256) /home/myHome/Pictures/misc/narrative_no_jpgs.tar.gz to /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140572954252992-328497-576150062.zip
Zipping error for /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140572954252992-328497-576150062.zip, OS error code: 0, VolumeNumber: 0, VolumeFileName: /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140572954252992-328497-576150062.zip, disk full (A).
Writing error during zipping
Zipping error for /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140572954252992-328497-576150062.zip, OS error code: 28, VolumeNumber: 0, VolumeFileName: /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140572954252992-328497-576150062.zip, disk full (A).
Zipping result false, InUpdate=FALSE, ProcessedFileCount=2, AddedFileCount=2, TimeSkippedCount=0, OnlyIfNewer=FALSE, SourceFileSpecificErrorOccurred=FALSE, GeneralZIPErrorOccurred=TRUE, JustCanceled=TRUE
Unknown Unix Error Code 28
Unknown Unix Error Code 28
23:37:55.160 Error adding to zip: /home/myHome/Pictures/misc/narrative_no_jpgs.tar.gz
Exception closing compressed archive file /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140572954252992-328497-576150062.zip: Operation Canceled
Unknown Unix Error Code 28
Error while zipping /home/myHome/Pictures/misc/narrative_no_jpgs.tar.gz, Result: 217. Operation Canceled, Source File Exists: Unknown Unix Error Code 28
TRUE
Still Working At 2023-06-17 23:37:59
Still Working At 2023-06-17 23:38:29
Still Working At 2023-06-17 23:38:59
Still Working At 2023-06-17 23:39:29
Still Working At 2023-06-17 23:39:59
Still Working At 2023-06-17 23:40:29
Still Working At 2023-06-17 23:40:59
Still Working At 2023-06-17 23:41:29
Still Working At 2023-06-17 23:41:59

Now, this is super helpful and userful information, but it's difficult to read and see details/errors/issues for one file. I get that Syncovery does parallel transfers so things are gonna be inter-mingled, but I feel like this could be handled with different log files.

What makes the most sense to me is:

1. One log file that just lists one file per row/line with each line having this data. The bold ones are important, the rest are optional/nice-to-have.
   1. Start date/time of the transfer
   2. End date/time of the transfer
   3. Left connector details
   4. Left path
   5. Right connector details
   6. Right path
   7. Transfer status like complete/success or error/failed
   8. Error code/reason
2. An error file that grouped error logs/details for each file. So there would be a line with the file details like above, and then 1+ lines with more verbose details. Then a few blank lines, or some kind of divider, and then the next file error details. In fact, this detail could be added to the file above as an obtional logging level.

Here is an example of what I mean:

Code: Select all

2023-01-01 12:34:00 | 2023-01-01 13:14:00 | file system @ /home/myHome | /some/folder/file.txt | B2 @ ext://bucket-name/myHome | /some/folder/file.txt.s1234.zip | success | completed
2023-01-01 12:35:00 | 2023-01-01 13:00:00 | file system @ /home/myHome | /some/other/filee.txt | B2 @ ext://bucket-name/myHome | /some/other/filee.txt.s4567.zip | error   | temp folder disk full
2023-01-01 13:14:00 | 2023-01-01 13:15:00 | file system @ /home/myHome | /some/other/fileb.txt | B2 @ ext://bucket-name/myHome | /some/other/fileb.txt.s1234.zip | success | completed

And, with more verbose logging:

Code: Select all

2023-01-01 12:34:00 | 2023-01-01 13:14:00 | file system @ /home/myHome | /some/folder/file.txt | B2 @ ext://bucket-name/myHome | /some/folder/file.txt.s1234.zip | success | completed
2023-01-01 12:35:00 | 2023-01-01 13:00:00 | file system @ /home/myHome | /some/other/filee.txt | B2 @ ext://bucket-name/myHome | /some/other/filee.txt.s4567.zip | error   | temp folder disk full
Encrypting (AES256) /some/other/filee.txt to /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140573262472896-361391-935229730.zip
Zipping error for /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140573262472896-361391-935229730.zip, OS error code: 0, VolumeNumber: 0, VolumeFileName: /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140573262472896-361391-935229730.zip, disk full (A).
Writing error during zipping
Zipping error for /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140573262472896-361391-935229730.zip, OS error code: 28, VolumeNumber: 0, VolumeFileName: /var/run/.syncoverytemp/Backup 5442.140573636773568.1.140573262472896-361391-935229730.zip, disk full (A).
Zipping result false, InUpdate=FALSE, ProcessedFileCount=2, AddedFileCount=2, TimeSkippedCount=0, OnlyIfNewer=FALSE, SourceFileSpecificErrorOccurred=FALSE, GeneralZIPErrorOccurred=TRUE, JustCanceled=TRUE
2023-01-01 13:14:00 | 2023-01-01 13:15:00 | file system @ /home/myHome | /some/other/fileb.txt | B2 @ ext://bucket-name/myHome | /some/other/fileb.txt.s1234.zip | success | completed

There are a few benefits to doing it this way:

• It is easier to look through the log file, line by line, to see what errored.
• With some data manipulation, you can export the "tabluar" data to a CSV to open it in a spreadsheet program

I am trying you ActionsCSVLogFile to see what kind of logs it outputs. If it does not work for my needs, I will see if I can get this working with the OnActionComplete function.

[IMTheNachoMan]

there is a feature to create action oriented logs in csv format.

This does not seem to log errors.

Some of my files didn't backup cause the temp folder (for zipping) is full, but those errors did not log in the ActionsCSVLogFile file.

[IMTheNachoMan]

there is a feature to create action oriented logs in csv format.

It looks like neither ActionsCSVLogFile or OnActionComplete log errors.

I just did a run and the main log file shows some errors, but neither were logged by the above.

I used ActionsCSVLogFile like you told and this is my OnActionComplete script:

Code: Select all

function OnActionComplete(  const StartTimeUTC, CompletionTimeUTC: Double;
                            const Success: Boolean;
                            const ActionStr, DirectionStr, Filename, LeftFile, RightFile, Subfolder, MovedTo, ErrorMsg: UnicodeString;
                            const ResultCode: Int64;
                            const ErrorSide: UnicodeString;
                            const ASize, ACompressedSize:Int64):Boolean;
begin
    
    AtomicAppendTextFileLine('/etc/.Syncovery/Logs/test.txt', IntToStr(Integer(Success)) + ''#9'' + ActionStr + ''#9'' + DirectionStr + ''#9'' + Filename + ''#9'' + LeftFile + ''#9'' + RightFile + ''#9'' + Subfolder + ''#9'' + MovedTo + ''#9'' + ErrorMsg + ''#9'' + IntToStr(ResultCode) + ''#9'' + ErrorSide)
    Result:=true;
end;