Feature Request: Test file presence / absence before running profile

[TBacker]

I'm not sure if this has much widespread utility, but I'll throw it up here anyway.

I have a profile that replicates a folder between main and backup servers. This folder is full of Microsoft Jet (access) database files that a few clients on the network access. Database clients are only accessing one server (the main) - no changes are normally being made to the secondary server.

While it's never advisable to use a jet based database on a network share due to it's lack of record level locking (and easily corruptible nature), the software manufacturer (and our users) insist that it be this way.

Anyway, I had to disable the scheduled profile in Syncovery as there were occasional file locking contentions between the DB client software and Syncovery copy operations. The client software doesn't handle this gracefully at all. What I ended up having to do is a mess - I have a batch script triggered every 5 minutes by Windows task scheduler. It looks for .ldb files which the jet engine uses to indicate an open database / lock. If there are no ldb files, it tells the SyncoveryService to run the appropriate profile (which I have disabled scheduling on in Syncovery).

It sure would be cool if there was a scheduling option internal to Syncovery to "Abort profile if file exists / is missing".

[tobias]

Hello,
this can be done via PascalScript, for example

Code: Select all

function OnGetProfilePathBeforeListing(const isRightSide:Boolean;
        var Path,UserName,Password:UnicodeString;
        var AuthKey,AuthKeyPassword:AnsiString;
        var Port:Integer):Boolean;
begin
  Result:=not FileExists('C:\Source\Stopflag.txt');
  end;

The script can be pasted into the profile via the PascalScript checkmark on the Job tab sheet.

See also
https://www.syncovery.com/pascalscript/

[TBacker]

Thanks Tobias!

Well now I feel like an idiot - didn't even see that capability. I did a little searching, but didn't think about Syncovery having internal scripting. This thing is so feature rich that it's easy to pass right over something!

I know nothing about Pascal. Will that sample function work with wildcards ('C:\Source Folder\*.ldb')?

[tobias]

Hello,
no, it doesn't work with wildcards, but I can add such a feature in 9.42.

[tobias]

Hello,
in Syncovery 9.42, you can now use FileExistsMatching instead of FileExists to allow the use of wildcards.

Code: Select all

function OnGetProfilePathBeforeListing(const isRightSide:Boolean;
        var Path,UserName,Password:UnicodeString;
        var AuthKey,AuthKeyPassword:AnsiString;
        var Port:Integer):Boolean;
begin
  Result:=not FileExistsMatching('C:\Source Folder\*.ldb');
  end;

In addition, to avoid any file locking conflicts, you can make sure Syncovery copies from a volume shadow instead of the live data. To do this, make sure Syncovery is running on the same machine (the server where the database is running and where the files are local), and under "Access & Retries", choose:

Volume Shadow: use for all, create shadow before building file list

However, volume shadows should not be used too frequently, since their creation alone can cause Windows to use CPU and disk more than normally for the time it takes to create the shadow. Which can be between 10 seconds and a minute, usually.

So it could be that you want to avoid volume shadowing, if the script solution works well.

[TBacker]

Wow! That was fast!! I'm guessing you were about to push an update anyway. Thank you.

The PascalScript is working as intended. I had a problem with the profile (which has real time monitoring enabled) restarting endlessly while the script returned false (aka an .ldb file is present), but I figured out it was because of my retry settings.

[TBacker]

Well, I had to retrograde back to 9.38d.

9.42 breaks all 4 of my profiles that use real time monitoring. Even though they all start up correctly, do the initial sync, and go into real time monitoring mode, they all eventually end up at "waiting for availability" permanently. Restarting the services makes no difference - it ends back up at waiting for availability. I verified the service is still running as the correct user. No profiles have custom network logins. The profiles can be run manually.

I tested on 9.41b (without the new script function) with the same results.

The same exact profile settings (single ini file, unchanged) work fine after the downgrade to 9.38d.

It's pretty late here now, so I'll leave it in on this version / config until tomorrow or a later time when I can investigate more. I'll look at the change log to see what has changed since 9.38d. I'll forward logs and configs tomorrow if you like (I backed up the program data before retrograde).

[tobias]

Hello,
that is strange because there have not been any changes in the realtime area since 9.38d.

Yes please send the Syncovery_realtime_status_log.txt file to support@syncovery.com.

[tobias]

Hello,
I've done more testing and can't confirm this issue. There haven't been any similar reports either. I think it was a singular occurrence. Maybe the errors were even legitimate because the other machine rebooted and was really offline for a while or something like that.

[TBacker]

I'm going to replicate it with logs, but am waiting for some users to wrap up their work so that the profile that handles those databases can run too.

It ran fine all night on 9.38d with the original settings.

I saved a copy of the \Program Data\Syncovery for reference, then upgraded to 9.42 without touching anything else. That combination has been running normally while I wait for those DB users to be done for the day.

My next step will be to shut down Syncovery and it's services, make another "snapshot" copy of the program data, start it all back up, and change the database folder replication profile to run with the PascalScript rather than being started externally.