ListingRequest

[adzlaw]

I have a basic WebDav written in XQuery. At some stage in the future, I would like to listen for a listing.request and return a number of binary files in response. A typical WEBDav listing in XML would be too big. Ideally I would to respond with a number of zipped files in some relatively small format eg JSON Zipped. Is there a specification for the response somewhere. I need to know what format to compress in ... the files are big due to the large number of small files (emls etc). Hopefully this is possible to use ZIP. I am using windows.

[tobias]

Hello,
if possible, it would be easier to just install our Syncovery Remote Service on the server. It will do all the work.

https://www.syncovery.com/remoteservice/

The WebDAV protocol does not need to be enhanced at all, because the communication works via files. Syncovery uploads a "listing request file", which the Remote Service sees and produces the listing file for Syncovery to download.

The Remote Service is highly optimized with multi-threading and a binary listing format that's also compressed. It's really the best choice for this purpose.

[adzlaw]

Hi Tobias,

The WebDav is custom inside an XML engine/database for indexing EML (Custom Http endpoints providing WEBDav functionality). The remote service cannot see the XML files as they are not written to disk (several 100's of GB).

I can see the listing request file

SPLITAFTER:2000000
ZIPOK
INCLUDE:*
EXCLUDE:\$RECYCLE.BI*,"Temporary Internet Files",DfsrPrivate,*.DS_Store,Thumbs.db,*.$$$,*.$syncovery,"Incoming Syncovery Packages"
EXCLUDEITEMS:RECYCLER,"SYSTEM VOLUME INFORMATION"
ONLYSELECTED
NEWINDEXEDFORMAT
WITHLOG
Syncovery.listingrequest.$$$.end

I would like to respond to the request in say X * 100mb files. Basically all the data will be grabbed from the Database XQueried -> XML and then serialized to say JSON or XSLTed/Xqueried to generate the response file(s) for Syncovery. If possible I would like to know how to respond in the NEWINDEXEDFORMAT etc. I can also respond in 1 file if that is how Syncovery works

I am having to deal with lots and lots of EML files (processed to XML for various reasons including special identifiers / properties ) and expect to battle with serialization speed versus filesize for the response. I am also using GO (along with Xquery/XSLT). I could send XMLEngine info -> GO (or another language) (for special processing)-> XMLEngine -> Syncovery.

The XML engine/database is a basically a store with the file names, dates and sizes with a custom webdav (Custom Http endpoints providing WEBDav functionality). The endpoints for another HTTP protocol could be coded (eg replicate say Drive https://developers.google.com/drive/api ... files/list). I am thinking that using WEBDAV with a Syncovery.listingrequest is the best way to go. WebDav is a commonly used and community protocol. It is unfortunate that it responds in such a verbose XML manner.

Any guidance would be much appreciated.

[tobias]

Hello,
the remote listing can have two modes: the original single-file mode, where large listings can be split by providing multiple sequential files.

And the new listing format with additional index, which is used to be able to create multi-threaded listings, where a number of separate listing files are created by separate threads simultaneously, plus an index. This is much more complicated.

I will try to write up some info on how to generate the sequential non-indexed listings.

[adzlaw]

Thanks Tobias this would be very much appreciated.