All file transfers use polling with a configurable frequency to monitor the source folder and compare snapshots.
On top of the polling interval, there is a configurable time interval which defines the delay used to observe and act on the changes.
Basically when the configured interval passes the product will read all files and folders that are found in the source folder and their attributes (size, last modification time, etc.).
Each snapshot is compared with the previous one and if there are changes between the snapshots, they are notified to the application. Before doing the actual notification, the application will wait for the changes to be considered stable.
The following criteria must all be met for a file to be considered stable:
To make sure that your file is always correctly processed by SFTPPlus or other managed file transfer product make sure that files with don’t have the final content, don’t have a final name nor are placed at a final location.
When using the product to transfer files that are generated dynamically by other software it will be very difficult to configure a stable delay that will work in all cases.
One option is to set up very long stable intervals, but this comes with the drawback of delaying the processing of all the files.
We’ve encountered situations where the files generated were written to with interruptions – significant delays between writes. Since both the file size and last modified timestamp where not changed SFTPPlus considers the file stable after the stable interval passes and attempts to transfer it.
Because it’s still kept open by the process that it’s generating it, you will either transfer an incomplete file or fail to transfer it in case the other process has opened it exclusively.
As a workaround and suggested approach for this cases we recommend using either a different (temporary) folder for generating the files or using an extension filter for ignoring them.
Both are detailed below.
The move/rename operation is atomic on Linux and Unix systems.
Based on our experience this operation is quick on all modern operating systems even if the file size is large.
When the final location or files with final file names have the final content you can set up a short stable delay interval and SFTPPlus will act upon the final files in the fastest way.
The solution is to store the file into a temporary folder that is different from the transfer source folder until it’s complete.
Once the file is complete it can be moved to the transfer source folder.
Another approach is to configure a filename filter for the transfer. This way only files matching the filter are considered valid for transferring.
The software that generates the file can use either a different extension or a suffix while the file is being generated and rename on completion.
If you have a significant number of files in the transfer source folder we recommend the previous solution as all the files will have to be checked against the filter and when there are thousands of them it might impact performance.