SFTPPlus Documentation

Start Page 8. User’s Guides 8.11. File transfer processing scenarios
client-side file-operation

8.11. File transfer processing scenarios

This section provides ideas and further information to a few different types of file transfer scenarios.

For those that have come across this guide and are new to SFTPPlus transfers features, the following resources will be of use:

Read more about configuration options here.

Read more about client-side transfer operations here.

8.11.1. Transferring files generated by slow producers

To ensure that your file is always correctly processed by SFTPPlus or other managed file transfer product, ensure that files:

  • Do not have the final content.
  • Do not have a final name.
  • Are not placed at a final location.

When using the product to transfer files that are generated dynamically by other software, it will be very difficult to configure a stable delay that will work in all cases.

One option is to set up very long stable intervals, but this comes with the drawback of delaying the processing of all the files.

We’ve encountered situations where the files generated were written to with interruptions – significant delays between writes. Since both the file size and last modified timestamp remain unchanged, SFTPPlus considers the file stable after the stable interval passes and thus attempts to transfer it.

Because it’s still kept open by the process that it’s generating it, you will either transfer an incomplete file or fail to transfer it in case the other process has opened it exclusively.

As a workaround and suggested approach for these cases, we recommend using either a different (temporary) folder for generating the files or using an extension filter for ignoring them.

Both are detailed below.

The move/rename operation is atomic on Linux and Unix systems.

Based on our experience, this operation is quick on all modern operating systems even if the file size is large.

When the final location or files with final file names have the final content, you can set up a short stable delay interval and SFTPPlus will act upon the final files in the fastest way.

8.11.1.1. Using a different folder

The solution is to store the file into a temporary folder that is different from the transfer source folder until it’s complete.

Once the file is complete it can be moved to the transfer source folder.

8.11.1.2. Using a filename filter

Another approach is to configure a filename filter for the transfer. This way, only files matching the filter are considered valid for transferring.

The software that generates the file can use either a different extension or a suffix while the file is being generated and rename on completion.

If you have a significant number of files in the transfer source folder we recommend the previous solution as all the files will have to be checked against the filter and when there are thousands of them it might impact performance.