Documentation
13.11. File transfer processing scenarios¶
This section provides ideas and further information to a few different types of file transfer scenarios.
For those that have come across this guide and are new to SFTPPlus transfers features, the following resources will be of use:
Read more about configuration options here.
Read more about client-side transfer operations here.
13.11.1. Transferring files generated by slow producers¶
To ensure that your file is always correctly processed by SFTPPlus or other managed file transfer product, ensure that files:
Do not have the final content.
Do not have a final name.
Are not placed at a final location.
When using the product to transfer files that are generated dynamically by other software, it will be very difficult to configure a stable delay that will work in all cases.
One option is to set up very long stable intervals, but this comes with the drawback of delaying the processing of all the files.
We've encountered situations where the files generated were written to with interruptions -- significant delays between writes. Since both the file size and last modified timestamp remain unchanged, SFTPPlus considers the file stable after the stable interval passes and thus attempts to transfer it.
Because it's still kept open by the process that it's generating it, you will either transfer an incomplete file or fail to transfer it in case the other process has opened it exclusively.
As a workaround and suggested approach for these cases, we recommend using either a different (temporary) folder for generating the files or using an extension filter for ignoring them.
Both are detailed below.
The move/rename operation is atomic on Unix-like systems.
Based on our experience, this operation is quick on all modern operating systems even if the file size is large.
When the final location or files with final file names have the final content, you can set up a short stable delay interval and SFTPPlus will act upon the final files in the fastest way.
13.11.1.1. Using a different folder¶
The solution is to store the file into a temporary folder that is different from the transfer source folder until it's complete.
Once the file is complete it can be moved to the transfer source folder.
13.11.1.2. Using a filename filter¶
Another approach is to configure a filename filter for the transfer. This way, only files matching the filter are considered valid for transferring.
The software that generates the file can use either a different extension or a suffix while the file is being generated and rename on completion.
If you have a significant number of files in the transfer source folder we recommend the previous solution as all the files will have to be checked against the filter and when there are thousands of them it might impact performance.
13.11.2. Transferring to legacy / restricted remote servers¶
In order to transfer a file, SFTPPlus performs a set of extra operations in addition to the actual request to upload the file's content. For example, it can check if a file already exists on destination in order to prevent on overwrite.
When dealing with legacy servers, some of these extra operations might not be supported. You can also have a remote server which restricts operations other than the actual file transfer itself.
SFTPPlus allows sending a file using only the basic file upload operation provided by the transfer protocol.
Below is a list of configuration options which trigger extra requests:
Set the remote location with idle_connection_keepalive_interval = 0. In this way, SFTPPlus will not trigger idle folder listing in order to force keeping the connection opened.
Disable destination overwriting rule with overwrite_rule = disable. In this way, SFTPPlus will not perform extra request to check if a remote file already exists.
13.11.3. Stop / Cancel a transferred file¶
When a transfer is stopped any pending file is canceled.
The current data transfer is stopped and no other actions are executed for that transfer.
13.11.4. Pushing files to mainframe system¶
The IBM z/OS (OS/390) mainframe operating system provides 2 main methods to access files:
MVS record-oriented data access methods (PDS PDS/E) backed by sequential or direct access storage devices (DASD).
UNIX System Services (USS) using regular Unix paths file /parent/file.txt.
The UNIX System Services provide a standard POSIX path and can be handled in a similar way to Linux of macOS file access.
SFTPPlus supports uploading files to a record-oriented dataset and setting the options for the upload.
Dovetail Co:Z SFTP for z/OS or Tectia SFTP Server for z/OS provide various extensions to facilitate interoperability for MVS datasets.
Below is an example for setting the record format and the logical record length for pushing a file to a server using Tectia SFTP. Each file is appended to the PDEB.OU11.DRSAFG data set. The destination path is set to the UNIX root (/), but it is ignored, as we configure the transfer for a fixed path. File transfer advice strings (FTADV) is used to specify z/OS specific options:
destination_uuid: 44cef305-4f01-43b4-9894-6394e91d50db
destination_path: /
destination_path_actions:
*, fixed-path, "/FTADV:RECFM=FB,LRECL=1024/___//'PDEB.OU11.DRSAFG(+1)'"
Here is an example for a similar transfer to a Co:Z SFTP server:
destination_uuid: 44cef305-4f01-43b4-9894-6394e91d50db
destination_path: /
execute_on_destination_before: ls /+lrecl=1024,recfm=fb
destination_path_actions:
*, fixed-path, //PDEB.OU11.DRSAFG(+1)