Transfers are the pro-active component of SFTPPlus. While a file transfer service only acts in response to requests from remote clients, transfers actively check local or remote paths and initiate transfers based on changes in those folders.
Transfers are executed by transferring files between a source location and a destination location.
Source location changes are detected by regularly checking the source folder for changes.
When a change is observed on the monitored source location, the monitor emits one of the following event types:
Here is a list of file system changes which are ignored:
For transfers of type Local filesystem only, created, moved, or removed folders will only emit an informational log event, without triggering the prior transfer command. The same behaviour is present when a file is removed.
For transfers of type other than Local filesystem only, folder events and file removed event are ignored, and no informational event is emitted.
Source events are observed only inside the configured path, and the server will ignore events outside the configured path. While in most cases this is the desired result, it affects ‘moved’ events. Moved events are only observed inside the location path. When you move a file from a path outside the monitored path, a create event is emitted. When you move a file to a path outside the monitored path, a delete event is emitted.
A transfer can monitor only direct ancestors of a folder, for non-recursive monitors, or all ancestors for recursive ones.
When the source location is not available (for example, its connection failed after all retries) the transfer will become stalled and no files will be transfered.
The destination’s locations are accessed only when a file which needs to be transferred is detected on the source location.
When the destination location is not available, the changes detected on the source location are still observed but no transfer is attempted. As soon as the destination location is available again, new changes observed on the destination will be processed.
A transfer can be in one of the following states:
A location can be in one of the following states:
SFTPPlus will do its best to perform the requested transfers and under certain circumstance will retry on transfer failures.
From the configuration, you can define the number of times to retry and the time interval to wait after retries.
Once all retries are exhausted, the transfer will fail and SFTPPlus will no longer try to process/transfer that file.
Files which were previously failed to be transferred, can be processed/transfered again by restarting the transfer.
When a transfer is started, SFTPPlus will check for the required permissions to monitor the source path for changes, the configured path exists and that this path is valid.
If the source path is no longer available after the transfer was started and the initial check operations were successful, SFTPlus will not consider the transfer failed. It will wait for the next configured changes_poll_interval to pass and will try again. This is done for an unlimited number of times and an event is emitted on each failure. Once the source path is available again, the transfer is resumed and new or changed files will be transferred.
The type of source path errors which are considered not critical for the purpose of source monitoring varies based on the file transfer protocol.
The most common errors found on all supported protocols are:
For the HTTP and WebDAV protocol, below is a non-exhaustive list of HTTP codes for which source monitoring will continue:
Resuming unfinished transfers, either upload or download, is supported across available file transfer protocols as follows:
Protocol | Download | Upload |
---|---|---|
FTP/FTPS | Yes | Yes |
SFTP | Yes | Yes |
SCP | No | No |
HTTP | Yes | No |
Note
Our roadmap includes adding support for resuming transfers across all available protocols. In the case that resume support is not available for your preferred transfer protocol, please contact us.
Hard links and symbolic links are observed as normal files.
When both the link and original are monitored, changes to the link will trigger notifications for the original file as well as any other hard links pointing to it.
When only the link is monitored, changes to the original source will also trigger notifications.
When an event occurs at the source location, it is not signaled right away. A buffer time interval is used to remove duplicate events and emit the event only once the file or folder is stable.
For example, when a large file is copied, the operating system will emit a stream of events after each intermediary change. The monitor will queue these changes and merge them into a single event.
The time frame in which duplicate events are resolved is 1 second. After an event occurs, it waits 1 second before emitting the event. If no event occurs for the same source, the event is emitted. Otherwise, the event is merged and the time interval is extended by another second in which it can be further merged with another event, or emitted.
Here is the list of events which are emitted. Path is used as a generic reference to a file or a folder:
Here are the rules based on which events are resolved:
SFTPPlus allows configuring a transfer to execute at certain transfer steps:
In the configuration file, these options, along with a configuration or sample path of the executable or shell script, are:
execute_after_success = Disabled
execute_before = path/execute_copy_arguments.sh
execute_after_failure = path\execute_copy_arguments.bat
SFTPPlus allows configuring a transfer to execute a list of commands via the following options:
These commands need to be available in the SFTPPlus client-shell implementation.
To check what commands are available, please start the client-shell and run:
> help
To find out more about a particular command, run:
> help COMMAND_NAME
These commands are configured as part of the Transfer section in the Local Manager GUI and it can be added in the configuration text file like below:
execute_on_destination_after_success = rename /path1 /path2; delete /path3
execute_on_destination_before = client-shell-COMMAND /file/path
execute_on_destination_before =
rename /path-to/ /path2
delete /path3
Transfers can executed either a single command or multiple commands.
Multiple commands are separated by a semicolon (;) or on a separate line.
When the changes_poll_interval passes, the application will read all files, folders, and attributes (size, last modification time, etc) found in the monitored folder. Each snapshot is then compared with the previous.
If there are changes between the snapshots, they are notified to the application.
Before doing the actual notification, the application will wait for changes to be considered stable as defined by another option, stable_interval.
Note
Lower values help detect changes quicker, but increase the load, CPU, and network usage for local and remote servers.
The poll interval might drift.
For example, with a value of 72000 and a transfer started at 15:35, the product might be busy at 15:35 so the check is done only at 15:36 and the next check will be scheduled for 17:36
When a new file is created or a file starts to be modified, it is not processed by SFTPPlus right away. This allows an external program to finish handling the file.
If no other changes to the file are observed after this interval, it is then processed.
Criteria for a file to be considered stable:
After the last modification is observed, this interval is then allowed to pass.
Each change will reset the stable interval for the file itself since each file has its own stable counter.
Since changes can only be observed with a resolution defined by changes_poll_interval, the stable_interval needs to be a multiple.
The following will check new files every hour and transfer stable files after two hours, even if the stable_interval is lower:
changes_poll_interval = 3600
stable_interval = 60