5.9. Client-Side Transfers Operations

5.9.1. Introduction

Transfers are the pro-active component of SFTPPlus. While a file transfer service only acts in response to requests from remote clients, transfers actively check local or remote paths and initiate transfers based on changes in those folders.

Transfers are executed by transferring files between a source location and a destination location.

5.9.2. Source Location

Source location changes are detected by regularly checking the source folder for changes.

When a change is observed on the monitored source location, the monitor emits one of the following event types:

  • File modified
  • File/folder created
  • File/folder deleted
  • File/folder moved

Here is a list of file system changes which are ignored:

  • Folder modified (since they are emitted whenever a member is created / removed / moved inside that folder)
  • Permissions / Security / Attribute changes

For transfers of type Local filesystem only, created, moved, or removed folders will only emit an informational log event, without triggering the prior transfer command. The same behaviour is present when a file is removed.

For transfers of type other than Local filesystem only, folder events and file removed event are ignored, and no informational event is emitted.

Source events are observed only inside the configured path, and the server will ignore events outside the configured path. While in most cases this is the desired result, it affects ‘moved’ events. Moved events are only observed inside the location path. When you move a file from a path outside the monitored path, a create event is emitted. When you move a file to a path outside the monitored path, a delete event is emitted.

A transfer can monitor only direct ancestors of a folder, for non-recursive monitors, or all ancestors for recursive ones.

When the source location is not available (for example, its connection failed after all retries) the transfer will become stalled and no files will be transfered.

5.9.3. Destination Location

The destination’s locations are accessed only when a file which needs to be transferred is detected on the source location.

When the destination location is not available, the changes detected on the source location are still observed but no transfer is attempted. As soon as the destination location is available again, new changes observed on the destination will be processed.

5.9.4. Transfer States

A transfer can be in one of the following states:

  • stopped - the transfer is not active and is not configured to be active at any time.
  • started - the transfer is active and processing files from the source.
  • suspended - the transfer was started, but it is outside of its activity schedule.
  • source failed - the source location is not available and files at source are not processed. Processing will be started as soon as the source location is available, either by fixing the errors and manually starting it or by an automated location reconnection process.
  • destination failed - The destination location is not available. Processing will resume as soon as the destination location is available, either by fixing the errors or manually starting it.
  • failed - a critical error occurred while executing the transfer and the transfer was stopped. Manual intervention is required to restart the transfer.

5.9.5. Location States

A location can be in one of the following states:

  • Stopped - no transfer is using this location and it was not manually started.
  • Started - the location is connected to the remote location.
  • Disconnected - the location is not connected to the remote location, but it will auto-reconnect when required by a transfer.
  • Failed - the location failed to connect to the remote location. Manual intervention is required to restart the location.

5.9.6. Fault-tolerant transfers

SFTPPlus will do its best to perform the requested transfers and under certain circumstance will retry on transfer failures.

From the configuration, you can define the number of times to retry and the time interval to wait after retries.

Once all retries are exhausted, the transfer will fail and SFTPPlus will no longer try to process/transfer that file.

Files which were previously failed to be transferred, can be processed/transfered again by restarting the transfer.

5.9.7. Fault-tolerant source monitoring

When a transfer is started, SFTPPlus will check for the required permissions to monitor the source path for changes, the configured path exists and that this path is valid.

If the source path is no longer available after the transfer was started and the initial check operations were successful, SFTPlus will not consider the transfer failed. It will wait for the next configured changes_poll_interval to pass and will try again. This is done for an unlimited number of times and an event is emitted on each failure. Once the source path is available again, the transfer is resumed and new or changed files will be transferred.

The type of source path errors which are considered not critical for the purpose of source monitoring varies based on the file transfer protocol.

The most common errors found on all supported protocols are:

  • Source path not found.
  • Source path without read or listing permissions.
  • Connection closed while source was checked.

For the HTTP and WebDAV protocol, below is a non-exhaustive list of HTTP codes for which source monitoring will continue:

  • 401 UNAUTHORIZED
  • 403 FORBIDDEN
  • 404 NOT FOUND
  • 429 TOO MANY REQUESTS
  • 500 INTERNAL SERVER ERROR
  • 503 SERVICE UNAVAILABLE
  • 504 GATEWAY TIMEOUT

5.9.8. Resumed transfers

Resuming unfinished transfers, either upload or download, is supported across available file transfer protocols as follows:

Protocol Download Upload
FTP/FTPS Yes Yes
SFTP Yes Yes
SCP No No
HTTP Yes No

Note

Our roadmap includes adding support for resuming transfers across all available protocols. In the case that resume support is not available for your preferred transfer protocol, please contact us.

5.9.10. Resolving duplicate / successive events

When an event occurs at the source location, it is not signaled right away. A buffer time interval is used to remove duplicate events and emit the event only once the file or folder is stable.

For example, when a large file is copied, the operating system will emit a stream of events after each intermediary change. The monitor will queue these changes and merge them into a single event.

The time frame in which duplicate events are resolved is 1 second. After an event occurs, it waits 1 second before emitting the event. If no event occurs for the same source, the event is emitted. Otherwise, the event is merged and the time interval is extended by another second in which it can be further merged with another event, or emitted.

Here is the list of events which are emitted. Path is used as a generic reference to a file or a folder:

  • Existing - for paths which are already present when the process is started.
  • Modify - for a path which has a changed size or modify time.
  • Delete - for a path which is no longer in the monitored path. Either removed from there or moved to a path which is not monitored.
  • Create - for a path which has appeared in the monitored path. Either created there or moved from a path which is not monitored.
  • Moved - for a path which was renamed / moved inside the monitored path.

Here are the rules based on which events are resolved:

  • existing + modify = existing
  • existing + delete = ignore
  • modify + modify = modify (and wait for next event)
  • modify + delete = delete (without delay)
  • modify + move = move + modify (modify is last)
  • create + modify = create (and wait for next event)
  • create + delete = discard (file was created for a very short time)
  • create + move = create (at the moved path)
  • delete.file + create.file = modify (the file was completely replaced)
  • delete.folder + create.file = create.file
  • moved + created = moved + created (with delay)
  • moved + modified = moved + modified (with delay)
  • moved + deleted = deleted for the source path

5.9.11. Executing commands using pre/post transfer executables or scripts

SFTPPlus allows configuring a transfer to execute at certain transfer steps:

  • Execute before processing a source file
  • Execute after success of a file transfer
  • Execute after failure of a file transfer

In the configuration file, these options, along with a configuration or sample path of the executable or shell script, are:

execute_after_success = Disabled

execute_before = path/execute_copy_arguments.sh

execute_after_failure = path\execute_copy_arguments.bat

5.9.12. Executing pre-/post- transfer destination commands

SFTPPlus allows configuring a transfer to execute a list of commands via the following options:

  • Execute after success on destination
  • Execute before on destination
  • Execute before success on destination.

These commands need to be available in the SFTPPlus client-shell implementation.

To check what commands are available, please start the client-shell and run:

> help

To find out more about a particular command, run:

> help COMMAND_NAME

These commands are configured as part of the Transfer section in the Local Manager GUI and it can be added in the configuration text file like below:

execute_on_destination_after_success = rename /path1 /path2; delete /path3

execute_on_destination_before = client-shell-COMMAND /file/path

execute_on_destination_before =
rename /path-to/  /path2
delete /path3

Transfers can executed either a single command or multiple commands.

Multiple commands are separated by a semicolon (;) or on a separate line.

5.9.13. Working with the file system monitor service

5.9.13.1. Understanding the changes poll interval

When the changes_poll_interval passes, the application will read all files, folders, and attributes (size, last modification time, etc) found in the monitored folder. Each snapshot is then compared with the previous.

If there are changes between the snapshots, they are notified to the application.

Before doing the actual notification, the application will wait for changes to be considered stable as defined by another option, stable_interval.

Note

Lower values help detect changes quicker, but increase the load, CPU, and network usage for local and remote servers.

The poll interval might drift.

For example, with a value of 72000 and a transfer started at 15:35, the product might be busy at 15:35 so the check is done only at 15:36 and the next check will be scheduled for 17:36

5.9.13.2. Changing the file stable period settings

When a new file is created or a file starts to be modified, it is not processed by SFTPPlus right away. This allows an external program to finish handling the file.

If no other changes to the file are observed after this interval, it is then processed.

Criteria for a file to be considered stable:

  • Size is not changed.
  • Last modified time attribute is not changed.
  • The configured stable interval has elapsed.

After the last modification is observed, this interval is then allowed to pass.

Each change will reset the stable interval for the file itself since each file has its own stable counter.

Since changes can only be observed with a resolution defined by changes_poll_interval, the stable_interval needs to be a multiple.

The following will check new files every hour and transfer stable files after two hours, even if the stable_interval is lower:

changes_poll_interval = 3600
stable_interval = 60