Destination path conversion#
SFTPPlus can change file names and paths when transferring files to a destination. Use destination path rules to rename files, select folders, or add timestamps. A fallback destination can receive files when delivery to the computed path fails.
For the configuration reference, see transfers configuration.
Configuring the Destination File Name and Path#
The destination_path_actions option can be defined for a transfer and will allow using a destination path and file name different than the source path and file name. In this way, you can define transfers to dynamic destination paths.
This is configured using a set of rules which are applied based on expression matching the source path.
To highlight misconfiguration or omission in configuration, when a transfer has a destination_path_actions configured and it tries to transfer files not matching any of the configured rules, SFTPPlus will stop the whole transfer with a failure and will not process any further files.
The base destination path, as defined in destination_path is not affected by the file path and file name transformation.
A transfer can have multiple rules defined for the same path and the actions will be applied in series. The result from one action will be passed to the next one. Having the following configuration:
[transfers/313a9679-bc15-42e9-80ef-e2305959a3d1]
enabled: yes
name: copy-to-remote
recursive: yes
source_path: c:\out\sales
source_filter: *
delete_source_on_success: Yes
destination_uuid: 5f304286-8d56-41f9-ba57-420a2303e90c
destination_path: /Outbox/
destination_path_actions:
*.pdf, only-filename
*.pdf, lowercase
the files will be transferred as follows:
c:\out\sales\Report.PDF -> /Outbox/report.pdf
c:\out\sales\18\Jan\Outcome.PDF -> /Outbox/outcome.pdf
In the following sections, you will find detailed information for each of the available actions.
no-action#
The no-action rule does not change the transferred path or name.
It is used to explicitly inform SFTPPlus that you don't want a transformation for certain file types and to transfer the file with the same path and name as is found on the source.
transform#
The transform action is the generic operation that can be used to define a destination path and name, different from the source path.
The whole path can be transformed, not only the filename.
Any transformed path will be transferred to a path relative to the path configured via destination_path. Use the {ignore_destination_path} placeholder at the start of transformation rule to ignore the configured destination path.
It can be used for generic rename operations for which the result is defined based on regular expression groups. Each regex matching group is available via the {group.N} variable, where {group.0} is the whole source and {group.1} is the first matched group.
Having the following configuration:
recursive: no
source_path: c:\out\sales
destination_path: /Outbox/
destination_path_actions:
m/(std-|prv-)(.*)/, transform, final_{group.2}
the files will be transferred as follows:
c:\out\sales\std-Report.PDF -> /Outbox/final_Report.PDF
c:\out\sales\prv-Outcome.txt -> /Outbox/final_Outcome.txt
The following placeholders are available based on the source file, relative to the base source path:
{path.source} - the path including file name, relative to the base source path for this transfer.
{path.directory_path} - the path of the source file's directory, relative to the configured base source path.
{path.file_name} - the file name, including the extension.
{path.file_base} - the file name without the extension.
{path.file_extension} - the file extension.
{ignore_destination_path} - to instruct that the configured destination_path should be ignored.
Having the following configuration:
source_path: c:\out\sales
destination_path: /Outbox/
destination_path_actions:
*.pdf, transform, {path.directory_path}/Preview-{path.file_name}
*.txt, transform, {path.directory_path}/{path.file_base}-Estimation{path.file_extension}
the files will be transferred as follows:
c:\out\sales\Report.PDF -> /Outbox/Preview-Report.PDF
c:\out\sales\Report.txt -> /Outbox/Report-Estimation.txt
c:\out\sales\EMEA\Income.PDF -> /Outbox/EMEA/Preview-Income.PDF
c:\out\sales\EMEA\Income.txt -> /Outbox/EMEA/Income-Estimation.txt
It can be used for inserting a timestamp in an arbitrary position, based on the current date and time. The following placeholders are available:
{now.cwa_14051} - Human-readable local date and time
{now.iso_8601} - ISO 8601 date and time
{now.iso_8601_fractional} - ISO 8601 date and time with fractional seconds
{now.iso_8601_local} - ISO 8601 date and time in the local time zone
{now.iso_8601_basic} - Basic ISO 8601 date and time
{now.iso_8601_compact} - ISO 8601 compatible with Windows file names
{now.timestamp} - Unix timestamp with decimals
{now.year_decimal} - Year as a decimal number, such as 2021
{now.year_no_century} - Zero-padded year without century, such as 21
{now.month_decimal} - Month as a decimal number, such as 1
{now.month_padded} - Zero-padded month, such as 01
{now.day_padded} - Zero-padded day of the month, such as 07
{now.day_decimal} - Day of the month as a decimal number, such as 7
{now.day_of_year_padded} - Zero-padded day of the year, such as 007
{now.hour_padded_24} - Zero-padded hour on a 24-hour clock
{now.hour_decimal_24} - Hour on a 24-hour clock as a decimal number
{now.hour_padded_12} - Zero-padded hour on a 12-hour clock
{now.hour_decimal_12} - Hour on a 12-hour clock as a decimal number
{now.hour_am_pm} - Locale equivalent of AM or PM
{now.minute_padded} - Zero-padded minute
{now.minute_decimal} - Minute as a decimal number
{now.second_padded} - Zero-padded second
{now.second_decimal} - Second as a decimal number
Having the following configuration:
source_path: c:\out\sales
destination_path: /Outbox/
destination_path_actions:
*.pdf, transform, {now.iso_8601_compact}-{group.0}
*.txt, transform, {path.directory_path}/{path.file_base}-{now.iso_8601_compact}{path.file_extension}
the files will be transferred as follows:
c:\out\sales\Report.PDF -> /Outbox/20210121T234503-Report.PDF
c:\out\sales\Report.txt -> /Outbox/Report-20210121T234503.txt
c:\out\sales\EMEA\Income.PDF -> /Outbox/EMEA/20210121T234503-Income.PDF
c:\out\sales\EMEA\Income.txt -> /Outbox/EMEA/Income-20210121T234503.txt
The transform action can be used for transferring files into different destination paths, in which the destination path is selected based on the source path. To implement that, the transformed destination path should be absolute paths.
Having the following configuration:
source_path: c:\out\sales
destination_path: /Out/
destination_path_actions:
*.pdf, transform, /Staging/{path.source}
*.txt, transform, final-{path.file_name}
the files will be transferred as follows:
c:\out\sales\Report.PDF -> /Staging/Report.PDF
c:\out\sales\Report.txt -> /Out/final-Report.txt
c:\out\sales\EMEA\Income.PDF -> /Staging/EMEA/Income.PDF
c:\out\sales\EMEA\Income.txt -> /Out/EMEA/Income.txt
If the transformed destination path looks like a relative path, it will be relative to the configured destination_path.
Use the {ignore_destination_path} placeholder at the start of a transformation expression to generate destination paths that are relative to the server's working directory.
Having the following configuration:
source_path: c:\out\sales
destination_path: /Out/
destination_path_actions:
*.txt, transform, final-{path.file_name}
*.confirm, transform, {ignore_destination_path}{path.file_base}.done
the files will be transferred as follows:
c:\out\sales\Report.txt -> /Out/final-Report.txt
c:\out\sales\EMEA\Income.txt -> /Out/Income.txt
c:\out\sales\EMEA\Income.confirm -> Income.done
only-filename#
The only-filename transformation is used to flatten a path defined across multiple directories by using only the file name.
The parent path as found in the source is ignored.
This can be used to transfer a recursive transfer on the source to a non-recursive transfer on the destination.
Having the following configuration:
[transfers/313a9679-bc15-42e9-80ef-e2305959a3d1]
enabled: yes
name: copy-to-remote
recursive: yes
source_path: c:\out\sales
source_filter: *
delete_source_on_success: Yes
destination_uuid: 5f304286-8d56-41f9-ba57-420a2303e90c
destination_path: /Outbox/
destination_path_actions:
*.pdf, only-filename
*.txt, no-action
the files will be transferred as follows:
c:\out\sales\Report.PDF -> /Outbox/Report.PDF
c:\out\sales\Report.txt -> /Outbox/Report.txt
c:\out\sales\18\Jan\Outcome.PDF -> /Outbox/Outcome.PDF
c:\out\sales\18\Jan\Outcome.txt -> /Outbox/18/Jan/Outcome.txt
temporary-suffix#
The temporary-suffix action allows uploading a file using a temporary name, then renaming to the initial name once all the file content was transferred.
Note
The temporary-suffix action is required only when a selected subset of files need to be transferred with temporary names.
It can also be used in combination with other actions.
When you want all files transferred with temporary names and no other actions are required for the files, it is recommended to use the dedicated destination_temporary_suffix configuration option.
It takes a single option consisting of the characters used as the temporary suffix.
With the following configuration, the file c:outsalesreport.pdf will be first transferred to /Outbox/report.pdf.tmp, then finally renamed to /Outbox/report.pdf:
[transfers/313a9679-bc15-42e9-80ef-e2305959a3d1]
enabled: yes
name: push-to-remote
source_path: c:\out\sales
source_filter: *
delete_source_on_success: Yes
destination_uuid: 5f304286-8d56-41f9-ba57-420a2303e90c
destination_path: /Outbox/
destination_path_actions:
*.pdf, temporary-suffix, .tmp
When a transfer fails before completing the transfer of the file's content, the incompletely transferred file on the destination keeps the temporary suffix.
temporary-prefix#
The temporary-prefix action allows uploading a file using a temporary prefixed name, renaming to the initial name once all the file content was transferred.
It takes a single option consisting of the characters used as the temporary prefix.
With the following configuration, the file c:outsalesreport.pdf will be first transferred to /Outbox/TEMP_report.pdf, then finally renamed to /Outbox/report.pdf:
[transfers/313a9679-bc15-42e9-80ef-e2305959a3d1]
enabled: yes
name: push-to-remote
source_path: c:\out\sales
source_filter: *
delete_source_on_success: Yes
destination_uuid: 5f304286-8d56-41f9-ba57-420a2303e90c
destination_path: /Outbox/
destination_path_actions:
*.pdf, temporary-prefix, TEMP_
When a transfer fails before completing the transfer of the file's content, the incompletely transferred file on the destination keeps the temporary prefix.
replace-separator#
The replace-separator transformation is used to flatten a path defined across multiple directories and convert it into a simple file name.
It takes a single option which is the character or characters used to substitute the path separators. Depending on the source filesystem, it will replace the / or \ character.
For recursive transfers, the destination path in this transformation is relative to the base source_path.
Having the following configuration:
[transfers/313a9679-bc15-42e9-80ef-e2305959a3d1]
enabled: yes
name: copy-to-remote
recursive: yes
source_path: c:\out\sales
source_filter: *
delete_source_on_success: Yes
destination_uuid: 5f304286-8d56-41f9-ba57-420a2303e90c
destination_path: /Outbox/
destination_path_actions:
*.pdf, replace-separator, --
*.txt, replace-separator, _
the files will be transferred as follows:
c:\out\sales\Report.PDF -> /Outbox/Report.PDF
c:\out\sales\Report.txt -> /Outbox/Report.txt
c:\out\sales\report.csv -> Fail as it does not match any rule.
c:\out\sales\18\Jan\Report.PDF -> /Outbox/18--Jan--Report.PDF
c:\out\sales\18\Jan\Report.txt -> /Outbox/18_Jan_Report.txt
c:\out\sales\18\Jan\report.csv -> Fail as it does not match any rule.
lowercase#
The lowercase is a simple transformation that will transfer the file using destination file path and file name all in lower case. It takes no options.
Note
If the remote filesystem is case-insensitive and a file with the same name (but different cases) exists, the remote name is kept.
Below is an example for the lowercase transformation:
[transfers/313a9679-bc15-42e9-80ef-e2305959a3d1]
enabled: yes
name: copy-to-remote
recursive: yes
source_path: /out/sales
source_filter: *
delete_source_on_success: Yes
destination_uuid: 5f304286-8d56-41f9-ba57-420a2303e90c
destination_path: /Outbox/
destination_path_actions:
*.pdf, lowercase
*, no-action
With the above configuration, the following files will be transferred as exemplified below. Note that the base destination path as defined in the transfer configuration, is not affected by the file path and file name transformation. Files like report.txt are transferred since they match the no-action rule:
/out/sales/Report.PDF -> /Outbox/report.pdf
/out/sales/Report.txt -> /Outbox/Report.txt
/out/sales/Jan/Report.PDF -> /Outbox/jan/report.pdf
/out/sales/Jan/Report.txt -> /Outbox/Jan/Report.txt
fixed-path#
The fixed-path action is a simple transformation that will transfer the file using a fixed destination. It takes a single option, the full destination path. The destination_path configuration is ignored for the fixed-path action.
Note
If this action is enabled, the value defined at destination_temporary_suffix is ignored. You can still enable the temporary file name functionality by using the dedicated temporary-prefix or temporary-suffix actions.
Below is an example for the fixed-path transformation:
[transfers/313a9679-bc15-42e9-80ef-e2305959a3d1]
enabled: yes
name: copy-to-remote
recursive: yes
source_path: /out/sales
source_filter: *
delete_source_on_success: Yes
destination_uuid: 5f304286-8d56-41f9-ba57-420a2303e90c
destination_path: /Outbox/
destination_path_actions:
*, fixed-path, //'PDEB.OU11.DRSAFG(+1)'
With the above configuration, the following files will be transferred as exemplified below. Note that the base destination path as defined in the transfer configuration, is ignored:
/out/sales/Report.PDF -> //'PDEB.OU11.DRSAFG(+1)'
/out/sales/Report.txt -> //'PDEB.OU11.DRSAFG(+1)'
/out/sales/Jan/Report.PDF -> //'PDEB.OU11.DRSAFG(+1)'
/out/sales/Jan/Report.txt -> //'PDEB.OU11.DRSAFG(+1)'
Fallback destination path#
When using the destination_path_actions to define a transfer with dynamic destination path, some transfers might fail as the computed destination path doesn't exist.
The destination_fallback_path configuration can be used as a catchall method to ensure the remote files are transferred to the destination location.
For example, if you want to pull reports from a server, and automatically route them into directories using destination_path_actions, and one of the route targets does not exist, you can route them to a fallback folder.
A transfer can have multiple actions defined for the same path, and the actions will be applied in series. The result from one action will be passed to the next one.
[transfers/313a9679-bc15-42e9-80ef-e2305959a3d1]
enabled: yes
name: pull-partner-reports
recursive: no
source_uuid: 5f304286-8d56-41f9-ba57-420a2303e90c
source_path: /remote/partner/
destination_path: D:\reports\acme-inc\others
destination_fallback_path: D:\reports\acme-inc\unknown
destination_path_actions:
m/.*report_(.*).pdf/, transform, D:\reports\acme-inc\{group.1}\{path.file_name}
*, no-action
Assuming that the following directories exist on the local filesystem, which is the destination:
D:\reports\acme-inc\sales\ -> receives sales reports
D:\reports\acme-inc\stocks\ -> receives stocks reports
D:\reports\acme-inc\unknown\ -> catchall for other report files
D:\reports\acme-inc\others\ -> catchall for any non-report file.
the files will be transferred as follows:
/remote/partner/report_sales_Jan.pdf -> D:\reports\acme-inc\sales\report_sales_Jan.pdf
/remote/partner/report_stocks_Jan.pdf -> D:\reports\acme-inc\stocks\report_stocks_Jan.pdf
/remote/partner/report_toys_Jan.pdf -> D:\reports\acme-inc\unknown\report_toys_Jan.pdf
/remote/partner/returns_Jan.pdf -> D:\reports\acme-inc\others\returns_Jan.pdf
Note that report_toys_Jan.pdf is not copied into the others directory. It's copied into the unknown directory. It has the pattern of a report file, but there is no dedicated report folder named toys on the destination to receive it.
Pushing files to mainframe systems#
The IBM z/OS (OS/390) mainframe operating system provides 2 main methods to access files:
MVS record-oriented data access methods (PDS PDS/E) backed by sequential or direct access storage devices (DASD).
UNIX System Services (USS) using regular Unix file paths such as /parent/file.txt.
The UNIX System Services provide a standard POSIX path which can be handled similarly to the way Linux or macOS file access works.
SFTPPlus supports uploading files to a record-oriented dataset and setting the options for the upload.
Dovetail Co:Z SFTP for z/OS or Tectia SFTP Server for z/OS provide various extensions to facilitate interoperability for MVS datasets.
Below is an example for setting the record format and the logical record length for pushing a file to a server using Tectia SFTP. Each file is appended to the PDEB.OU11.DRSAFG data set. The destination path is set to the UNIX root (/), but it is ignored, as we configure the transfer for a fixed path. File transfer advice strings (FTADV) are used to specify z/OS specific options:
destination_uuid: 44cef305-4f01-43b4-9894-6394e91d50db
destination_path: /
destination_path_actions:
*, fixed-path, "/FTADV:RECFM=FB,LRECL=1024/___//'PDEB.OU11.DRSAFG(+1)'"
Here is an example for a similar transfer to a Co:Z SFTP server:
destination_uuid: 44cef305-4f01-43b4-9894-6394e91d50db
destination_path: /
execute_on_destination_before: ls /+lrecl=1024,recfm=fb
destination_path_actions:
*, fixed-path, //PDEB.OU11.DRSAFG(+1)