Articles and news

Endpoint FTPS and SFTP server for DWP GFTS

Tue 02 April 2019 | blog

A red floppy disk.

The electronic data interchange (EDI) of the Department for Work and Pensions (DWP) in the United Kingdom is done via the Generic File Transfer Service (GFTS) gateway.

This article is aimed at companies which need to exchange files and data with the DWP. These entities are referred by DWP as creditor server or endpoint FTPS server.

The information is also valid for the E-Transfer systems used by local councils.

In practice, this means that as a partner to DWP you will have to set up and host an Explicit FTPS server. DWP is operating an FTPS client and actively pushes data to you.

Electronic data interchange (EDI) is the concept of electronically communicating information that was traditionally communicated on paper, such as purchase orders and invoices.

Connection Security

The connection between your company and DWP is secured using certificate-based mutual TLS authentication (mTLS) (also referred to as two-way authentication). DWP will provide the SSL certificate used by their client, while your company will have to provide the SSL certificate used by your FTPS server.

With SFTPPlus you can use a certificate generated by any certificate authority (public or your private CA).

Integration with the Let's Encrypt Certificate Authority is provided via the HTTP-01 challenge. SFTPPlus can seamlessly obtain and use a certificate from the Let's Encrypt CA. The certificate is automatically renewed.

On top of the security provided by the TLS/SSL layer, username/password credentials are used to identify the requests from DWP.

SFTPPlus can support a multi-channel architecture, allowing you to use the same SFTPPlus server for exchanging files with multiple partners, not only with DWP.

Read more about securing FTPS server with SFTPPlus in our dedicated documentation page.

Client / Server Data Exchange

FTPS is an open standard file transfer protocol built on a client-server model architecture.

The client is the active component which controls when and what type of file transfer operation to perform. The client generates an authenticated connection to the server and asks the server to push or pull files. DWP will act as a client.

The server is the reactive component which controls who can perform file transfer operations and what kind of file operations are allowed. The server stays idle and only becomes active once it receives a connection from the client. Your system will act as a server.

Once the data is pushed by DWP, it will reside as files on your system. From there it will be further processed and consumed by your business system.

ProAtria DWP Expertise

ProAtria, the developer of SFTPPlus, is a long-term partner for the project deployed at DWP. We have helped with the migration from insecure FTP to Explicit and Implicit FTPS systems and with the migration from legacy Solaris-based systems to a modern Linux-based cloud infrastructure.

We are involved in the delivery and maintenance of the Digital Children’s Platform (DOS 012) and the data exchange between DWP and the Scottish Government.

We offer broad expertise into the data exchange with DWP and DVLA. Our customers benefit of help and consultancy for their DWP and DVLA related projects without any additional cost.

A server rack.

Evaluating SFTPPlus MFT

The features listed in this article are just a selected few out of many integration and configuration options that are available today. Feel free to talk to the Support team about your requirements with file transfer software.

SFTPPlus MFT Server supports FTP, Explicit FTPS, Implicit FTPS, SFTP, SCP, HTTP and HTTPS.

SFTPPlus MFT is available as an on-premise solution supported on Windows, Linux, and macOS.

It is also available on the cloud as Docker containers, AWS or Azure instances and many other cloud providers.

Request a trial using the form below.

• • •

SFTPPlus Release 3.46.0

Mon 11 March 2019 | general release

We are announcing the latest release of SFTPPlus version 3.46.0.

New Features

  • The HTTP/HTTPS file transfer service now supports downloading multiple files at once as a Zip file. [server-side][web-api][http][https] [#5093]
  • It is now possible to set up password expiration for accounts and groups. [server-side][security] [#5146]
  • It is now possible to configure the preferred size of the group in the SSH Diffie-Hellman group key exchange method. [server-side][sftp][scp] [#5205]
  • The file dispatcher event handler now supports the copy action. This will copy the source file to one or more destinations, without removing the source file. [server-side][client-side] [#5210]
  • The file dispatcher event handler now supports the rename action. This will rename the source file (with an atomic move operation) without overwriting an existing file. [server-side][client-side] [#5220]

Defect Fixes

  • An event is now emitted when a file is closed after it was open for reading through the HTTP file transfer service. [server-side][http][https] [#5093]
  • The HTTP/HTTPS file transfer service now responds with 401 Unauthorized for requests made with 100 Continue when no credentials are provided in the request. [server-side][http][https] [#5223]

You can check the full release notes here.

• • •

FTP client uploads with temporary names

Thu 21 February 2019 | ftp client-side blog

A drawer with file tabs.

When closely investigating managed file transfers, pushing a file to a remote FTP server turns out to be just a phase in a series of interlinked processes.

Once the file arrives on the FTPS server, it is read and further processed. The next step might involve downloading the file or copying it to another processing area.

When uploading a large file, copying or pulling it before completing the transfer can result in corrupted file data. For example, a pull operation might start before the file is fully uploaded, with only a fragment of the original file available for download.

Another common case in which data corruption may happen is when a partial upload occurs because of connection failures during transfer. A client starts sending a file to the server, but at some point connection is lost. Maybe the client VM was powered off unexpectedly or the network became temporarily unavailable for too long. This will result in a partial file being left on the server, which can be accidentally processed by the next stage in our process.

This is a serious issue with FTP and FTPS connections. FTP protocols do not mandate sending the total file size before an upload. Furthermore, they do not make use of an explicit end-of-file marker. An FTP client signals the completion of an upload by simply closing the data connection.

To mitigate this problem, a file locking mechanism can be implemented by uploading files using temporary names and then renaming them back to their initial names once all the data was pushed by the client.

Clients like WinSCP will use temporary names formed by appending a non-configurable .filepart extension to the initial file names.

In SFTPPlus you can configure a file transfer to use any suffix / extension during the upload, you are not restricted to the .filepart one. . For example, you can use the .tmp or .incomplete extensions.

Screenshot with transfer destination in SFTPPlus.

By using temporary names you can implement a process in which transferred files are locked while their contents are being uploaded. The chained process will ignore files with temporary names, only handling transferred files after the final rename operation.

On most file systems the rename operation is atomic and very fast.

The same technique can be used to lock a file while uploading through SFTP transfers.

The SCP protocol does not provide a rename operation, but the total file size is advertised in the SCP upload request, which happens before the client starts pushing the content of the file.

Read more about transferring files with temporary names in our documentation page.

Evaluating SFTPPlus MFT

The features listed in this article are just a selected few out of many integration and configuration options that are available today. Feel free to talk to the Support team about your requirements with file transfer software.

SFTPPlus MFT Server supports FTP, Explicit FTPS, Implicit FTPS, SFTP, SCP, HTTP and HTTPS.

SFTPPlus MFT is available as an on-premise solution supported on Windows, Linux, and macOS.

It is also available on the cloud as Docker containers, AWS or Azure instances and many other cloud providers.

Request a trial using the form below.

• • •

Restrict user with trusted IPs for SFTP and FTPS

Fri 15 February 2019 | security blog

Computer security illustration.

It is common practice to secure a file transfer server using firewall rules which only allow incoming connections from trusted partners.

Let's assume you have a US partner named "ACME Inc", connecting to your server from IP 1.1.1.1 using the user acme-inc, and another German partner called "AlleWerkzeuge AG", connecting to your server from IP 5.5.5.5 using the user alle-werkzeuge-ag.

You can configure your firewall to only allow connections from a list of trusted IPs like 1.1.1.1 and 5.5.5.5, but the firewall doesn't know about usernames. So it will allow the account acme-inc to connect even if the connection is initiated from 5.5.5.5, which is an IP outside of the ACME Inc network.

To complement firewall restrictions, SFTPPlus allows defining a fixed list of trusted IP rules from which it will allow connections for a specific user.

Such a configuration can be defined per user, but also per group, with multiple users inheriting their configuration from the group.

To restrict a specific user to connect through SFTP or FTPS to the file transfer server only from a certain IP (or IPs), you can use the source_ip_filter configuration option in SFTPPlus.

The remote access is denied when the user connects from a source address which is not whitelisted.

Below is a screenshot from our web-based management console demonstrating such a configuration.

Screenshot of SFTPlus account configuration.

Read more about securing your SFTP/FTPS and HTTPS services with SFTPPlus in our documentation page.

Evaluating SFTPPlus MFT

The features listed in this article are just a selected few out of many integration and configuration options that are available today. Feel free to talk to the Support team about your requirements with file transfer software.

SFTPPlus MFT Server supports FTP, Explicit FTPS, Implicit FTPS, SFTP, SCP, HTTP and HTTPS.

SFTPPlus MFT is available as an on-premise solution supported on Windows, Linux, and macOS.

It is also available on the cloud as Docker containers, AWS or Azure instances and many other cloud providers.

Request a trial using the form below.

• • •

SFTPPlus Release 3.45.0

Thu 14 February 2019 | general release

We are announcing the latest release of SFTPPlus version 3.45.0.

New Features

  • It is now possible for SFTP/SCP clients to change their own password using the SSH command execution service. [server-side][sftp][scp] [#5129]
  • It is now possible to transfer files using temporary names, renaming to their initial names once successfully transferred. [client-side] [#5156]
  • Events emitted when a file is closed after a server-side SFTP or SCP transfer now include transferred size, duration, and average speed. [server-side][scp][sftp] [#5196]
  • You can now configure an account to allow authentication only from a specific list of source IP addresses. [server-side][security] [#5201]

Defect Fixes

  • The SFTP/SCP file transfer service no longer generates an internal server error when the SCP protocol is requested as an SSH subsystem. [server-side][scp] [#5129]
  • For move transfers, the removal of the source file is now retried when the operation fails. In previous versions, once the file was transferred, the source removal was attempted only once. [client-side] [#5156-1]
  • The transfer of a file is now retried when the operation to check the existence of the remote file fails. [client-side] [#5156]
  • For the SCP protocol, the event with ID 30042 is no longer emitted when the client is sending the whole file without an end of file marker. In previous versions, if the SCP client uploaded all the file data, but did not send the explicit confirmation for the end of file or stream, SFTPPlus was emitting the event 30042 to inform that the transfer was not complete. [server-side][scp] [#5196]

Deprecations and Removals

  • The events emitted for rename operations now have the destination path as the default path attribute. In previous versions the source path was used. The from attribute will contain the source path. The following event IDs are affected: 60043, 60044, 30025, 30026, 30027 [server-side][client-side] [#5156]
  • Support for FreeBSD 10.x on X86_64 was removed. [#5170]

You can check the full release notes here.

• • •