Articles from ftp category

When closely investigating managed file transfers, pushing a file to a remote FTP server turns out to be just a phase in a series of interlinked processes.

Once the file arrives on the FTPS server, it is read and further processed. The next step might involve downloading the file or copying it to another processing area.

When uploading a large file, copying or pulling it before completing the transfer can result in corrupted file data. For example, a pull operation might start before the file is fully uploaded, with only a fragment of the original file available for download.

Another common case in which data corruption may happen is when a partial upload occurs because of connection failures during transfer. A client starts sending a file to the server, but at some point connection is lost. Maybe the client VM was powered off unexpectedly or the network became temporarily unavailable for too long. This will result in a partial file being left on the server, which can be accidentally processed by the next stage in our process.

This is a serious issue with FTP and FTPS connections. FTP protocols do not mandate sending the total file size before an upload. Furthermore, they do not make use of an explicit end-of-file marker. An FTP client signals the completion of an upload by simply closing the data connection.

To mitigate this problem, a file locking mechanism can be implemented by uploading files using temporary names and then renaming them back to their initial names once all the data was pushed by the client.

Clients like WinSCP will use temporary names formed by appending a non-configurable .filepart extension to the initial file names.

In SFTPPlus you can configure a file transfer to use any suffix / extension during the upload, you are not restricted to the .filepart one. . For example, you can use the .tmp or .incomplete extensions.

By using temporary names you can implement a process in which transferred files are locked while their contents are being uploaded. The chained process will ignore files with temporary names, only handling transferred files after the final rename operation.

On most file systems the rename operation is atomic and very fast.

The same technique can be used to lock a file while uploading through SFTP transfers.

The SCP protocol does not provide a rename operation, but the total file size is advertised in the SCP upload request, which happens before the client starts pushing the content of the file.

Read more about transferring files with temporary names in our documentation page.