SFTPPlus Documentation

Start Page 8. User’s Guides 8.13. Managing logs in high volume installations
server-side client-side configuration

8.13. Managing logs in high volume installations

8.13.1. Introduction

There are 3 main phases when dealing with application logs:

  • Log generation
  • Log centralization
  • Log analysis

The focus for SFTPPlus is to provide a robust file transfer solution which can be easily integrated with any external dedicated log management tool.

In the case in which you did not find it straightforward to integrate SFTPPlus with your preferred log management tool, please get in touch and we will work towards providing a better experience.

8.13.2. Log generation

In SFTPPlus, the logs are generated using the event handlers. An event handler will receive the events generated by SFTPPlus and transform it into a format usable for the logging system.

This simple transformation is to generate an entry into a plain text log file.

SFTPPlus can store the logs as comma-separated values (CSV), making it easy to parse the input and transform it according to your needs.

SFTPPlus can also generate the events directly as syslog entries or as HTTP requests. In this case, these logs are stored on the local SFTPPlus instance. While this can help reduce the disk usage on the SFTPPlus instance, this approach is susceptible to data loss, in the case in which the network connection is broken between STPPlus and the remote server. This approach is not recommended for high volume installations.

8.13.3. Log rotation

With a default configuration SFTPPlus will rotate the log file at the end of each day. That is, it will rename the previous log file and start with an empty file.

In SFTPPlus there are 2 main types of rotations:

  • size based rotation
  • time based rotation

The size based rotation will rotate the file as soon as the log file reaches a certain size. The files on disk will look like:

path/to/server.log
path/to/server.log.1
path/to/server.log.2
path/to/server.log.3
path/to/server.log.4
path/to/server.log.5

where path/to/server.log is the main log file and the others are rotated files containing older logs.

The time based rotation will rotate the file at a specific time or regular time intervals, regardless of the size of the log file. The time based rotated files will look on disk like the example below. The actual time stamp used for the rotated files differs based on the rotation rule:

path/to/server.log
path/to/server.2018-07-30_15-17-10.log
path/to/server.2018-07-30_15-17-43.log
path/to/server.2018-07-30_15-18-16.log
path/to/server.2018-07-30_15-19-01.log
path/to/server.2018-07-30_15-21-00.log

where path/to/server.log is the main log file and the others are rotated files containing older logs.

When rotating logs, you have the option to automatically delete older rotated files and keep rotated log files up to a certain number.

8.13.4. Log centralization

You can achieve log centralization in SFTPPlus by the use of databases. SFTPPlus can store the log entries in a database. Multiple SFTPPlus instances can use the same database to become a central place for log storage.

The embedded SFTPPlus database is not suitable for high volume installations. We recommend using specialized tools for log centralization.

The standard way to achieve log centralization is by using the syslog protocol. By configuring a syslog event handler, SFTPPlus can send the logs to a remote syslog server.

The syslog protocol may not be enough and for new installations, we recommend using modern log management tools like Fluentd, Splunk, Datadog, Sumologic, or Loggly (just to name a few in no particular order).

For efficient and fault tolerant operations, we recommend storing the SFTPPlus logs as a rotated local file and use the agent provided by your logging management tool to forward the logs to a centralized server.

In this way, even if the network between the SFTPPlus instance and the centralized server is down for a brief period of time, the logs are buffered inside the local file and can be forwarded once the connection is established. Having pending logs buffered in a local file provides improved performance while at the same time mitigate data losses associated to power failures, as only queues stored in memory will be lost.

SFTPPlus provides advanced functionality for rotating log files and can automatically delete older rotated files.

The default installation will rotate log files at the end of each time. For better control over disk space used by log files, you should choose to rotate files based on size.

8.13.5. Log analysis

For log analysis, SFTPPlus only provides a basic tool to search and filter the logs, which is not suitable for high volume installations.

The log analysis component is implemented in SFTPPlus as the “Past Activity” page from the Local Manager web console.

We recommend using the analysis and filtering functionalities provided by the dedicated logging tools.