Syslog-ng is a freely available and open-source interpretation of the Syslog protocol. It functions as an improved log daemon and offers extensive support for various input and output techniques, including Syslog, unstructured text, queueing, SQL, and NoSQL.
In addition to that, it is also a log management tool that enhances the effectiveness of your SIEM (Security Information and Event Management) solution by decreasing the volume of data and enhancing the data quality that is fed into your SIEM system.
Installing Syslog-ng
Syslog-ng is included in the package of most popular Linux distributions, and it is typically available for installation from official repositories. If you only require the basic functionalities of Syslog, you can simply install the package from your distribution repository.
However, if you wish to utilize the advanced features offered by newer versions of syslog-ng (such as sending log messages to Elasticsearch or Apache Kafka), you will need to either compile the Syslog-ng from its source code or install it from unofficial repositories.
Install Syslog-ng on Ubuntu or Debian
To install syslog-ng on Ubuntu or Debian, follow the steps below. These instructions are specific to Ubuntu 20.04 but can be adapted with minor modifications for other supported distributions by adjusting the URLs.
1. Begin by downloading and installing the release key. Run the following command:
wget -qO - https://ose-repo.syslog-ng.com/apt/syslog-ng-ose-pub.asc | sudo apt-key add -
2. Next, add the repository that contains the latest stable build of Syslog to the APT sources. For example, on Ubuntu 20.04, execute the following command:
echo "deb https://ose-repo.syslog-ng.com/apt/ stable ubuntu-focal" | sudo tee -a /etc/apt/sources.list.d/syslog-ng-ose.list
3. Update the package lists by running the following command:
apt-get update
4. Finally, install syslog-ng and any desired sub-packages using the following command:
apt-get install syslog-ng-core syslog-ng-scl
By following these steps, you can successfully install syslog-ng on your Ubuntu or Debian system.
Configuring Syslog-ng to Apica
apica.io facilitates the process of receiving data from Syslog-ng for ingestion. To configure log forwarding from Syslog-ng to Apica, certain steps need to be followed by modifying the Syslog-ng configuration file.
It offers support for both TCP and UDP protocols, whereas Apica only allows log ingestion over the TCP protocol. This choice is made to ensure that packets are not lost or dropped during transmission. TCP relies on acknowledgments from the receiver to guarantee the successful delivery of packets.
Apica hosts the Syslog protocol at port 514 by default, but it is possible to modify and customize the ports according to specific requirements. Additionally, an additional port for TLS (Transport Layer Security) is available at port 7514, offering an added layer of security for log transmission if needed.
INPUT
@version: 3.36
@include "scl.conf"
options {
time-reap(30);
mark-freq(10);
keep-hostname(yes);
chain-hostnames(no);
};
source s_local {
system();
internal();
};
The following configuration commonly forwards various system and internal logs, including sshd, systemd, and NetworkManager.
@version: 3.36
@include "scl.conf"
log {
source { system(); };
destination { file("/var/log/syslog"); };
};
OUTPUT
Python syslog-ng destination for apica.io
The Syslog-ng python destination driver for apica.io is provided as a Python package and can be installed using pip. To enable support for the Python destination, you need to first install the Python destination support for it. Below is an example of how to do this on an Ubuntu system:
apt-get install syslog-ng-mod-python
For information on installing the syslog-ng python destination on other operating systems, you can refer to the syslog-ng website. Once you have completed this step, you can proceed to install the apica.io driver. Please note that this assumes you already have Python 3 installed on your system.
You can find more details about the apica.io driver at the following link: https://pypi.org/project/Apicaaidstsyslogng/
To install the apica.io driver, you can use either of the following commands, depending on your Python installation:
pip install --upgrade Apicaaidstsyslogng
pip3 install --upgrade Apicaaidstsyslogng
Checking Python2/3 linkage for syslog-ng
@version: 3.38
source s_dummy {
};
python {
import sys
class VersionTest(object):
def init(self, options):
print(sys.version)
return True
def send(self, msg):
return True
};
destination d_test {
python(
class("VersionTest")
);
};
log { source(s_dummy); destination(d_test); };
By creating and running this configuration file, you can verify if the python linkage with syslog is functioning correctly.
#syslog-ng -f ./dummy-syslog-ng.conf --foreground
3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0]
# syslog-ng -f ./dummy-syslog-ng.conf --foreground
2.7.17 (default, Jul 1 2022, 15:56:32)
[GCC 7.5.0]
Creating a Apica.conf for the driver
[Apica]
# Host or IP for apica.io
host = my-Apica-cluster.example.com
# Protocol https
protocol = https
# API Ingest token
key = <JWT Token>
# destination reachability timeout in seconds
timeout = 5
Creating a syslog-ng destination for apica.io
You are now ready to update your syslog-ng conf to add a apica.io python destination.
destination d_Apica {
python(
batch-timeout(500)
batch-lines(400)
class("Apicaaidstsyslogng.LogDestination")
value-pairs(
key(ISODATE)
scope(rfc5424 nv-pairs)
)
options(config "/etc/syslog-ng/Apica.conf")
options(workers-batch 8)
options(worker-batch-lines 25)
options(loglevel WARN)
);
};
# Connect the source to the Apica destination
log { source(s_local); destination(d_Apica); };
To ensure compatibility with apica.io, it is important to note that it only supports RFC3339/ISO8601 timestamps. Therefore, when using the Python destination, it is necessary to include the ISODATE key in the log message, which will be sent as the timestamp.
Additionally, the scope for value-pairs should include rfc5424. This allows the extraction of standard syslog fields such as host, application, pid, sdata, and others.
The python destination offers some useful options to enhance functionality and control. These options may include features like filtering log messages, manipulating log data, or performing custom actions based on specific conditions.
Name | Values | Default | Notes |
|---|---|---|---|
batch-timeout | e.g. 500 (milliseconds) | NA | Refer to syslog-ng documentation. This allows for batching logs |
batch-lines | e.g. 400 | NA | Refer to syslog-ng documentation. This limits the max batch size allowed before python plugin calls a flush |
class | Apicaaidstsyslogng.LogDestination | NA | Specifies the apica.io python driver class to load |
The apica.io driver provides various options, you can check them all out here.
Organizing data in apica.io
With apica.io, data is organized into flows, which are comprised of a Namespace, an application name, and one or more subflows or ProcIds. This flexible structure enables efficient mapping of various legacy and cloud-native environments within apica.io, requiring minimal configuration.
To configure the namespace, application, and cluster_id mappings, it is necessary to set them in the syslog-ng.conf file. Failure to configure these mappings will result in the utilization of default mappings provided by apica.io.
Additional output methods
Alternate methods for data transmission via syslog-ng are documented; however, it is important to note that these methods are not recommended for production and scaling purposes. While they may provide additional options for data push, they may lack the necessary robustness and efficiency required for large-scale deployments. It is advised to follow the recommended methods provided by apica.io for optimal performance and reliability.
