Examples

Fluent Bit

We recommend using Fluent-bit for our Observability Service because Fluent-bit is easier to configure and supports multiple log types.

Fluent Bit configuration files are composed of three or four main sections, each serving a specific purpose in defining how Fluent Bit ingests, processes, and forwards log data.

Explanation of the configs below:

In the first section [SERVICE], we set the initial configuration for the system in order to send logs.
After that we have inputs, the inputs are the way we gather data from our sources. It can have multiple [INPUT] elements, as each of them has its own configuration. There are multiple types of inputs, in this example we are using Tail and Systemd.
Finally we have the [OUTPUT] section, here we configure where we want to receive our logs, the destination where the data will go, in our case, it is enough to fill the placeholders between <> in order to receive the logs.
Note: You will need to replace placeholders like OPENSEARCH_HOST, USERNAME, PASSWORD, and ALIAS_NAME with the corresponding values.

How Fluent Bit Works

Inputs: Fluent Bit collects logs from various sources using plugins.
Parsers: Parsers turn raw log data into a structured format for easier analysis.
Filters: Filters allow you to select, convert, or remove specific logs.
Outputs: Processed logs are sent to destinations.

How to Install Fluent Bit

Setup Fluent Bit on Ubuntu:

The first step is to add our server GPG key to your keyring to ensure you can get our signed packages.

curl https://packages.fluentbit.io/fluentbit.key | gpg --dearmor > /usr/share/keyrings/fluentbit-keyring.gpg

On Ubuntu, you need to add our APT server entry to your sources lists, please add the following content at bottom of your /etc/apt/sources.list file - ensure to set CODENAME to your specific Ubuntu release name.
Now let your system update the apt package manager:
- sudo apt-get update
Using the following apt-get command you are able now to install the latest fluent-bit:
- sudo apt-get install fluent-bit
Now the following step is to instruct systemd to enable the service:
- sudo systemctl start fluent-bit

info

For more information and installation options, visit Getting Started with Fluent Bit

Example 1: Sending Container Logs

In this case we are configuring a service that listens on address 0.0.0.0 and allows monitoring a text or several text files using the Tail input type. Also it uses systemd the input which allows to collect log messages from the Journald daemon on Linux environments. After, the output is sent to the configured observability service.

config:
  service: |
    [SERVICE]
        Daemon Off
        Flush {{ .Values.flush }}
        Log_Level {{ .Values.logLevel }}
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port {{ .Values.metricsPort }}
        Health_Check On

  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On
    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On
        
  outputs: |
    [OUTPUT]
        Name               dtcloud
        Match              *
        Host               <OPENSEARCH_HOST>
        Port               443
        Path               /backend/_bulk
        HTTP_User          <USERNAME>
        HTTP_Passwd        <PASSWORD>
        Index              <ALIAS_NAME>
        Type               _doc
        Suppress_Type_Name On
        tls                On

Example 2: Sending CPU Metrics

In this simple configuration we see that we are using flush 5 which will send the output data to our configured observability service each 5 seconds. It monitors the general usage of the CPU of the VM.

[SERVICE]
     Flush 5
     Daemon off
     Log_Level info
[INPUT]
     Name cpu
     Tag  cpu
[OUTPUT]
     Name               dtcloud
     Match              *
     Host               <OPENSEARCH_HOST>
     Port               443
     Path               /backend/_bulk
     HTTP_User          <USERNAME>
     HTTP_Passwd        <PASSWORD>
     Index              <ALIAS_NAME>
     Type               _doc
     Suppress_Type_Name On
     tls                On

Example 3: Sending Syslog Logs

In this example we see that we are using the Parsers_File key which allows us to import a configuration file where our parsers are defined. We are using the this pages input which allows to collect Syslog messages through a Unix socket server (UDP or TCP) or over the network using TCP or UDP. We send the data received from Syslog to our configured Observability service.

[SERVICE]
     Flush 5
     Daemon off
     Log_Level info
[INPUT]
     Name                syslog
     Path                /tmp/in_syslog
     Buffer_Chunk_Size   32000
     Buffer_Max_Size     64000
     Receive_Buffer_Size 512000
[OUTPUT]
     Name               dtcloud
     Match              *
     Host               <OPENSEARCH_HOST>
     Port               443
     Path               /backend/_bulk
     HTTP_User          <USERNAME>
     HTTP_Passwd        <PASSWORD>
     Index              <ALIAS_NAME>
     Type               _doc
     Suppress_Type_Name On
     tls                On

Example 4: Sending Log Directly from Log or Txt file

In this example you can send log directly from your machine with any text file. Just remember that you have to give correct configuration to parsers.conf file.

Your regex in your parser.conf file should look like this:

[PARSER]
    Name        simple
    Format      regex
    Regex       ^(?<time>[^,]+), (?<message>[^,]+), (?<TEST>.+)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z

It says that log file divided by comma and each part is the group of log like time, message, detail etc.

Finally fluent-bit.conf file should look like this:

[SERVICE]
     Parsers_File /path/to/parsers.conf
[INPUT]
     Name                tail
     Parser              Parser name
     Path                /path/to/local/logfile.log
     Read_from_Head      True (it can be false if you want to read from tail)
[OUTPUT]
     Name               dtcloud
     Match              *
     Host               <OPENSEARCH_HOST>
     Port               443
     Path               /backend/_bulk
     HTTP_User          <USERNAME>
     HTTP_Passwd        <PASSWORD>
     Index              <ALIAS_NAME>
     Type               _doc
     Suppress_Type_Name On
     tls                On

Start, stop and restart commands for FluentBit:

This section provides instructions for controlling the Fluent Bit service using systemd.

Stop Fluent Bit

sudo systemctl stop fluent-bit

This will gracefully stop Fluent Bit, allowing it to finish processing any pending events before shutting down.

Start Fluent Bit

systemctl start fluent-bit

This will initiate the Fluent Bit service, allowing it to start processing logs according to its configuration.

Restart Fluent Bit

systemctl restart fluent-bit

This will stop the running Fluent Bit service, reload its configuration, and then start it again.

Examples

Fluent Bit​

How Fluent Bit Works​

How to Install Fluent Bit​

Example 1: Sending Container Logs​

Example 2: Sending CPU Metrics​

Example 3: Sending Syslog Logs​

Example 4: Sending Log Directly from Log or Txt file​

Start, stop and restart commands for FluentBit:​