Notes on Forwarding Kubernetes Logs to rsyslog with fluent-bit
⚠️ This article was automatically translated by OpenAI (gpt-4o-mini). It may be edited eventually, but please be aware that it may contain incorrect information at this time.
I couldn't find a proper sample that includes structured data when forwarding K8s logs to rsyslog with fluent-bit, so here are my notes.
rsyslog Configuration
rsyslog is often pre-installed on the OS. You can check the status of rsyslog with the following command.
$ sudo service rsyslog status
Redirecting to /bin/systemctl status rsyslog.service
● rsyslog.service - System Logging Service
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2024-10-08 09:38:33 JST; 6min ago
Docs: man:rsyslogd(8)
https://www.rsyslog.com/doc/
Main PID: 526 (rsyslogd)
Tasks: 3 (limit: 128204)
Memory: 1.9M
CGroup: /system.slice/rsyslog.service
└─526 /usr/sbin/rsyslogd -n
Oct 08 09:38:33 syslog systemd[1]: Starting System Logging Service...
Oct 08 09:38:33 syslog rsyslogd[526]: [origin software="rsyslogd" swVersion="8.2102.0-15.el8" x-pid="526" x-info="https://www.rsyslog.com"] start
Oct 08 09:38:33 syslog rsyslogd[526]: imjournal: No statefile exists, /var/lib/rsyslog/imjournal.state will be created (ignore if this is first run): No such file or direc
tory [v8.2102.0-15.el8 try https://www.rsyslog.com/e/2040 ]
Oct 08 09:38:33 syslog systemd[1]: Started System Logging Service.
Oct 08 09:38:33 syslog rsyslogd[526]: imjournal: journal files changed, reloading... [v8.2102.0-15.el8 try https://www.rsyslog.com/e/0 ]
If it is not installed, you can install it with the following command.
sudo yum install rsyslog
I tried the following version of rsyslog.
$ rsyslogd -v
rsyslogd 8.2102.0-15.el8 (aka 2021.02) compiled with:
PLATFORM: aarch64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Config file: /etc/rsyslog.conf
PID file: /var/run/rsyslogd.pid
Number of Bits in RainerScript integers: 64
See https://www.rsyslog.com for more information.
Open /etc/rsyslog.conf and uncomment the following lines.
module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514")
Also, to output rfc5424 STRUCTURED-DATA, remove the following configuration,
# Use default timestamp format
module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")
And instead, add the following configuration.
module(load="builtin:omfile" Template="RSYSLOG_SyslogProtocol23Format")
Restart rsyslog.
sudo service rsyslog restart
Use the following command to tail the logs.
sudo tail -f /var/log/messages
Verify the operation. Access the rsyslog server from another terminal using telnet and send a log in rfc5424 format.
$ telnet <syslog server IP> 514
Trying *.*.*.*...
Connected to *.*.*.*.
Escape character is '^]'.
<14>1 2021-07-12T14:37:35.569848Z myhost myapp 1234 ID98 [uls@0 logtype="access" clustername="mycluster" namespace="mynamespace"] Sample app log message.
If the same log appears on the rsyslog side that you were tailing, it is OK.
<14>1 2021-07-12T14:37:35.569848Z myhost myapp 1234 ID98 [uls@0 logtype="access" clustername="mycluster" namespace="mynamespace"] Sample app log message.
If the configuration module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat") remains, the output will look like this.
Jul 12 14:37:35 myhost myapp[1234] Sample app log message.
fluent-bit Configuration
Although not essential, fluent-bit is installed using the Tanzu (Carvel) Package. The key point in this article is the configuration of fluent-bit itself, and there is not much difference regardless of the installation method. Similar configurations should work even if installed via Helm or other methods.
Note
For instructions on setting up the Tanzu Package, please refer to this article. This article assumes that the following repository is already configured.
tanzu package repository add tanzu-standard \
--url projects.registry.vmware.com/tkg/packages/standard/repo:v2024.8.21 \
--namespace tanzu-install --create-namespace
Create the following configuration file. Please replace <syslog server IP> with the appropriate value.
cat <<'EOF' > fluent-bit-values.yaml
---
namespace: tanzu-system-logging
fluent_bit:
config:
service: |
[SERVICE]
Daemon Off
Flush 1
Log_Level info
Parsers_File parsers.conf
Health_Check On
HTTP_Server On
HTTP_Port 2020
HTTP_Listen 0.0.0.0
inputs: |
[INPUT]
Name tail
Path /var/log/containers/*.log
DB /var/log/flb_kube.db
parser cri
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
parsers: |
[PARSER]
# http://rubular.com/r/tjUt3Awgg4
Name cri
Format regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
filters: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc.cluster.local:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
[FILTER]
Name lua
Match *
call add_structured_data
code function add_structured_data(tag, timestamp, record) new_record = record new_record["k8s@8"] = {} new_record["k8s@8"]["node"] = record["kubernetes"]["host"] new_record["k8s@8"]["app"] = record["kubernetes"]["labels"]["app"] new_record["k8s@8"]["container_image"] = record["kubernetes"]["container_image"] return 1, timestamp, new_record end
[FILTER]
Name nest
Match *
Operation lift
Nested_Under kubernetes
outputs: |
[OUTPUT]
Name syslog
Match *
Host <syslog server IP>
Port 514
Mode tcp
Syslog_Format rfc5424
Syslog_Hostname_key namespace_name
Syslog_Appname_key pod_name
Syslog_Procid_key container_name
Syslog_Message_key message
Syslog_SD_key k8s@8
---
EOF
Install fluent-bit with the following command.
tanzu package install -n tanzu-install fluent-bit -p fluent-bit.tanzu.vmware.com -v 2.1.6+vmware.2-tkg.1 --values-file fluent-bit-values.yaml
If the following logs appear on the rsyslog side, it is OK.
<14>1 2024-10-08T01:10:01.231783Z harbor harbor-portal-d6c99f896-2v8zm portal - [k8s@8 node="kind-worker2" container_image="docker.io/goharbor/harbor-portal:v2.11.1" app="harbor"] 10.244.2.1 - - [08/Oct/2024:01:10:01 +0000] "GET / HTTP/1.1" 200 785 "-" "kube-probe/1.30"
<14>1 2024-10-08T01:10:02.539216Z harbor harbor-registry-59f445685-7ltfq registryctl - [k8s@8 node="kind-worker3" container_image="docker.io/goharbor/harbor-registryctl:v2.11.1" app="harbor"] 10.244.1.1 - - [08/Oct/2024:01:10:02 +0000] "GET /api/health HTTP/1.1" 200 9
<14>1 2024-10-08T01:10:04.894222Z harbor harbor-registry-59f445685-7ltfq registry - [k8s@8 node="kind-worker3" container_image="docker.io/goharbor/registry-photon:v2.11.1" app="harbor"] 10.244.1.1 - - [08/Oct/2024:01:10:04 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.30"
<14>1 2024-10-08T01:10:05.741468Z kube-system kindnet-ltt8c kindnet-cni - [k8s@8 app="kindnet" node="kind-worker" container_image="docker.io/kindest/kindnetd:v20240813-c6f155d6"] I1008 01:10:05.741186 1 main.go:295] Handling node with IPs: map[192.168.228.4:{}]
<14>1 2024-10-08T01:10:05.741488Z kube-system kindnet-ltt8c kindnet-cni - [k8s@8 app="kindnet" node="kind-worker" container_image="docker.io/kindest/kindnetd:v20240813-c6f155d6"] I1008 01:10:05.741220 1 main.go:322] Node kind-control-plane has CIDR [10.244.0.0/24]
Modify the add_structured_data code as necessary.
Tip
If the configuration does not reflect when updating, restart fluent-bit with the following command.
kubectl rollout restart ds/fluent-bit -n tanzu-system-logging