--- title: Notes on Forwarding Kubernetes Logs to rsyslog with fluent-bit tags: ["Kubernetes", "TKG", "Tanzu", "Fluent Bit", "syslog", "rsyslog"] categories: ["Dev", "CaaS", "Kubernetes", "TKG", "Extensions", "FluentBit"] date: 2024-10-08T01:13:09Z updated: 2024-10-08T04:05:55Z --- > ⚠️ This article was automatically translated by OpenAI (gpt-4o-mini). > It may be edited eventually, but please be aware that it may contain incorrect information at this time. I couldn't find a proper sample that includes structured data when forwarding K8s logs to rsyslog with fluent-bit, so here are my notes. ### rsyslog Configuration rsyslog is often pre-installed on the OS. You can check the status of rsyslog with the following command. ``` $ sudo service rsyslog status Redirecting to /bin/systemctl status rsyslog.service ● rsyslog.service - System Logging Service Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2024-10-08 09:38:33 JST; 6min ago Docs: man:rsyslogd(8) https://www.rsyslog.com/doc/ Main PID: 526 (rsyslogd) Tasks: 3 (limit: 128204) Memory: 1.9M CGroup: /system.slice/rsyslog.service └─526 /usr/sbin/rsyslogd -n Oct 08 09:38:33 syslog systemd[1]: Starting System Logging Service... Oct 08 09:38:33 syslog rsyslogd[526]: [origin software="rsyslogd" swVersion="8.2102.0-15.el8" x-pid="526" x-info="https://www.rsyslog.com"] start Oct 08 09:38:33 syslog rsyslogd[526]: imjournal: No statefile exists, /var/lib/rsyslog/imjournal.state will be created (ignore if this is first run): No such file or direc tory [v8.2102.0-15.el8 try https://www.rsyslog.com/e/2040 ] Oct 08 09:38:33 syslog systemd[1]: Started System Logging Service. Oct 08 09:38:33 syslog rsyslogd[526]: imjournal: journal files changed, reloading... [v8.2102.0-15.el8 try https://www.rsyslog.com/e/0 ] ``` If it is not installed, you can install it with the following command. ``` sudo yum install rsyslog ``` I tried the following version of rsyslog. ``` $ rsyslogd -v rsyslogd 8.2102.0-15.el8 (aka 2021.02) compiled with: PLATFORM: aarch64-redhat-linux-gnu PLATFORM (lsb_release -d): FEATURE_REGEXP: Yes GSSAPI Kerberos 5 support: Yes FEATURE_DEBUG (debug build, slow code): No 32bit Atomic operations supported: Yes 64bit Atomic operations supported: Yes memory allocator: system default Runtime Instrumentation (slow code): No uuid support: Yes systemd support: Yes Config file: /etc/rsyslog.conf PID file: /var/run/rsyslogd.pid Number of Bits in RainerScript integers: 64 See https://www.rsyslog.com for more information. ``` Open `/etc/rsyslog.conf` and uncomment the following lines. ``` module(load="imtcp") # needs to be done just once input(type="imtcp" port="514") ``` Also, to output rfc5424 [STRUCTURED-DATA](https://datatracker.ietf.org/doc/html/rfc5424#section-6.3), remove the following configuration, ``` # Use default timestamp format module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat") ``` And instead, add the following configuration. ``` module(load="builtin:omfile" Template="RSYSLOG_SyslogProtocol23Format") ``` Restart rsyslog. ``` sudo service rsyslog restart ``` Use the following command to tail the logs. ``` sudo tail -f /var/log/messages ``` Verify the operation. Access the rsyslog server from another terminal using telnet and send a log in rfc5424 format. ``` $ telnet 514 Trying *.*.*.*... Connected to *.*.*.*. Escape character is '^]'. <14>1 2021-07-12T14:37:35.569848Z myhost myapp 1234 ID98 [uls@0 logtype="access" clustername="mycluster" namespace="mynamespace"] Sample app log message. ``` If the same log appears on the rsyslog side that you were tailing, it is OK. ``` <14>1 2021-07-12T14:37:35.569848Z myhost myapp 1234 ID98 [uls@0 logtype="access" clustername="mycluster" namespace="mynamespace"] Sample app log message. ``` If the configuration `module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")` remains, the output will look like this. ``` Jul 12 14:37:35 myhost myapp[1234] Sample app log message. ``` ### fluent-bit Configuration Although not essential, fluent-bit is installed using the Tanzu (Carvel) Package. The key point in this article is the configuration of fluent-bit itself, and there is not much difference regardless of the installation method. Similar configurations should work even if installed via Helm or other methods. > [!NOTE] > For instructions on setting up the Tanzu Package, please refer to this article. This article assumes that the following repository is already configured. > > ``` > tanzu package repository add tanzu-standard \ > --url projects.registry.vmware.com/tkg/packages/standard/repo:v2024.8.21 \ > --namespace tanzu-install --create-namespace > ``` Create the following configuration file. Please replace `` with the appropriate value. ```yaml cat <<'EOF' > fluent-bit-values.yaml --- namespace: tanzu-system-logging fluent_bit: config: service: | [SERVICE] Daemon Off Flush 1 Log_Level info Parsers_File parsers.conf Health_Check On HTTP_Server On HTTP_Port 2020 HTTP_Listen 0.0.0.0 inputs: | [INPUT] Name tail Path /var/log/containers/*.log DB /var/log/flb_kube.db parser cri Tag kube.* Mem_Buf_Limit 5MB Skip_Long_Lines On parsers: | [PARSER] # http://rubular.com/r/tjUt3Awgg4 Name cri Format regex Regex ^(?