Quick Guide for FluentD Agent for Logs

Integrating FluentD with Kloudfuse

To configure FluentD to send data to Kloudfuse stack:

1. Modify the agent’s config.

   For Helm installations, modify the values.yaml file.

2. Configure HTTP output plugin to forward data to Kloudfuse.

HTTP Plugin Integration

Add the following code to FluentD agent’s config for HTTP configuration.

<match *> # Match everything
@type http
endpoint http://<KFUSE_INGESTER_IP>:80/ingester/v1/fluentd
open_timeout 2
<format>
@type json
</format>
<buffer>
chunk_limit_size 1048576 # 1MB
flush_interval 10s
</buffer>
</match>

Host IP/DNS name can be the either internal or external Kloudfuse ingress IP, depending on the Kloudfuse stack configuration.

• Internal ingress: use port 80.
• External ingress: use port 443.

Change the buffer configuration to match your application needs. The end-point URI path must remain the same.

Fluentd is an open source data collector that unifies data collection and consumption for a better use and understanding of data.

Integrating Fluentd with Kloudfuse Stack

To configure Fluentd to send data to Kloudfuse stack, you must modify the agent’s config, or values.yaml if you install using helm.

You must configure HTTP output plugin to forward data to Kloudfuse.

Configuration

Kloudfuse UI enables you to filter log events based on log labels and tags. Find the label selectors and filter on the left navigation bar.

To have a seamless experience with Kloudfuse while using Fluentd agent, we recommend the following configuration and customizations:

Kubernetes Labels

Use the kubernetes filter to enrich the log events using Kubernetes metadata. Here’s an example configuration:

  <filter *>  # Match everything
    @type kubernetes_metadata
    @id filter_kube_metadata_catalog
    skip_labels false
    skip_container_metadata false
    skip_namespace_metadata false
    skip_master_url true
  </filter>

Log source

By default, Kloudfuse looks for the container_name in the message payload, as the log source. This information is present only if the agent operates with a kubernetes filter. To use a different key as the log source, include the code in Example under the logs-parser section in the Kloudfuse custom-values.yaml file.

Extract Log Source Label

kf_parsing_config:
  config: |-
    - remap:
        args:
          kf_source:
            - "$.<KEY_FOR_LOG_SOURCE>" # must be JSONPath
        conditions:
          - matcher: "__kf_agent"
            value: "fluentd"
            op: "=="

Log message

The Fluent Bit agent includes the log event message key in the payload. Occasionally, the agent configuration overwrites it.

Customize the key that the Kloudfuse Platform gets to easily tag log event messages by including the code following section under the logs-parser section in the Kloudfuse custom-values.yaml file.

Extract Log Message Label

kf_parsing_config:
  config: |-
    - remap:
        args:
          kf_msg:
            - "$.<MSG_KEY_FROM_AGENT_CONFIG>" # must be JSONPath
        conditions:
          - matcher: "__kf_agent"
            value: "fluentd"
            op: "=="

Agent extracted key-value pairs

Fluentd supports parsers that extract key-value pairs from an unstructured log.

By default, Kloudfuse adds all key-value pairs to log facets, which can be filtered in the UI.

Extract Static Custom Labels as Labels and Tags

kf_parsing_config:
  config: |-
    - remap:
        args:
          kf_additional_tags: (1)
            - "$.<PREFIX_KEY_FOR_AGENT_KV>" # must be JSONPath
        conditions:
          - matcher: "__kf_agent"
            value: "fluentd"
            op: "=="