# Fluentd integration

Fluentd is an open source data collector that unifies data collection and consumption for a better use and understanding of data.

See [Fluentd Documentation](https://docs.fluentd.org/).

## Integrating Fluentd with Kloudfuse Stack

See Fluentd documentation on [Installation](https://docs.fluentd.org/installation).

To configure Fluentd to send data to Kloudfuse stack, you must modify the agent’s `config`, or `values.yaml` if you install using helm.

You must configure HTTP output plugin to forward data to Kloudfuse.

## HTTP Plugin Integration

Add the configuration in [Configure HTTP Plugin](https://docs.kloudfuse.com/platform/3.2.5/agent-fluentd/#ex-http) to Fluentd agent’s `config` for HTTP configuration.

You can change this example buffer configuration to suit your application needs. However, you cannot change the end-point URI path.

Kloudfuse supports `json` and `msgpack` types for Fluentd HTTP plugin.

If you configured external ingress with TLS for Kloudfuse, use `https://<KFUSE_INGESTER_IP>:443/ingester/v1/fluentd` as the endpoint, instead.

Configure HTTP Plugin

```yaml
<match *> # Match everything
  @type http

endpoint http://<KFUSE_INGESTER_IP>:80/ingester/v1/fluentd
  open_timeout 2

<format>
    @type json
  </format>
  <buffer>
    chunk_limit_size 1048576 # 1MB
    flush_interval 10s
  </buffer>
</match>
```

## Configuration

Kloudfuse UI enables you to filter log events based on log labels and tags. Find the label selectors and filter on the left navigation bar.

|     |     |
| --- | --- |
|  | When adding recommended filters, ensure that they don’t conflict with existing filter definitions. |

To have a seamless experience with Kloudfuse while using Fluentd agent, we recommend the following configuration and customizations:

- [Kubernetes Labels](https://docs.kloudfuse.com/platform/3.2.5/agent-fluentd/#kubernetes-labels)

- [Log source](https://docs.kloudfuse.com/platform/3.2.5/agent-fluentd/#log-source)

- [Log message](https://docs.kloudfuse.com/platform/3.2.5/agent-fluentd/#log-message)

- [Agent extracted key-value pairs](https://docs.kloudfuse.com/platform/3.2.5/agent-fluentd/#key-value)

### Kubernetes Labels

Use the `kubernetes` filter to enrich the log events using Kubernetes metadata. See Fluentd download and documentation for [fluent-plugin-kubernetes_metadata_filter](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter).

If your applications deploy in a Kubernetes environment, be sure to enable this filter.

Here’s an example configuration:

Extract Kubernetes Labels

```yaml
  <filter *>  # Match everything
    @type kubernetes_metadata
    @id filter_kube_metadata_catalog
    skip_labels false
    skip_container_metadata false
    skip_namespace_metadata false
    skip_master_url true
  </filter>
```

### Log source

By default, Kloudfuse looks for the `container_name` in the message payload, as the log source. This information is present _only_ if the agent operates with a `kubernetes` filter. To use a different key as the log source, include the code in Example [Extract Log Source Label](https://docs.kloudfuse.com/platform/3.2.5/agent-fluentd/#example-log-source) under the `logs-parser` section in the Kloudfuse `custom-values.yaml` file.

Extract Log Source Label

```yaml
kf_parsing_config:
  config: |-
    - remap:
        args:
          kf_source:
            - "$.<KEY_FOR_LOG_SOURCE>" # must be JSONPath
        conditions:
          - matcher: "__kf_agent"
            value: "fluentd"
            op: "=="
```

### Log message

The Fluent Bit agent includes the log event message key in the payload. Occasionally, the agent configuration overwrites it.

|     |     |
| --- | --- |
|  | Kloudfuse looks in the payload for log message with the following key names, by default: `log`, `LOG`, `Log`, `message`, `msg`, `MSG`, and `Message`. |

Customize the key that the Kloudfuse Platform gets to easily tag log event messages by including the code in Example [Extract Log Message Label](https://docs.kloudfuse.com/platform/3.2.5/agent-fluentd/#example-log-message) following section under the `logs-parser` section in the Kloudfuse `custom-values.yaml` file.

Extract Log Message Label

```yaml
kf_parsing_config:
  config: |-
    - remap:
        args:
          kf_msg:
            - "$.<MSG_KEY_FROM_AGENT_CONFIG>" # must be JSONPath
        conditions:
          - matcher: "__kf_agent"
            value: "fluentd"
            op: "=="
```

### Agent extracted key-value pairs

Fluentd supports parsers that extract key-value pairs from an unstructured log; see Fluentd documentation on [Parser Plugins](https://docs.fluentd.org/parser).

By default, Kloudfuse adds all key-value pairs to log facets, which can be filtered in the UI.

Kloudfuse **cannot** differentiate between these key-value pairs and metadata fields added by any filter other than kubernetes. However, you can customize the Kloudfuse stack by adding a list of prefix labels. To accomplish this, include the following section under the `logs-parser` section in your Kloudfuse `custom-values.yaml` file.

|     |     |
| --- | --- |
|  | Any keys specified in log_source, message, or metadata fields from kubernetes filter are automatically treated as metadata, and not included as a log facet. |

Extract Static Custom Labels as Labels and Tags

```yaml
kf_parsing_config:
  config: |-
    - remap:
        args:
          kf_additional_tags: (1)
            - "$.<PREFIX_KEY_FOR_AGENT_KV>" # must be JSONPath
        conditions:
          - matcher: "__kf_agent"
            value: "fluentd"
            op: "=="
```

|     |     |
| --- | --- |
| **1** | `kf_additional_tags` is a list of key prefixes. Any key that is a prefix matches at the top level json gets included as a log label/tag, and not included as a log facet. |