# Quick Guide for FluentD Agent for Logs

## Integrating FluentD with Kloudfuse

To configure FluentD to send data to Kloudfuse stack:

1. Modify the agent’s `config`.
   
   For Helm installations, modify the `values.yaml` file.
2. Configure HTTP output plugin to forward data to Kloudfuse.

## HTTP Plugin Integration

Add the following code to FluentD agent’s `config` for HTTP configuration.

```none
<match *> # Match everything
@type http
endpoint http://<KFUSE_INGESTER_IP>:80/ingester/v1/fluentd
open_timeout 2
<format>
@type json
</format>
<buffer>
chunk_limit_size 1048576 # 1MB
flush_interval 10s
</buffer>
</match>
```

Host IP/DNS name can be the either internal or external Kloudfuse ingress IP, depending on the Kloudfuse stack configuration.

- Internal ingress: use port 80.
- External ingress: use port 443.

Change the buffer configuration to match your application needs. The end-point URI path must remain the same.

Fluentd is an open source data collector that unifies data collection and consumption for a better use and understanding of data.

## Integrating Fluentd with Kloudfuse Stack

To configure Fluentd to send data to Kloudfuse stack, you must modify the agent’s `config`, or `values.yaml` if you install using helm.

You must configure HTTP output plugin to forward data to Kloudfuse.

## Configuration

Kloudfuse UI enables you to filter log events based on log labels and tags. Find the label selectors and filter on the left navigation bar.

|     |     |
| --- | --- |
|  | When adding recommended filters, ensure that they don’t conflict with existing filter definitions. |

To have a seamless experience with Kloudfuse while using Fluentd agent, we recommend the following configuration and customizations:

### Kubernetes Labels

Use the `kubernetes` filter to enrich the log events using Kubernetes metadata. Here’s an example configuration:

```yaml
  <filter *>  # Match everything
    @type kubernetes_metadata
    @id filter_kube_metadata_catalog
    skip_labels false
    skip_container_metadata false
    skip_namespace_metadata false
    skip_master_url true
  </filter>
```

### Log source

By default, Kloudfuse looks for the `container_name` in the message payload, as the log source. This information is present _only_ if the agent operates with a `kubernetes` filter. To use a different key as the log source, include the code in Example under the `logs-parser` section in the Kloudfuse `custom-values.yaml` file.

Extract Log Source Label

```yaml
kf_parsing_config:
  config: |-
    - remap:
        args:
          kf_source:
            - "$.<KEY_FOR_LOG_SOURCE>" # must be JSONPath
        conditions:
          - matcher: "__kf_agent"
            value: "fluentd"
            op: "=="
```

### Log message

The Fluent Bit agent includes the log event message key in the payload. Occasionally, the agent configuration overwrites it.

|     |     |
| --- | --- |
|  | Kloudfuse looks in the payload for log message with the following key names, by default: `log`, `LOG`, `Log`, `message`, `msg`, `MSG`, and `Message`. |

Customize the key that the Kloudfuse Platform gets to easily tag log event messages by including the code following section under the `logs-parser` section in the Kloudfuse `custom-values.yaml` file.

Extract Log Message Label

```yaml
kf_parsing_config:
  config: |-
    - remap:
        args:
          kf_msg:
            - "$.<MSG_KEY_FROM_AGENT_CONFIG>" # must be JSONPath
        conditions:
          - matcher: "__kf_agent"
            value: "fluentd"
            op: "=="
```

### Agent extracted key-value pairs

Fluentd supports parsers that extract key-value pairs from an unstructured log.

By default, Kloudfuse adds all key-value pairs to log facets, which can be filtered in the UI.

Extract Static Custom Labels as Labels and Tags

```yaml
kf_parsing_config:
  config: |-
    - remap:
        args:
          kf_additional_tags: (1)
            - "$.<PREFIX_KEY_FOR_AGENT_KV>" # must be JSONPath
        conditions:
          - matcher: "__kf_agent"
            value: "fluentd"
            op: "=="
```

|     |     |
| --- | --- |
| **1** | `kf_additional_tags` is a list of key prefixes. Any key that is a prefix matches at the top level json gets included as a log label/tag, and not included as a log facet. |