# Fluent Bit for Kinesis Firehose Datastream

You can configure the Fluent Bit agent to ingest logs from Kinesis Firehose datastream directly into Kloudfuse. This integration forwards logs to a different datastore/backend, not Kloudfuse. For example, you may want to forward logs to Elastic search.

## Architecture

The following figure represents high-level architecture of the Kloudfuse-FluentBit integration.

These are the integration components:

1. Configure and define log streams in **AWS Kinesis Firehose** data stream, and connect it to an **AWS CloudWatch subscription**. See [Configure AWS Kinesis Firehose](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/aws/cloudwatch-logs/#firehose).

2. The **nginx load balancer** (deployed as part of Kloudfuse stack) forwards the incoming data to the Ingester service.

3. The **Ingester** deploys as a Kubernetes deployment within the Kloudfuse stack. It unpacks the data and streams individual log events to a Fluent-bit service, also deployed as part of the Kloudfuse stack.

4. **Fluent-bit** deploys within the Kloudfuse stack, with an HTTP plugin (see Fluent Bit [HTTP](https://docs.fluentbit.io/manual/pipeline/inputs/http) documentation) to accept incoming traffic locally within the Kubernetes cluster.

5. **Fluent-bit** subsequently forwards the incoming log events to the **ElasticSearch** cluster, using the ES plugin; see Fluent Bit documentation for [Elasticsearch](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch).

## Configuration

Consider your specific usage scenario when configuring this integration. See these sections:

- [Enable Fluent Bit Deployment](https://docs.kloudfuse.com/platform/4.0.2/data-collection/agents/fluentbit/kinesis-firehose/#enable)
- [Routing Rules](https://docs.kloudfuse.com/platform/4.0.2/data-collection/agents/fluentbit/kinesis-firehose/#routing)
- [Add, Remove, and Modify Fields](https://docs.kloudfuse.com/platform/4.0.2/data-collection/agents/fluentbit/kinesis-firehose/#change-fields)
- [Monitor](https://docs.kloudfuse.com/platform/4.0.2/data-collection/agents/fluentbit/kinesis-firehose/#monitor)

### Enable Fluent Bit Deployment

The Kloudfuse stack supports integration with [Fluent Bit agent](https://docs.kloudfuse.com/platform/4.0.2/data-collection/agents/fluentbit/overview/). These steps illustrate the specific scenario of supporting Kloudfuse integration for Elasticsearch using Fluent Bit, to configure the ingested to forward traffic to Elasticsearch.

1. Add the following `kfuse-fbit` section to global values in your `custom-values.yaml` file.

```yaml
     kfuse-fbit:
       enabled: true
```

2. Add the following code to the ingester configuration in your `custom-values.yaml` file.

```yaml
ingester:
     config:
       logs:
         msgFormat: fluent-bit
       # Possible value for 'msgFormat' are 'kfuse', 'fluent-bit' or 'all'
       # The default value is 'kfuse'. 'all' will forward logs to both Kloudfuse
       # backend and Elasticsearch backend.
```

3. Enable Fluent Bit deployment.

```yaml
fluent-bit:
     # Add any affinity rules, if you have affinity setup in your Kubernetes cluster.
     affinity: {}
     # Add any tolerations, if you have them enabled in your Kubernetes cluster.
     tolerations: []
     config:
       inputs: |-
         [INPUT]
             Name   http
             listen 0.0.0.0
             # default port is 9880. If you want to override it, uncomment line
             # below, update the value
             # port 9880
       outputs: |-
         [OUTPUT]
             Name es
             Match *
             Host <ES_HOST> (1)
             Index <ES_INDEX_NAME> (2)
             HTTP_User <ES_USER> (3)
             HTTP_Passwd <ES_PASSWD> (4)
             Retry_Limit False
             compress gzip
             Suppress_Type_Name On
     extraPorts:
     # ensure that this port matches the port defined in the input HTTP plugin.
     # If you're going with the default value, you don't need to change anything
     # below.
  - containerPort: 9880
    name: in-http
    port: 9880
    protocol: TCP
    targetPort: 9880
imagePullSecrets:
  - name: kfuse-image-pull-credentials
kind: Deployment
```

|     |     |
| --- | --- |
| **1** | Specify Elasticsearch `host`. |
| **2** | Specify Elasticsearch `index name`. |
| **3** | Specify Elasticsearch `user`. |
| **4** | Specify Elasticsearch `password`. |

### Routing Rules

The Fluent Bit agent routes log events based on tag values; see Fluent Bit documentation on [Tags](https://docs.fluentbit.io/manual/concepts/key-concepts#tag).

To forward logs to different Elasticsearch hosts or indices, use the `rewrite_tag` filter; see Fluent Bit documentation on [Rewrite Tag](https://docs.fluentbit.io/manual/pipeline/filters/rewrite-tag). Use as many `rewrite_tag` plugins as necessary for your specific situation.

|     |     |
| --- | --- |
|  | After applying a new rule defined in the `rewrite_tag` plugin, Fluent bit re-ingests the log event with the new tag from the beginning of the pipeline.<br>Be careful not to write multiple rules that lead to an infinite loop and cause runtime issues. |

### Add, Remove, and Modify Fields

To add, remove, and modify fields, use the `records_modifier` filter, as documented by Fluent Bit in [Record Modifier](https://docs.fluentbit.io/manual/pipeline/filters/record-modifier). Use as many `records_modifier` plugins as necessary for your specific situation.

### Monitor

You can also use Fluent Bit to report prometheus-style metrics. To enable collection and scraping of these metrics and ingest them into the Kloudfuse stack, add the following configuration to your `custom_values.yaml` file.

Fluent-bit publishes metrics across various input, filter, and output plugins, including the number of records processed and dropped for each plugin. See Fluent Bit documentation on [Monitoring](https://docs.fluentbit.io/manual/administration/monitoring#metric-descriptions).

|     |     |
| --- | --- |
|  | This configuration uses common keys that are shared with the basic configuration. Make sure that you merge them properly. |

```yaml
config:
  inputs: |-
    [INPUT]
        Name fluentbit_metrics
        Tag internal_metrics
        scrape_interval 15
  outputs: |-
    [OUTPUT]
        Name prometheus_exporter
        match internal_metrics
        host 0.0.0.0
        port 8080

extraPorts:
  - containerPort: 8080
    name: out-metrics
    port: 8080
    protocol: TCP
    targetPort: 8080

podAnnotations:
  prometheus.io/path: /metrics
  prometheus.io/port: "8080"
  prometheus.io/scrape: "true"
```