# Fluent Bit for Kinesis Firehose Datastream

You can configure the Fluent Bit agent to ingest logs from Kinesis Firehose datastream directly into Kloudfuse. This integration forwards logs to a different datastore/backend, not Kloudfuse. For example, you may want to forward logs to Elastic search.

## Architecture

The following figure represents high-level architecture of the Kloudfuse-FluentBit integration.

These are the integration components:

1. Configure and define log streams in **AWS Kinesis Firehose** data stream, and connect it to an **AWS CloudWatch subscription**. See [Configure AWS Kinesis Firehose](https://docs.kloudfuse.com/platform/3.2.5/cloud-service-aws-cloudwatch-logs/#firehose).
2. The **nginx load balancer** (deployed as part of Kloudfuse stack) forwards the incoming data to the Ingester service.
3. The **Ingester** deploys as a Kubernetes deployment within the Kloudfuse stack. It unpacks the data and streams individual log events to a Fluent-bit service, also deployed as part of the Kloudfuse stack.
4. **Fluent-bit** deploys within the Kloudfuse stack, with an HTTP plugin (see Fluent Bit [HTTP](https://docs.fluentbit.io/manual/pipeline/inputs/http) documentation) to accept incoming traffic locally within the Kubernetes cluster.
5. **Fluent-bit** subsequently forwards the incoming log events to the **ElasticSearch** cluster, using the ES plugin; see Fluent Bit documentation for [Elasticsearch](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch).

## Configuration

Consider your specific usage scenario when configuring this integration. See these sections:

- [Enable Fluent Bit Deployment](https://docs.kloudfuse.com/platform/3.2.5/agent-fluentbit-kinesis-firehouse/#enable)
- [Routing Rules](https://docs.kloudfuse.com/platform/3.2.5/agent-fluentbit-kinesis-firehouse/#routing)
- [Add, Remove, and Modify Fields](https://docs.kloudfuse.com/platform/3.2.5/agent-fluentbit-kinesis-firehouse/#change-fields)
- [Monitor](https://docs.kloudfuse.com/platform/3.2.5/agent-fluentbit-kinesis-firehouse/#monitor)

### Enable Fluent Bit Deployment

The Kloudfuse stack supports integration with [Fluent Bit agent](https://docs.kloudfuse.com/platform/3.2.5/agent-fluentbit/). These steps illustrate the specific scenario of supporting Kloudfuse integration for Elasticsearch using Fluent Bit, to configure the ingested to forward traffic to Elasticsearch.

1. Add the following `kfuse-fbit` section to global values in your `custom-values.yaml` file.

```yaml
       kfuse-fbit:
         enabled: true
   ```
   
2. Add the following code to the ingester configuration in your `custom-values.yaml` file.

```yaml
   ingester:
        config:
          logs:
            msgFormat: fluent-bit
          # Possible value for 'msgFormat' are 'kfuse', 'fluent-bit' or 'all'
          # The default value is 'kfuse'. 'all' will forward logs to both Kloudfuse
          # backend and Elasticsearch backend.
   ```
   
3. Enable Fluent Bit deployment.

```yaml
   fluent-bit:
        # Add any affinity rules, if you have affinity setup in your Kubernetes cluster.
        affinity: {}
        # Add any tolerations, if you have them enabled in your Kubernetes cluster.
        tolerations: []
        config:
          inputs: |-
            [INPUT]
                Name   http
                listen 0.0.0.0
                # default port is 9880. If you want to override it, uncomment line
                # below, update the value
                # port 9880
          outputs: |-
            [OUTPUT]
                Name es
                Match *
                Host <ES_HOST> (1)
                Index <ES_INDEX_NAME> (2)
                HTTP_User <ES_USER> (3)
                HTTP_Passwd <ES_PASSWD> (4)
                Retry_Limit False
                compress gzip
                Suppress_Type_Name On
        extraPorts:
        # ensure that this port matches the port defined in the input HTTP plugin.
        # If you're going with the default value, you don't need to change anything
        # below.
     - containerPort: 9880
       name: in-http
       port: 9880
       protocol: TCP
       targetPort: 9880
   imagePullSecrets:
     - name: kfuse-image-pull-credentials
   kind: Deployment
   ```
   
   |     |     |
   | --- | --- |
   | **1** | Specify Elasticsearch `host`. |
   | **2** | Specify Elasticsearch `index name`. |
   | **3** | Specify Elasticsearch `user`. |
   | **4** | Specify Elasticsearch `password`. |

### Routing Rules

The Fluent Bit agent routes log events based on tag values; see Fluent Bit documentation on [Tags](https://docs.fluentbit.io/manual/concepts/key-concepts#tag).

To forward logs to different Elasticsearch hosts or indices, use the `rewrite_tag` filter; see Fluent Bit documentation on [Rewrite Tag](https://docs.fluentbit.io/manual/pipeline/filters/rewrite-tag). Use as many `rewrite_tag` plugins as necessary for your specific situation.

|     |     |
| --- | --- |
|  | After applying a new rule defined in the `rewrite_tag` plugin, Fluent bit re-ingests the log event with the new tag from the beginning of the pipeline.<br>Be careful not to write multiple rules that lead to an infinite loop and cause runtime issues. |

### Add, Remove, and Modify Fields

To add, remove, and modify fields, use the `records_modifier` filter, as documented by Fluent Bit in [Record Modifier](https://docs.fluentbit.io/manual/pipeline/filters/record-modifier). Use as many `records_modifier` plugins as necessary for your specific situation.

### Monitor

You can also use Fluent Bit to report prometheus-style metrics. To enable collection and scraping of these metrics and ingest them into the Kloudfuse stack, add the following configuration to your `custom_values.yaml` file.

Fluent-bit publishes metrics across various input, filter, and output plugins, including the number of records processed and dropped for each plugin. See Fluent Bit documentation on [Monitoring](https://docs.fluentbit.io/manual/administration/monitoring#metric-descriptions).

|     |     |
| --- | --- |
|  | This configuration uses common keys that are shared with the basic configuration. Make sure that you merge them properly. |

```yaml
   config:
     inputs: |-
       [INPUT]
           Name fluentbit_metrics
           Tag internal_metrics
           scrape_interval 15
     outputs: |-
       [OUTPUT]
           Name prometheus_exporter
           match internal_metrics
           host 0.0.0.0
           port 8080

extraPorts:
     - containerPort: 8080
       name: out-metrics
       port: 8080
       protocol: TCP
       targetPort: 8080

podAnnotations:
     prometheus.io/path: /metrics
     prometheus.io/port: "8080"
     prometheus.io/scrape: "true"
   ```