# Routing Logs to Additional Endpoints

Kloudfuse integrates with the Fluentbit agent to ingest logs from the Kinesis Firehose data stream. This enables you to forward logs to a different datastore or backend, instead of just Kloudfuse. For example, you could examine logs through ElasticSearch using the appropriate configuration on Fluentbit.

## Architecture

Kloudfuse Integrates with Fluentbit Agent to stream data to an ElasticSearch Database

1. Configure and define log streams in AWS Kinesis Firehose data stream, and connect it them to an AWS CloudWatch subscription.

See [AWS CloudWatch Logs Integration](https://docs.kloudfuse.com/platform/3.3.6/cloud-service-aws-cloudwatch-logs/).

2. The Kloudfuse stack receives the log data stream, and process it:

1. The **Nginx load balancer** forwards the incoming data to the Kloudfuse Ingester service.
   2. The **Ingester**, a Kubernetes instance within the Kloudfuse stack, unpacks the data and streams individual log events to the Fluentbit service.
   3. **Fluentbit**, deployed with an HTTP plugin within the Kloudfuse stack, accepts the incoming traffic within the Kubernetes cluster.

See Fluentbit documentation on [HTTP](https://docs.fluentbit.io/manual/pipeline/inputs/http).
   4. Subsequently, Fluentbit forwards the log events to the ElasticSearch cluster using the ES Plugin.

See Fluentbit documentation on [Elasticsearch](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch).

## Configuration

### Enable Fluentbit Deployment

The Kloudfuse stack independently supports integration with the Fluentbit agent. See [Fluent Bit API for Ingesting Custom Logs](https://docs.kloudfuse.com/platform/3.3.6/agent-fluentbit-api/).

Use the following configuration, specific to supporting Kloudfuse integration for ElasticSearch, to enable the ingester to forward traffic:

1. Add this code to the global values section in the `custom-values.yaml` file.

```yaml
         kfuse-fbit:
           enabled: true
   ```

2. Add this code to the ingester configuration section in your `custom-values.yaml` file.

```yaml
   ingester:
         config:
           logs:
             msgFormat: fluent-bit (1)
   ```

|     |     |
   | --- | --- |
   | **1** | `msgFormat`: Possible values are `kfuse` (default)``, `fluent-bit``, or `all` (forwards logs to both Kloudfuse backend and ElasticSearch backend). |

3. Add this code to enable Fluentbit deployment.

Be sure to replace the placeholders, `<…​>`, with appropriate values.

```yaml
   fluent-bit:
         affinity: {} (1)
         tolerations: [] (2)
         config:
           inputs: |-
             [INPUT]
                 Name   http
                 listen 0.0.0.0
                 # default port is 9880. If you want to override it, uncomment line
                 # below, update the value
                 # port 9880
           outputs: |-
             [OUTPUT]
                 Name es
                 Match *
                 Host <ES_HOST>
                 Index <ES_INDEX_NAME>
                 HTTP_User <ES_USER>
                 HTTP_Passwd <ES_PASSWD>
                 Retry_Limit False
                 compress gzip
                 Suppress_Type_Name On
         extraPorts: (3)
      - containerPort: 9880
        name: in-http
        port: 9880
        protocol: TCP
        targetPort: 9880
   imagePullSecrets:
     - name: kfuse-image-pull-credentials
   kind: Deployment
   ```

|     |     |
   | --- | --- |
   | **1** | `affinity`: Add any affinity rules if affinity is set up in your Kubernetes cluster. |
   | **2** | `tolerations`: Add any tolerations that are enabled in your Kubernetes cluster. |
   | **3** | `extraPorts`: Ensure that the port matches the port definition in the input HTTP plugin. If using the default value, don’t change anything. |

### Routing Rules

The Fluentbit agent routes log events based on tag values; see Fluentbit documentation on [Tags](https://docs.fluentbit.io/manual/concepts/key-concepts#tag).

To forward logs to a different ElasticSearch host or index, use the `rewrite_tag` filter; see Fluentbit documentation on the [Rewrite Tag](https://docs.fluentbit.io/manual/pipeline/filters/rewrite-tag) filter. You can add as many `rewrite_tag` plugins as necessary for your business scenario.

When you apply the rule defined by the `rewrite_tag` filter, the Fluentbit service re-ingests the log event from the beginning of the pipeline, using the new tag.

### Manage Fields: Add, Remove, Modify

Fluentbit enables you to add and remove fields using the `record_modifier` filter. Similarly to using the `rewrite_tag` filter, you can add and define as many filters as you want.

See Fluentbit documentation on [Record Modifier](https://docs.fluentbit.io/manual/pipeline/filters/record-modifier).

### Monitor

Fluentbit can also report Prometheus-style metrics. To enable the collection and scraping of these metrics and ingest them into the Kloudfuse stack, add the following configuration to your `custom_values.yaml` file.

|     |     |
| --- | --- |
|  | Fluentbit publishes metrics across various input, filter, and output plugins. This includes the number of records processed and dropped for each plugin.<br>This configuration contains some common keys that are shared with the basic configuration; ensure that you merge them correctly. |

```yaml
config:
  inputs: |-
    [INPUT]
        Name fluentbit_metrics
        Tag internal_metrics
        scrape_interval 15
  outputs: |-
    [OUTPUT]
        Name prometheus_exporter
        match internal_metrics
        host 0.0.0.0
        port 8080

extraPorts:
  - containerPort: 8080
    name: out-metrics
    port: 8080
    protocol: TCP
    targetPort: 8080

podAnnotations:
  prometheus.io/path: /metrics
  prometheus.io/port: "8080"
  prometheus.io/scrape: "true"
```

See Fluentbit documentation on [Monitoring](https://docs.fluentbit.io/manual/administration/monitoring#metric-descriptions).