# Routing Logs to Additional Endpoints

Kloudfuse integrates with the Fluentbit agent to ingest logs from the Kinesis Firehose data stream. This enables you to forward logs to a different datastore or backend, instead of just Kloudfuse. For example, you could examine logs through ElasticSearch using the appropriate configuration on Fluentbit.

## Architecture

1. Configure and define log streams in AWS Kinesis Firehose data stream, and connect it them to an AWS CloudWatch subscription.

See [AWS CloudWatch Logs Integration](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/aws/cloudwatch-logs/).

2. The Kloudfuse stack receives the log data stream, and process it:
   
   1. The **Nginx load balancer** forwards the incoming data to the Kloudfuse Ingester service.
   
   2. The **Ingester**, a Kubernetes instance within the Kloudfuse stack, unpacks the data and streams individual log events to the Fluentbit service.
   
   3. **Fluentbit**, deployed with an HTTP plugin within the Kloudfuse stack, accepts the incoming traffic within the Kubernetes cluster.
   
      See Fluentbit documentation on [HTTP](https://docs.fluentbit.io/manual/pipeline/inputs/http).
  
   4. Subsequently, Fluentbit forwards the log events to the ElasticSearch cluster using the ES Plugin.
      
      See Fluentbit documentation on [Elasticsearch](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch).

## Configuration

### Enable Fluentbit Deployment

The Kloudfuse stack independently supports integration with the Fluentbit agent. See [Fluent Bit API for Ingesting Custom Logs](https://docs.kloudfuse.com/platform/4.0.2/data-collection/agents/fluentbit/api/).

Use the following configuration, specific to supporting Kloudfuse integration for ElasticSearch, to enable the ingester to forward traffic:

1. Add this code to the global values section in the `custom-values.yaml` file.

```yaml
     kfuse-fbit:
       enabled: true
```

2. Add this code to the ingester configuration section in your `custom-values.yaml` file.

```yaml
ingester:
     config:
       logs:
         msgFormat: fluent-bit (1)
```

|     |     |
| --- | --- |
| **1** | `msgFormat`: Possible values are `kfuse` (default)``, `fluent-bit``, or `all` (forwards logs to both Kloudfuse backend and ElasticSearch backend). |

3. Add this code to enable Fluentbit deployment.

Be sure to replace the placeholders, `<…​>`, with appropriate values.

```yaml
fluent-bit:
     affinity: {} (1)
     tolerations: [] (2)
     config:
       inputs: |-
         [INPUT]
             Name   http
             listen 0.0.0.0
             # default port is 9880. If you want to override it, uncomment line
             # below, update the value
             # port 9880
       outputs: |-
         [OUTPUT]
             Name es
             Match *
             Host <ES_HOST>
             Index <ES_INDEX_NAME>
             HTTP_User <ES_USER>
             HTTP_Passwd <ES_PASSWD>
             Retry_Limit False
             compress gzip
             Suppress_Type_Name On
     extraPorts: (3)
  - containerPort: 9880
    name: in-http
    port: 9880
    protocol: TCP
    targetPort: 9880
imagePullSecrets:
  - name: kfuse-image-pull-credentials
kind: Deployment
```

|     |     |
| --- | --- |
| **1** | `affinity`: Add any affinity rules if affinity is set up in your Kubernetes cluster. |
| **2** | `tolerations`: Add any tolerations that are enabled in your Kubernetes cluster. |
| **3** | `extraPorts`: Ensure that the port matches the port definition in the input HTTP plugin. If using the default value, don’t change anything. |

### Routing Rules

The Fluentbit agent routes log events based on tag values; see Fluentbit documentation on [Tags](https://docs.fluentbit.io/manual/concepts/key-concepts#tag).

To forward logs to a different ElasticSearch host or index, use the `rewrite_tag` filter; see Fluentbit documentation on the [Rewrite Tag](https://docs.fluentbit.io/manual/pipeline/filters/rewrite-tag) filter. You can add as many `rewrite_tag` plugins as necessary for your business scenario.

When you apply the rule defined by the `rewrite_tag` filter, the Fluentbit service re-ingests the log event from the beginning of the pipeline, using the new tag.

### Manage Fields: Add, Remove, Modify

Fluentbit enables you to add and remove fields using the `record_modifier` filter. Similarly to using the `rewrite_tag` filter, you can add and define as many filters as you want.

See Fluentbit documentation on [Record Modifier](https://docs.fluentbit.io/manual/pipeline/filters/record-modifier).

### Monitor

Fluentbit can also report Prometheus-style metrics. To enable the collection and scraping of these metrics and ingest them into the Kloudfuse stack, add the following configuration to your `custom_values.yaml` file.

|     |     |
| --- | --- |
|  | Fluentbit publishes metrics across various input, filter, and output plugins. This includes the number of records processed and dropped for each plugin.<br>This configuration contains some common keys that are shared with the basic configuration; ensure that you merge them correctly. |

```yaml
config:
  inputs: |-
    [INPUT]
        Name fluentbit_metrics
        Tag internal_metrics
        scrape_interval 15
  outputs: |-
    [OUTPUT]
        Name prometheus_exporter
        match internal_metrics
        host 0.0.0.0
        port 8080

extraPorts:
  - containerPort: 8080
    name: out-metrics
    port: 8080
    protocol: TCP
    targetPort: 8080

podAnnotations:
  prometheus.io/path: /metrics
  prometheus.io/port: "8080"
  prometheus.io/scrape: "true"
```

See Fluentbit documentation on [Monitoring](https://docs.fluentbit.io/manual/administration/monitoring#metric-descriptions).