Ingestion Authentication with API Key :: Kloudfuse Docs

Ingestion Authentication with API Key

To enable authentication for ingestion using API keys, perform these steps:

Generate the AUTH_TOKEN

Generate an AUTH_TOKEN, and store its value in a safe location. You use it repeatedly.

AUTH_TOKEN=`cat /dev/urandom | env LC_ALL=C tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1`

Encode the AUTH_TOKEN using Base64; this generates the AUTH_TOKEN_ENCODED.
```
AUTH_TOKEN_ENCODED=`echo -n $AUTH_TOKEN | base64`
```

Configure Telemetry Agents and Sources

Prometheus Remote Write

Update the prometheus remote write configuration:

prometheus.yml:
     remote_write:
  - url: https://<customer>.kloudfuse.io/ingester/write
    authorization:
      credentials: <AUTH_TOKEN>

When using the Prometheus operator, use the following configuration that contains the AuthToken:

remoteWrite:
  - authorization:
      credentials:
        key: <AUTH_TOKEN> (1)
        name: kf-auth-ingest (2)
    url: https://<customer>.kloudfuse.io/ingester/write


1	`<AUTH_TOKEN>`: The key inside this secret that contains the encoded `authToken`.
2	`kf-auth-ingest`: The secret name.

Fluent Bit

In the HTTP plugin section of the fluent-bit configuration file, update or add the header field using the new AUTH_TOKEN value (from the previous step):

[OUTPUT]
       Name http
       Match <match_pattern>
       Host <kfuse_ingress_ip>
       Port 443
       TLS on
       URI /ingester/v1/fluent_bit
       header Kf-Api-Key <AUTH_TOKEN>

Use the Kubernetes Secret to set the Kf-Api-Key.

Create the secret:

apiVersion: v1
kind: Secret
metadata:
  name: <<secret-name>>
type: Opaque
data:
  AUTH_KEY_ENV: AUTH_TOKEN_ENCODED

Update the fluent-bit helm using the `custom-values.yaml` file:

env:
- name: AUTH_KEY_ENV
valueFrom:
      secretKeyRef:
        name: <<secret-name>>
        key: AUTH_KEY_ENV

config:
outputs: |
    [OUTPUT]
        Name http
        Header Kf-Api-Key ${AUTH_KEY_ENV}
        Match *
        Host dev.kloudfuse.io
        Port 443
        TLS on
        URI /ingester/v1/fluent_bit
        Format {json|json_lines|json_stream|msgpack}

Fluentd

In the HTTP plugin section of the Fluentd configuration file, update or add the headers field using the new Kf_Api-Key and AUTH_TOKEN value (from the previous step):
```
<match *> # Match everything
@type http
endpoint http://<KFUSE_INGESTER_IP>:80/ingester/v1/fluentd
headers {"Kf-Api-Key" : "<AUTH_TOKEN>"}
...
</match>
```

Filebeat

In the output section, include the api_key field to update or add a filebeat configuration.

output.elasticsearch:
hosts: ["http://<ingress-ip>:80/ingester/api/v1/filebeat"]
api_key: "<AUTH_TOKEN>"

OLTP Collector for Metrics, Logs, and Traces

In the exporters section, include the api_key field using AUTH_TOKEN value to update or add an OLTP configuration.

exporters:
otlphttp:
    endpoint: https://<ingress-address>/ingester/otlp/metrics
    traces_endpoint: https://<ingress-address>/ingester/otlp/traces
    headers:
      kf-api-key: <AUTH_TOKEN>

DataDog agent

Include the api_key field using AUTH_TOKEN value to update or add a DataDog configuration.

datadog:
apiKey: <AUTH_TOKEN>

AWS CloudWatch Metrics and Logs (Kinesis)

When configuring kinesis firehose data stream to send logs or metrics from Cloudwatch, use the AUTH_TOKEN value as the access token.

AWS Eventbridge Events

When configuring Eventbridge to ingest to Kloudfuse, use AUTH_TOKEN as the value for the Kf-Api-Key.

Heroku Logs

For Heroku logs authentication, follow the steps in Configure kfuse and apply the encoded auth token to the kfuse-auth-ingest secret.

Configure kfuse

Use the base64-encoded value of the AUTH_TOKEN, AUTH_TOKEN_ENCODED, to create a kubernetes secret kfuse-auth-ingest:

apiVersion: v1
kind: Secret
metadata:
name: kfuse-auth-ingest
type: Opaque
data:
authToken: <AUTH_TOKEN_ENCODED>

Multiple Authorization Keys

You can configure multiple authorization tokens inside the secret. These tokens can contain any string value, and become a human-readable identifier to reference the AUTH_TOKEN.

Example:

apiVersion: v1
kind: Secret
metadata:
name: kfuse-auth-ingest
type: Opaque
data:
authkey1: <AUTH_TOKEN_ENCODED>
authkey2: <AUTH_TOKEN_ENCODED2>

Update the `custom-values.yaml` file to enable ingestion authentication:

global:
authConfig:
    enabled: true

Additional Labels Based on AUTH_TOKEN

You can configure additional labels to attach to ingested time data, and base these labels on the auth token of the incoming payload. To configure additional labels, add the following settings to the custom-values.yaml file. Note that Kloudfuse uses the same keys in the secret and in the ingester configuration:

ingester:
config:
     authKeyAdditionalLabels:
        authkey1:
          - name: label1
            value: val1
          - name: label2
            value: val2
        authkey2:
          - name: label1
            value: val3
          - name: label4
            value: val4

Restart the Ingester

After saving the configuration, restart the ingester service:

kubectl rollout restart statefulset ingester