# Ingestion Authentication with API Key

## Configuration Methods

Starting with Kloudfuse 4.1, you can manage API keys and their optional additional labels directly from the Kloudfuse UI. Open **Admin > Settings**, then click **Configure** on the **Auth key labels** card. The UI is the recommended way to manage ingestion API keys going forward.

YAML-based configuration (Kubernetes secret `kfuse-auth-ingest` plus `ingester.config.authKeyAdditionalLabels` in `custom_values.yaml`) continues to work but is **deprecated**. YAML-sourced entries appear in the UI as read-only, badged with `YAML`. Edit, delete, and token-management actions are disabled for these rows.

|     |     |
| --- | --- |
|  | **UI-managed entries always take precedence over YAML.** The ingester resolves auth keys against the UI snapshot first and falls back to YAML only when no UI entry matches. When a UI-managed entry and a YAML entry share the same display name, the UI-managed entry wins — the shadowed YAML entry is hidden from the list view, and the surviving UI row is tagged with an info icon noting that the YAML labels for that display name are not effective. |

See [Migrate YAML-managed Auth Keys to the UI](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#migrate-yaml-to-ui) for migration steps.

## Configure in the UI (Recommended, 4.1+)

01. Sign in to Kloudfuse as an admin. From the left sidebar, open **Admin > Settings**, then click **Configure** on the **Auth key labels** card.

02. Click **Create** to open the **Create auth key label** form.

03. Enter a **Display name**. This is the human-readable identifier used to look up the key (equivalent to the secret key in the YAML approach, e.g. `authkey1`).

04. Set the **Token** (required):

- **Auto-generate** (default) produces a token in the form `kf_<64hex>`.
    - **Paste your own** lets you supply an existing `AUTH_TOKEN` value — useful when migrating from YAML so you can reuse the token your agents already send. To generate a random token to paste in, see [Generate the AUTH
_TOKEN](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#auth-token).

|     |     |
      | --- | --- |
      |  | Copy the token immediately. Kloudfuse stores only a SHA-256 hash of it, so the plaintext value is never shown again after the entry is created. |

05. Optionally set an **Expires at** date. After that timestamp the token is rejected at lookup time. Leave blank for no expiry.

06. Optionally add one or more **Labels** (`name` / `value` pairs). Labels are attached to ingested data authenticated with this key — the UI equivalent of `ingester.config.authKeyAdditionalLabels`. Label names must be Prometheus-compatible.

07. Optionally check **Enable rate control** to auto-create rate-control classes for this auth key. Select one or more signal streams (METRICS, LOGS, EVENTS, TRACES, RUM) — Kloudfuse creates a class per stream, each with a quota of `0`. Set the actual quotas on the **Rate control** page after creation. See [Manage Ingestion Rate Control](https://docs.kloudfuse.com/platform/4.1.0/data-management/routing/rate-control/) for details.

08. Click **Create**. The new entry is now active — `config-mgmt-service` polls for changes and the ingester atomically swaps in the new snapshot, with no restart required.

09. To add additional tokens later (for rotation) or to manage existing tokens, open the entry and switch to the **Tokens** tab.

10. Use the token in the `Kf-Api-Key` (or `authorization`) header of the agent or source you are configuring — see [Configure Telemetry Agents and Sources](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#agents).

## Configure in YAML (Deprecated)

|     |     |
| --- | --- |
|  | The YAML-based configuration described below is deprecated as of 4.1. New deployments should use the UI. Existing deployments should follow [the migration steps](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#migrate-yaml-to-ui) when convenient. |

The full YAML flow is:

1. [Generate the AUTH
token](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#auth-token)
2. [Create the `kfuse-auth-ingest` secret](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#kfuse)
3. (Optional) [Configure multiple authorization keys](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#multiple-auth-keys)
4. (Optional) [Configure additional labels based on the auth token](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#additional-tables)
5. [Restart the ingester](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#restart)

### Generate the AUTH_TOKEN

1. Generate an `AUTH_TOKEN`, and store its value in a safe location. You use it repeatedly.

```console
    AUTH_TOKEN=`cat /dev/urandom | env LC_ALL=C tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1`
    ```

2. Encode the `AUTH_TOKEN` using Base64; this generates the `AUTH_TOKEN_ENCODED`.

```console
    AUTH_TOKEN_ENCODED=`echo -n $AUTH_TOKEN | base64`
    ```

### Create the kfuse-auth-ingest Secret

Use the base64-encoded value of the `AUTH_TOKEN`, `AUTH_TOKEN_ENCODED`, to create a kubernetes secret `kfuse-auth-ingest`:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: kfuse-auth-ingest
type: Opaque
data:
  authToken: <AUTH_TOKEN_ENCODED>
```

### Multiple Authorization Keys

You can configure multiple authorization tokens inside the secret. These tokens can contain any string value, and become a human-readable identifier to reference the AUTH_TOKEN.

Note that in the following example, we have both `authkey1` and `authkey2`:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: kfuse-auth-ingest
type: Opaque
data:
  authkey1: <AUTH_TOKEN_ENCODED>
  authkey2: <AUTH_TOKEN_ENCODED2>
```

Update the `custom_values.yaml` file to enable ingestion authentication:

```yaml
global:
  authConfig:
    enabled: true
```

### Additional Labels Based on AUTH_TOKEN

You can configure additional labels to attach to ingested time data, and base these labels on the auth token of the incoming payload.

To configure additional labels, add the following settings to the `custom_values.yaml` file. Note that Kloudfuse uses the same keys in the secret and in the ingester configuration:

```yaml
ingester:
  config:
     authKeyAdditionalLabels:
        authkey1:
          - name: label1
            value: val1
          - name: label2
            value: val2
        authkey2:
          - name: label1
            value: val3
          - name: label4
            value: val4
```

### Restart the Ingester

After saving the YAML configuration, restart the ingester service to pick up the changes:

```console
kubectl rollout restart statefulset ingester
```

## Migrate YAML-managed Auth Keys to the UI

If you previously configured ingestion auth keys through the `kfuse-auth-ingest` Kubernetes secret and `ingester.config.authKeyAdditionalLabels` in `custom_values.yaml`, you can migrate to the UI after upgrading to 4.1. The migration is non-disruptive: YAML-sourced entries continue to authenticate ingest traffic until you remove them.

The migration relies on the precedence rule: as soon as you create a UI entry with the same display name as a YAML entry, the UI definition takes effect and the YAML entry is no longer used. This lets you cut over one key at a time without coordinating a YAML cleanup.

|     |     |
| --- | --- |
|  | If you have more than a handful of YAML entries, skip the per-entry manual recreation in step 2 and use the [migration helper script](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#migrate-script) instead. It reads the secret and ConfigMap directly and creates matching UI entries in one pass, reusing the existing tokens so no agent reconfiguration is needed. You still complete steps 1, 3, 4, and 5 manually. |

High-level migration steps:

1. **Upgrade to 4.1 or later.** In the UI, open **Admin > Settings** and click **Configure** on the **Auth key labels** card. YAML-sourced entries appear in the list with a `YAML` badge and are read-only.

2. **Recreate each YAML entry in the UI.** You can do this manually (one-by-one) or use the [migration helper script](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#migrate-script) to automate it.

For the manual approach, for each YAML entry:

1. Use the same display name as the YAML key (e.g. `authkey1`). When the display names match, the UI-managed entry shadows the YAML entry — Kloudfuse uses the UI definition for new traffic.
2. Reuse the existing token value (paste it into the **Tokens** tab as a bring-your-own token) so agents that already authenticate against the YAML key continue to work without rotation.
3. Recreate any labels (`name` / `value` pairs) that were defined under `authKeyAdditionalLabels` for that key.

3. **Verify ingestion still succeeds** and that the labels you expect are attached to the ingested data.

4. **Delete the YAML configuration.** Once every YAML entry has a corresponding UI entry and traffic is healthy:

1. Delete the Kubernetes secret:

```console
        kubectl delete secret kfuse-auth-ingest -n <namespace>
        ```

2. Remove the `ingester.config.authKeyAdditionalLabels` block (and the entire `kfuse-auth-ingest` secret stanza, if you manage it via Helm) from `custom_values.yaml`, then re-apply Helm values.

5. **Restart `config-mgmt-service`** so it stops surfacing the now-removed YAML entries:

```console
    kubectl rollout restart deploy/config-mgmt-service -n <namespace>
    ```

### Migration Helper Script

A bash helper script that automates step 2 (creating UI entries from the YAML data) is available as [migrate-yaml-auth-keys.sh](https://docs.kloudfuse.com/platform/4.1.0/_attachments/migrate-yaml-auth-keys.sh).

The script:
- Reads tokens from the auth secret (default `kfuse-auth-ingest`) by base64-decoding each `data.<displayName>` entry.
- Reads `authKeyAdditionalLabels` from the ingester ConfigMap (default `ingester`, key `config.yaml`), lowercasing display names to match the ingester’s Viper-loaded semantics.
- Joins each token with its matching label entry by display name, then calls `createAuthKeyLabel` against `config-mgmt-service` to create a UI-managed entry that **reuses the existing token** — agents authenticated against the YAML key keep working without rotation.
- Skips display names that already have a UI-managed entry, so re-runs are idempotent.

|     |     |
| --- | --- |
|  | The script **only** creates UI entries — it does **not** delete the Kubernetes secret or modify `custom_values.yaml`. Running it leaves your existing YAML config in place; the new UI entries simply **shadow** the YAML ones via the precedence rule. You must complete the cleanup manually ( [step 4](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#migrate-cleanup) and [step 5](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#migrate-restart) above) after verifying ingestion is healthy. Until you do, the YAML data remains visible in the UI as read-only `YAML`-badged rows. |

Prerequisites: `kubectl`, `jq`, `python3` (with PyYAML), and a port-forward to config-mgmt-service:

```console
kubectl port-forward -n <namespace> svc/config-mgmt-service 8080:8080
```

Usage (defaults to namespace `kfuse`):

```console
./migrate-yaml-auth-keys.sh [--dry-run]
```

Pass `-n <namespace>` if your Kloudfuse install uses a different namespace. Run with `--dry-run` first to see exactly what would be created, then re-run without it to perform the migration. See `--help` for all options (custom secret/configmap names, port, admin email).

After the script completes successfully:

1. [Verify ingestion is healthy](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#migrate-verify) (step 3 above).
2. [Delete the Kubernetes secret and remove `authKeyAdditionalLabels`](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#migrate-cleanup) from `custom_values.yaml` (step 4 above).
3. [Restart `config-mgmt-service`](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#migrate-restart) so it stops surfacing the now-removed YAML entries (step 5 above).

## Configure Telemetry Agents and Sources

Once you have a token (from either the UI or YAML approach), configure your telemetry agents and sources to send it as the API key.

### Prometheus Remote Write

1. Update the prometheus remote write configuration:

```yaml
    prometheus.yml:
         remote_write:
      - url: https://<customer>.kloudfuse.io/ingester/write
        authorization:
          credentials: <AUTH_TOKEN>
    ```

2. When using the Prometheus operator, use the following configuration that contains the `AuthToken`:

```yaml
    remoteWrite:
      - authorization:
          credentials:
            key: <AUTH_TOKEN> (1)
            name: kf-auth-ingest (2)
        url: https://<customer>.kloudfuse.io/ingester/write
    ```

|     |     |
| --- | --- |
| **1** | `<AUTH_TOKEN>`: The key inside this secret that contains the encoded `authToken`. |
| **2** | `kf-auth-ingest`: The secret name. |

### Fluent Bit

1. In the HTTP plugin section of the fluent-bit configuration file, update or add the `header` field using the new `AUTH_TOKEN` value (from the previous step):

```yaml
    [OUTPUT]
           Name http
           Match <match_pattern>
           Host <kfuse_ingress_ip>
           Port 443
           TLS on
           URI /ingester/v1/fluent_bit
           header Kf-Api-Key <AUTH_TOKEN>
    ```

2. Use the Kubernetes Secret to set the `Kf-Api-Key`.

1. Create the secret:

```yaml
        apiVersion: v1
        kind: Secret
        metadata:
          name: <<secret-name>>
        type: Opaque
        data:
          AUTH_KEY_ENV: AUTH_TOKEN_ENCODED
        ```

2. Update the fluent-bit helm using the `custom_values.yaml` file:

```yaml
        env:
    - name: AUTH_KEY_ENV
    valueFrom:
          secretKeyRef:
            name: <<secret-name>>
            key: AUTH_KEY_ENV

config:
    outputs: |
        [OUTPUT]
            Name http
            Header Kf-Api-Key ${AUTH_KEY_ENV}
            Match *
            Host dev.kloudfuse.io
            Port 443
            TLS on
            URI /ingester/v1/fluent_bit
            Format {json|json_lines|json_stream|msgpack}
        ```

### Fluentd

1. In the HTTP plugin section of the Fluentd configuration file, update or add the `headers` field using the new `Kf_Api-Key` and `AUTH_TOKEN` value (from the previous step):

```yaml
    <match *> # Match everything
    @type http
    endpoint http://<KFUSE_INGESTER_IP>:80/ingester/v1/fluentd
    headers {"Kf-Api-Key" : "<AUTH_TOKEN>"}
    ...
    </match>
    ```

### Filebeat

In the output section, include the `api_key` field to update or add a filebeat configuration.

```yaml
output.elasticsearch:
hosts: ["http://<ingress-ip>:80/ingester/api/v1/filebeat"]
a_api_key: "<AUTH_TOKEN>"
```

### OLTP Collector for Metrics, Logs, and Traces

In the exporters section, include the `api_key` field using `AUTH_TOKEN` value to update or add an OLTP configuration.

```yaml
exporters:
otlphttp:
    endpoint: https://<ingress-address>/ingester/otlp/metrics
    traces_endpoint: https://<ingress-address>/ingester/otlp/traces
    headers:
      kf-api-key: <AUTH_TOKEN>
```

### DataDog agent

Include the `api_key` field using `AUTH_TOKEN` value to update or add a DataDog configuration.

```yaml
datadog:
apiKey: <AUTH_TOKEN>
```

### AWS CloudWatch Metrics and Logs (Kinesis)

When configuring kinesis firehose data stream to send logs or metrics from Cloudwatch, use the `AUTH_TOKEN` value as the access token.

### AWS Eventbridge Events

When [AWS EventBridge Integration with Kloudfuse](https://docs.kloudfuse.com/platform/4.1.0/data-collection/cloud-services/aws/eventbridge/) to ingest to Kloudfuse, use `AUTH_TOKEN` as the value for the `Kf-Api-Key`.

### Heroku Logs

For Heroku logs authentication, create the auth key entry in the UI ( [Configure in the UI](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#kfuse-ui)) or via YAML ( [Create the kfuse-auth-ingest secret](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/#kfuse)).

Once the auth key is created, add the token as the `kf-api-key` parameter in the drain URL as described in [Heroku Log Ingestion](https://docs.kloudfuse.com/platform/4.1.0/data-collection/agents/other/agent-heroku/).