Cloud-Provider Managed Services

Overview

By default, Kloudfuse bundles PostgreSQL, Redis, and Kafka within the Helm chart. For production deployments — particularly those requiring compliance, high availability, or multi-zone availability — you can replace these with cloud-provider managed equivalents:

Common reasons to use managed services:

• Compliance — Databases and message brokers must reside on managed, audited infrastructure.
• Corporate policy — Centralized operations teams manage data services separately.
• Multi-zone availability — Services must be reachable across availability zones or regions.
• Operational simplicity — Offload backup, patching, and scaling to the cloud provider.

Managed Database (PostgreSQL)

Kloudfuse uses PostgreSQL for storing configuration, alert rules, RBAC policies, and other platform state across multiple databases (configdb, alertsdb, beffedb, logsconfigdb, and others).

The bundled deployment runs a kfuse-configdb StatefulSet inside the cluster. When switching to a managed PostgreSQL service, you point Kloudfuse at the external endpoint and disable the bundled StatefulSet.

For full details, see Cloud Provider Backend Database.

Requirements

• PostgreSQL version 14 or later (version 18+ required for FIPS environments)
• Minimum 2 cores, 2 GB memory, 50 GB disk
• Port 5432 (the chart uses port 5432; custom ports are not currently supported)
• A database user with CREATEDB privilege (the user creates and owns all Kloudfuse databases)
• For FIPS/compliance environments, use a non-default username (not postgres) — see FIPS Installation Guide
• Network accessible from the Kubernetes cluster

Cloud Providers

Credentials

Create a Kubernetes secret containing the database password:

export PG_CREDENTIAL="YOUR_PASSWORD"
kubectl create secret generic kfuse-pg-credentials \
  --from-literal=postgres-password="$PG_CREDENTIAL" \
  --from-literal=postgresql-password="$PG_CREDENTIAL" \
  --from-literal=postgresql-replication-password="$PG_CREDENTIAL"
unset PG_CREDENTIAL

Helm Configuration

Update your custom-values.yaml with the following parameters:

installKfusePgCredentials: false  (1)

global:
  configDB:
    host: "your-rds-endpoint.rds.amazonaws.com"  (2)
    username: "<your-db-username>"  (3)
  orchestratorDB:
    host: "your-rds-endpoint.rds.amazonaws.com"  (2)
    username: "<your-db-username>"  (3)

kfuse-configdb:
  enabled: false  (4)

ingester:
  postgresql:
    enabled: false  (5)

TLS Configuration

To enable SSL/TLS for PostgreSQL connections, create a Kubernetes secret containing the CA certificate used to verify the server’s identity.

For AWS RDS, download the global CA bundle:

curl -o rds-ca-bundle.pem https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem
kubectl create secret generic pg-tls-ca-cert \
  --from-file=ca.crt=rds-ca-bundle.pem

For Google Cloud SQL or Azure Database for PostgreSQL, download the CA certificate from the cloud console and substitute it above.

Then update your custom-values.yaml to enable TLS:

global:
  configDB:
    tls:
      enabled: true
      sslmode: "verify-full"   (1)
      existingSecret: "pg-tls-ca-cert"  (2)
      certSecretKey: "ca.crt"           (3)
  orchestratorDB:
    tls:
      enabled: true
      sslmode: "verify-full"
      existingSecret: "pg-tls-ca-cert"
      certSecretKey: "ca.crt"

Create Databases

Kloudfuse requires the following databases to be created on your managed PostgreSQL instance before deployment (as of version 3.5.3):

The Helm chart automatically creates all required databases during installation via an init job. The databases are created with OWNER set to the configured username, so the application user has full privileges within each database.

Verify

After applying the Helm changes:

• The kfuse-configdb pod should not be running.
• The ingester and az-service pods should be in a Running state.
• Check pod logs for database connection errors:

kubectl logs -l app=ingester --tail=50 | grep -i "postgres\|database\|connection"
kubectl logs -l app=az-service --tail=50 | grep -i "postgres\|database\|connection"

Managed Redis

Kloudfuse uses Redis for caching and session state across multiple services (ingester, query-service, logs-query-service, logs-parser, kfuse-auth, and others).

The bundled Redis deployment uses Sentinel for high availability (clients connect on port 26379). When switching to a managed Redis service, clients connect directly to the managed endpoint on port 6379 — no Sentinel is involved.

Requirements

• Redis 6.0 or later (Redis 7.x recommended for AWS ElastiCache and GCP Memorystore)
• Non-clustered (single-node or replication) mode recommended for simplicity
• Network accessible from the Kubernetes cluster (see note above about VPC-private endpoints)
• Size the instance based on your workload — a starting point is 2 GB of memory for small deployments

Cloud Providers

Credentials

If your managed Redis instance requires authentication, create a Kubernetes secret containing the Redis password:

kubectl create secret generic kfuse-redis-credentials \
  --from-literal=redis-password="YOUR_REDIS_PASSWORD"

Replace YOUR_REDIS_PASSWORD with the actual password for your managed Redis instance.

Helm Configuration

Update your custom-values.yaml with the following parameters:

global:
  redis:
    enabled: false  (1)
    external:
      enabled: true  (2)
      host: "your-redis-endpoint.cache.amazonaws.com"  (3)
      port: 6379  (4)

When global.redis.external.enabled is true, all Kloudfuse services connect directly to the external endpoint instead of using Sentinel.

With Authentication

If your managed Redis requires a password, add the auth configuration:

global:
  redis:
    enabled: false
    external:
      enabled: true
      host: "your-redis-endpoint.cache.amazonaws.com"
      port: 6379
      auth:
        enabled: true  (1)
        existingSecret: "kfuse-redis-credentials"  (2)

With TLS (In-Transit Encryption)

If your managed Redis has in-transit encryption enabled (required for AWS ElastiCache with TransitEncryptionEnabled: true), add the TLS configuration:

global:
  redis:
    enabled: false
    external:
      enabled: true
      host: "your-redis-endpoint.cache.amazonaws.com"
      port: 6379
      auth:
        enabled: true
        existingSecret: "kfuse-redis-credentials"
      tls:
        enabled: true  (1)
        skipVerify: false  (2)

Verify

After applying the Helm changes:

• Confirm the bundled kfuse-redis pods are not running:

kubectl get pods | grep kfuse-redis

• Verify that services can connect to the managed Redis endpoint by checking logs for connection errors:

kubectl logs -l app=ingester --tail=50 | grep -i redis
kubectl logs -l app=kfuse-auth --tail=50 | grep -i redis

Managed Kafka

Kloudfuse uses Kafka as the central streaming backbone. Approximately 13 services connect to Kafka for ingestion, transformation, querying, and storage.

The bundled Kafka deployment uses KRaft mode (no ZooKeeper) with plaintext connections on port 9092. Managed Kafka services typically require SASL authentication and TLS encryption.

Requirements

• Apache Kafka 2.8 or later (KRaft-compatible)
• Minimum 3 brokers for production
• SASL/SCRAM-SHA-512 authentication (recommended for AWS MSK)
• TLS encryption enabled
• Auto-topic creation disabled (Kloudfuse manages its own topics)
• Network accessible from the Kubernetes cluster

Cloud Providers

Credentials

Create a Kubernetes secret containing the SASL password:

kubectl create secret generic kfuse-kafka-sasl-credentials \
  --from-literal=kafka-sasl-password="YOUR_SASL_PASSWORD" \
  --from-literal=password="YOUR_SASL_PASSWORD"

Replace YOUR_SASL_PASSWORD with the actual password for your Kafka SASL user. Both keys are required — kafka-sasl-password is used by the init containers and password is used by the Pinot stream configs.

Helm Configuration

Update your custom-values.yaml with the following parameters:

global:
  kafka:
    deployLegacy: false  (1)
    deployKraft: false  (2)
    bootstrapServers: "broker-1:9096,broker-2:9096,broker-3:9096"  (3)
    sasl:
      enabled: true  (4)
      mechanism: "SCRAM-SHA-512"  (5)
      username: "your-kafka-username"  (6)
      existingSecret: "kfuse-kafka-sasl-credentials"  (7)
      passwordKey: "kafka-sasl-password"  (8)
    tls:
      enabled: true  (9)
      skipVerify: false  (10)

AWS MSK Example

For an AWS MSK cluster with 3 brokers in us-west-2:

global:
  kafka:
    deployLegacy: false
    deployKraft: false
    bootstrapServers: "b-1.mycluster.abc123.c14.kafka.us-west-2.amazonaws.com:9096,b-2.mycluster.abc123.c14.kafka.us-west-2.amazonaws.com:9096,b-3.mycluster.abc123.c14.kafka.us-west-2.amazonaws.com:9096"
    sasl:
      enabled: true
      mechanism: "SCRAM-SHA-512"
      username: "kfuse-admin"
      existingSecret: "kfuse-kafka-sasl-credentials"
      passwordKey: "kafka-sasl-password"
    tls:
      enabled: true

To find your MSK SASL bootstrap brokers:

# Private VPC endpoint (port 9096) — use when Kloudfuse is in the same VPC or connected via peering:
aws kafka get-bootstrap-brokers --cluster-arn <YOUR_MSK_CLUSTER_ARN> \
  --query 'BootstrapBrokerStringSaslScram' --output text

# Public endpoint (port 9196) — only if public access is enabled on the MSK cluster:
aws kafka get-bootstrap-brokers --cluster-arn <YOUR_MSK_CLUSTER_ARN> \
  --query 'BootstrapBrokerStringPublicSaslScram' --output text

Topic Creation

Kloudfuse requires a set of Kafka topics for each data stream (metrics, logs, traces, events, etc.). The required topics are defined in global.kafkaTopics in the Helm chart values.

When using managed Kafka, you have two options:

1. Pre-create topics manually — Use kafka-topics.sh or your cloud provider’s console/CLI to create the required topics before deploying Kloudfuse. This is the recommended approach when your managed Kafka restricts topic creation via ACLs.

First, create a client.properties file with your SASL/TLS credentials:

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
     username="your-kafka-username" \
     password="your-kafka-password";

Then create topics:

kafka-topics.sh --bootstrap-server <YOUR_BOOTSTRAP_SERVERS> \
     --command-config /tmp/client.properties \
     --create --topic kf_metrics_topic --partitions 1 --replication-factor 3

2. Automatic topic creation — The Helm chart includes a topic creator Job that attempts to create all required topics at install/upgrade time. Ensure the SASL user has the CREATE ACL permission on the cluster for this to succeed.

Verify

After applying the Helm changes:

• The kafka-kraft and kafka-broker pods should not be running.
• If not in FIPS mode, the topic creator job should complete successfully:

kubectl get jobs | grep kafka-external-topic
kubectl logs job/kfuse-kafka-external-topic-creation

• Verify services can connect to the external Kafka:

kubectl logs -l app=ingester --tail=50 | grep -i "kafka\|sasl\|bootstrap"
kubectl logs -l app=logs-transformer --tail=50 | grep -i "kafka\|sasl\|bootstrap"

• Check that topics were created (using the client.properties file):

kafka-topics.sh --bootstrap-server <YOUR_BOOTSTRAP_SERVERS> \
  --command-config /tmp/client.properties --list

Combined Configuration Example

To use managed PostgreSQL, Redis, and Kafka together (for example, in a multi-zone deployment):

installKfusePgCredentials: false

global:
  # --- Managed PostgreSQL ---
  configDB:
    host: "your-rds-endpoint.rds.amazonaws.com"
    username: "<your-db-username>"
  orchestratorDB:
    host: "your-rds-endpoint.rds.amazonaws.com"
    username: "<your-db-username>"
  # --- Managed Redis ---
  redis:
    enabled: false
    external:
      enabled: true
      host: "your-redis.cache.amazonaws.com"
      port: 6379
      auth:
        enabled: true
        existingSecret: "kfuse-redis-credentials"
  # --- Managed Kafka ---
  kafka:
    deployLegacy: false
    deployKraft: false
    bootstrapServers: "<YOUR_MSK_BOOTSTRAP_SERVERS>"  (1)
    sasl:
      enabled: true
      mechanism: "SCRAM-SHA-512"
      username: "kfuse-admin"
      existingSecret: "kfuse-kafka-sasl-credentials"
      passwordKey: "kafka-sasl-password"
    tls:
      enabled: true

kfuse-configdb:
  enabled: false

ingester:
  postgresql:
    enabled: false

Troubleshooting

Redis

Kafka

General

If services fail to start after switching to managed services, check pod logs for connection errors:

kubectl get pods --field-selector=status.phase!=Running
kubectl describe pod <POD_NAME>
kubectl logs <POD_NAME> --tail=100

Ensure that network connectivity allows the Kubernetes cluster to reach the managed service endpoints.