Upgrade Instructions :: Kloudfuse Docs

Upgrade Instructions

Follow these step-by-step instructions to upgrade Kloudfuse using Helm, including the upgrade command and version-specific notes for all supported releases.

Helm upgrade command

  1. Before performing an upgrade, validate that the upgrade won’t revert any customization on your cluster.

  2. To check which Kloudfuse version you have, run the following command:

    helm list

  3. Run the upgrade command.

    helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
     -n kfuse \
     --version <VERSION> \
     -f custom-values.yaml
    1 Replace <VERSION> with a valid Kloudfuse release value. See Release Notes for the latest release.

Version Specific Instructions

Upgrade to 4.2.2

Pre-Upgrade Steps

GCP Logs Pub/Sub Subscriptions Now a List (Breaking Change)

Release 4.2.2 configures GCP Pub/Sub log collection with a subscriptions list under global.gcpConfig.pubsub, replacing the previous single subscriptionId field. Update the pubsub section of your custom-values.yaml file to the new list format before upgrading:

global:
  gcpConfig:
    secretName: "kfuse-gcp-credentials"
    projectId: "<your-gcp-project-id>"
    pubsub:
      enabled: true
      subscriptions: 
        - projectId: "<your-gcp-project-id>"
          subscriptionId: "kloudfuse-gcp-subscription"

Post-Upgrade Steps

Review Saved Queries That Use parse

Release 4.2.2 rejects an anchored parse pattern whose alias count does not match the number of extractions. Queries of this form were previously accepted and silently dropped extractions or left aliases empty. Saved searches, scheduled searches, and alerts using these patterns fail after upgrading and must be corrected.

Update MCP Clients That Call search_logs

Release 4.2.2 makes FuseQL query_logs the default logs tool in the MCP server. The legacy LogQL tools search_logs and query_log_metrics are no longer on the default tool surface. Clients that call them must either move to query_logs or request the legacy toolset explicitly with ?toolsets=logql.

Update MCP Clients That Decode Prometheus Responses

Release 4.2.2 changes the response shape of the query_prometheus, search_prometheus_label_names, and search_prometheus_label_values tools. These tools now return parsed objects rather than JSON-encoded strings. Clients that decoded the previous string response must be updated. Other tool families are unchanged.

Upgrade to 4.2.1

Pre-Upgrade Steps

Add kf_metrics_staging_topic Kafka Topic

Release 4.2.1 requires a new Kafka topic. Before upgrading, add the following entry to global.kafkaTopics in your custom_values.yaml:

global:
  kafkaTopics:
    - name: kf_metrics_staging_topic
      partitions: 1
      replicationFactor: 1

Upgrade to 4.2.0

Pre-Upgrade Steps

Metric Shaping Kafka Topic

Release 4.2.0 introduces the Metric Shaping Rules feature, which requires the kf_metrics_staging_topic Kafka topic. Before upgrading, ensure that this topic is included in your global.kafkaTopics section in the custom-values.yaml file:

        - name: kf_metrics_staging_topic
          partitions: <number-of-nodes>
          replicationFactor: 1

Post-Upgrade Steps

Query Scheduling Enabled by Default

Query scheduling is enabled by default in Release 4.2.0. Administrators can configure per-user, per-group, and per-service-account query priorities — including the new blocked priority, which rejects matching queries instead of delaying or queueing them — from the admin UI. To disable query scheduling entirely, set the following in your custom_values.yaml:

global:
  queryScheduler:
    enabled: false

Archive Writer Disabled by Default

The new archive writer is disabled by default. It can be turned on per deployment when you are ready — contact Kloudfuse support to opt in. Enabling it requires the kf_logs_archive_v2_topic Kafka topic; see Archive Writer Kafka Topic (Conditional) in the pre-upgrade steps.

Scheduled Searches Disabled by Default

Scheduled searches are disabled by default and can be enabled on request — contact Kloudfuse support to enable the feature for your tenant.

Upgrade to 4.1.0

Pre-Upgrade Steps

Set events-query-service.config.rbacdb.user (Externally-Managed Postgres Only)

If your configDB is hosted on an externally-managed Postgres instance (AWS RDS, GCP Cloud SQL, Azure Database for PostgreSQL) where the application user is not the default postgres, you must add an explicit events-query-service.config.rbacdb.user override to your custom_values.yaml. The events-query-service chart currently ships with a hardcoded rbacdb.user: postgres default that masks the global.configDB.username fallback, so without this override events-query-service connects to rbacdb as postgres and fails with no pg_hba.conf entry for host …​, user "postgres", database "rbacdb". Skip this step if global.configDB.username is postgres (the default).

events-query-service:
  config:
    rbacdb:
      user: <your-configdb-app-username>  # match global.configDB.username

Post-Upgrade Steps

Migrate Ingestion Auth Keys to the UI (Optional, Recommended)

Starting in 4.1.0, ingestion API keys and their optional additional labels can be managed from the Kloudfuse UI. Open Admin > Settings, then click Configure on the Auth key labels card. The legacy YAML configuration (kfuse-auth-ingest secret + ingester.config.authKeyAdditionalLabels) continues to work but is deprecated. YAML-sourced entries appear in the UI as read-only.

After upgrading, recreate each YAML-managed entry through the UI (reusing the existing token to avoid agent reconfiguration), then delete the legacy YAML configuration:

kubectl delete secret kfuse-auth-ingest -n <namespace>

Upgrade to 4.0.2

Pre-Upgrade Steps

No special pre-upgrade steps are required for this release.

4.0.1

Pre-Upgrade Steps

TLS Certificate Key Type Check (Required for Envoy Gateway deployments)

Release 4.0.1 upgrades the Envoy proxy to v1.36 with FIPS-compliant cryptography. The FIPS proxy rejects RSA private keys for TLS certificates delivered via SDS. All TLS secrets should be checked and any RSA certificates should be recreated as ECDSA.

  1. Scan all TLS secrets for RSA keys:
for secret in $(kubectl get secrets -n <namespace> --field-selector type=kubernetes.io/tls -o jsonpath='{.items[*].metadata.name}'); do
     KEY_TYPE=$(kubectl get secret $secret -n <namespace> -o jsonpath='{.data.tls\.key}' | base64 -d | openssl pkey -noout -text 2>/dev/null | head -1)
     if echo "$KEY_TYPE" | grep -q "2048\|4096\|3072"; then
       echo "RSA: $secret — needs recreation"
     else
       echo "ECDSA: $secret — OK"
     fi
done
  1. Delete internal envoy certificates (certgen certs):

These are regenerated automatically by the certgen hook during the helm upgrade. The 4.0.1 certgen generates ECDSA P-256 keys.

kubectl delete secret envoy envoy-gateway envoy-rate-limit -n <namespace>
  1. Proceed with the helm upgrade to 4.0.1.

Post-Upgrade Steps

TLS Listener Certificate Reissue (Required for cert-manager deployments)

After the upgrade, the Gateway resource has ECDSA annotations (cert-manager.io/private-key-algorithm: ECDSA). If your listener TLS certificate was RSA (identified in the pre-upgrade check), it must be deleted and reissued as ECDSA.

  1. Verify the Gateway has ECDSA annotations:
kubectl get gateway kfuse -n <namespace> -o yaml | grep -i 'private-key'
  1. Delete the listener TLS certificate:
kubectl delete secret <tls-secret-name> -n <namespace>
  1. Verify the new certificate is ECDSA:
kubectl get secret <tls-secret-name> -n <namespace> \
     -o jsonpath='{.data.tls\.key}' | base64 -d | \
     openssl pkey -noout -text 2>/dev/null | head -1
  1. Restart the Envoy proxy to load the new certificate:
kubectl rollout restart deploy/envoy -n <namespace>
  1. Verify HTTPS is working:
curl -sk https://<your-hostname> -o /dev/null -w "%{http_code}"

Why is this needed?

The Envoy proxy FIPS build uses SafeLogic CryptoComply for cryptographic operations. When a TLS certificate is delivered to the proxy via SDS (Secret Discovery Service), the private key is imported into the FIPS cryptographic module. SafeLogic’s implementation runs a Pairwise Consistency Test (PCT) on RSA key import, which fails. ECDSA keys are not affected.

3.5.3-p1

There are no specific pre-upgrade or post-upgrade steps for upgrading to Release 3.5.3-p1.

3.5.3

Pre-Upgrade Steps

Metrics Transformer GOMEMLIMIT

Multi-resolution rollup increases memory usage in the metrics transformer. Review and increase the GOMEMLIMIT setting for the metrics transformer if needed, especially for high-cardinality deployments.

Kafka Rollup Topic

Metrics rollup is enabled by default in 3.5.3. Before upgrading, ensure that kf_metrics_rollup_topic is included in your Kafka topics list in the Helm values. Specify the same number of partitions as the existing kf_metrics_topic.

Post-Upgrade Steps

Multi-Rollup Resolution

After the upgrade, wait for the setup-pinot job to complete, then restart the Pinot STS (StatefulSet) to pick up the new rollup schema and table configuration. Existing metrics data is automatically compatible with multi-rollup resolution.

Pod Security Configuration (Optional)

Review your Helm values if you use custom security configurations. All services now run as non-root users by default and support configurable service accounts and security context. Existing deployments without custom security settings are unaffected.

Scheduled Views

Scheduled views have been redesigned. You must recreate your scheduled views after upgrading. Data continuity from the previous implementation cannot be guaranteed.

3.5.2

Pre-Upgrade Steps

There are no specific pre-upgrade steps for upgrading to Release 3.5.2.

Post-Upgrade Steps

Container Image Signature Verification (Optional)

Starting with 3.5.2, all Kloudfuse container images and Helm charts are signed. You can optionally verify image signatures before deployment.

Logs Parser Restart

After upgrading to 3.5.2, you must restart the logs-parser to ensure proper functionality.

kubectl scale sts logs-parser -n kfuse --replicas=0

kubectl scale sts logs-parser -n kfuse --replicas=<numNodes>

3.4.4 - p1

This guide covers upgrading to version 3.4.4-p1. This guide can be used for upgrading from 3.4.3 to 3.4.4-p1 and for upgrading from 3.4.4 to 3.4.4-p1; the only difference is in Phase 1 configuration.

Important Notes

Indentation Matters

The kafka-kraft section must be at the same indentation level as the kafka section (root level), NOT under the global section.

Disk Size

Always copy the persistence disk size from your existing kafka broker to the kafka-kraft broker configuration.

Pre-Upgrade Steps

Update kfuse-vector Configuration

The kfuse-vector component has been renamed to kfuse-archival-vector. If your values.yaml contains a kfuse-vector section, you must rename it before upgrading to 3.4.4:

# Old configuration (3.4.3 and earlier)
kfuse-vector:
  <your-configuration>

# New configuration (3.4.4 and later)
kfuse-archival-vector:
  <your-configuration>

Phase 1: Deploy Both Legacy Kafka and Kafka-Kraft

Deploy both legacy kafka and kafka-kraft services, but continue using legacy kafka for all operations.

  1. Update custom_values.yaml

Add the three legacy flags under the global.kafka section:

global:
     kafka:
       deployLegacy: true
       useLegacy: true
       ingesterUseLegacy: true
  1. Configure Kafka Services

Add the kafka-kraft section at the same indentation level as the kafka section (NOT under global).

  1. Run Helm Upgrade
helm upgrade -n kfuse kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse -f custom_values.yaml --version 3.4.4-p1

Post-Upgrade Steps

Let the New Kafka-Kraft Bake for 24hrs

After successful migration and a waiting period of 24hrs, the legacy kafka-broker and kafka-zookeeper PVCs should be deleted:

kubectl get pvc -n kfuse | grep kafka-zookeeper
kubectl delete pvc data-kafka-zookeeper-0

kubectl get pvc -n kfuse | grep kafka-broker
kubectl delete pvc data-kafka-broker-0

3.4.3

Pre-Upgrade Steps

If you plan to use GCP Stackdriver metrics and enrichment features, create a GCP service account secret before upgrading.

  1. Follow the instructions at GCP Metrics — Service Account to create a service account with the required permissions.

  2. Create the secret in your Kubernetes cluster:

kubectl create secret generic kfuse-gcp-credentials \
     --from-file=key.json=<path-to-service-account-json>

Post-Upgrade Steps

After upgrading to 3.4.3, perform a rolling restart of all Pinot components to ensure proper initialization:

kubectl rollout restart statefulset -l app=pinot

Verify all Pinot pods are running:

kubectl get pods -l app=pinot