Upgrade Kloudfuse :: Kloudfuse Docs

Upgrade Kloudfuse

Upgrade command

Before performing an upgrade, validate that the upgrade won’t revert any customization on your cluster. See Upgrade validation
To check which Kloudfuse version you have, run the following command:

   helm list

Run the upgrade command.

   helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
        -n kfuse \
        --version <VERSION> \
        -f custom-values.yaml


1	`version`: Valid Kloudfuse release value; use the most recent one.

Upgrading to Latest Kloudfuse Releases

3.4.4 - p1

This guide covers upgrading to version 3.4.4-p1. This guide can be used for upgrading from 3.4.3 to 3.4.4-p1 and for upgrading from 3.4.4 to 3.4.4-p1; the only difference is in Phase 1 configuration.

Important Notes

Indentation Matters: The kafka-kraft section must be at the same indentation level as the kafka section (root level), NOT under the global section.
Disk Size: Always copy the persistence disk size from your existing kafka broker to the kafka-kraft broker configuration.
Version Consistency: Use version 3.4.4-p1 for all helm upgrade commands throughout the process.

Scripts referenced in this guide are available at GitHub scripts:

pause_consumption.sh - Pauses Pinot consumption on all tables
resume_consumption.sh - Resumes Pinot consumption on all tables
get_consuming_segments_info.sh - Gets current status of consuming segments

Phase 1: Deploy Both Legacy Kafka and Kafka-Kraft

Deploy both legacy kafka and kafka-kraft services, but continue using legacy kafka for all operations.

Update custom_values.yaml

Add the three legacy flags under the global.kafka section:

global:
     kafka:
       deployLegacy: true
       useLegacy: true
       ingesterUseLegacy: true

Configure Kafka Services

Add the kafka-kraft section at the same indentation level as the kafka section (NOT under global).


	Ensure that the existing `kafka.broker` disk size is copied to `kafka-kraft.broker`. For example, if your existing kafka has persistence of 200Gi, copy it to the kafka-kraft section.

If upgrading from 3.4.3

kafka:
     broker:
       persistence:
         size: 200Gi

kafka-kraft:
     broker:
       persistence:
         size: 200Gi

If upgrading from 3.4.4

Make sure your kafka section has the zookeeper and the kraft flag both set as below. Also add the kafka-kraft section as shown:

kafka:
     image:
       registry: us.gcr.io/mvp-demo-301906
       repository: kfuse/kafka
       tag: 4.1.0-9cc8552
     broker:
       persistence:
         size: 200Gi
     zookeeper:
       enabled: false
     kraft:
       enabled: true

kafka-kraft:
     broker:
       persistence:
         size: 200Gi

Run Helm Upgrade

   helm upgrade -n kfuse kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse -f custom_values.yaml --version 3.4.4-p1

Wait for Deployment

Wait for kafka-kraft-broker and kafka-kraft-controller pods to be up and running, and for the kafka topic creator job to finish.

Phase 2: Switch Ingester to Kafka-Kraft

Switch the ingester to use the new kafka-kraft by removing ingesterUseLegacy from custom_values.yaml.

Update custom_values.yaml

Remove only the ingesterUseLegacy flag from the global.kafka section. The kafka and kafka-kraft sections remain unchanged:

global:
     kafka:
       deployLegacy: true
       useLegacy: true

Run Helm Upgrade

helm upgrade -n kfuse kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse -f custom_values.yaml --version 3.4.4-p1

Check Kafka Consumer Lag

Check kafka consumer lag on kafka-broker-0 by running the below code snippet. Output of the code will show multiple topics with multiple columns; once the values in the lag column is all zero, then move onto the next step.

kubectl exec -ti -n kfuse kafka-broker-0 -- bash
unset JMX_PORT
/opt/bitnami/kafka/bin/kafka-consumer-groups.sh \
     --bootstrap-server :9092 --describe --all-groups

Pause Pinot Consumption

Pause Pinot consumption by first port-forwarding to pinot-controller-0 and then running the pause_consumption.sh script:

kubectl port-forward -n kfuse pinot-controller-0 9000:9000

Then run the pause_consumption.sh script.

Wait for Segment Sealing

Run get_consuming_segments_info.sh (pinot-controller needs to be port forwarded) to get the current status. To continue with the upgrade the segments need to be sealed, which can be verified if the map for _segmentToConsumingInfoMap element doesn’t contain any element in {}.

Example output when segments are sealed:

~/get_consuming_segments_info.sh
Fetching realtime tables...
Found tables:
kf_events_REALTIME
kf_logs_REALTIME
kf_logs_views_REALTIME
kf_metrics_REALTIME
kf_metrics_rollup_REALTIME
kf_rum_actions_REALTIME
kf_rum_errors_REALTIME
kf_rum_longtasks_REALTIME
kf_rum_resources_REALTIME
kf_rum_views_REALTIME
kf_traces_REALTIME
kf_traces_errors_REALTIME

Getting consuming segments info for: kf_events
     (from kf_events_REALTIME)
{"serversFailingToRespond":0,
    "serversUnparsableRespond":0,
    "_segmentToConsumingInfoMap":{}}

Phase 3: Switch All Services to Kafka-Kraft

Switch all other services to use kafka-kraft.

Update custom_values.yaml

Remove the global.kafka and kafka sections from custom_values.yaml. Only the kafka-kraft section is needed. The default helm configuration for 3.4.4-p1 already uses the new kafka for all services.

Re-enable Pinot Consumption

Once the setup-pinot job has completed, re-enable pinot consumption on all tables by running resume_consumption.sh.

Post-upgrade Steps

Let the New Kafka-Kraft Bake for 24hrs

After successful migration and a waiting period of 24hrs, the legacy kafka-broker and kafka-zookeeper PVCs should be deleted:

kubectl get pvc -n kfuse | grep kafka-zookeeper
# Add the pvc names for all kafka-zookeeper
kubectl delete pvc data-kafka-zookeeper-0

kubectl get pvc -n kfuse | grep kafka-broker
# Add the pvc names for all kafka-broker instances
kubectl delete pvc data-kafka-broker-0

3.4.4

Pre-upgrade Steps

Update kfuse-vector Configuration

The kfuse-vector component has been renamed to kfuse-archival-vector. If your values.yaml contains a kfuse-vector section, you must rename it before upgrading to 3.4.4:

# Old configuration (3.4.3 and earlier)
kfuse-vector:
  <your-configuration>

# New configuration (3.4.4 and later)
kfuse-archival-vector:
  <your-configuration>

Kafka 4.1 Migration


	This release upgrades Kafka to version 4.1 with KRaft mode, which requires a multi-phase migration from ZooKeeper. Follow all phases carefully to ensure a successful migration.

Phase 1: Enable KRaft Migration Mode

Record your ZooKeeper cluster ID:
```
kubectl exec -it <zk-pod> -- zkCli.sh get /cluster/id
```
The output should look like: {"version":"1","id":"gUC0M1nBRH2gkWHLSxD5lQ"}

Note the cluster ID value (e.g., gUC0M1nBRH2gkWHLSxD5lQ)

Configure Kafka with KRaft migration mode in your values.yaml:

kafka:
     kraft:
       enabled: true
       clusterId: "<YOUR_CLUSTER_ID>"
     controller:
       controllerOnly: true
       zookeeperMigrationMode: true
     broker:
       zookeeperMigrationMode: true

Apply the changes using Helm upgrade with version 3.4.3

Verify migration is running by checking controller status:

kubectl exec -it kafka-controller-0 -- /bin/bash
kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 describe --status

Verify migration completed:

kubectl logs kafka-controller-0 | grep -E "Completed migration of metadata from ZooKeeper to KRaft"

Phase 2: Disable Broker Migration Mode

Once migration reaches running state and brokers have caught up, turn off migration on brokers:

kafka:
     kraft:
       enabled: true
       clusterId: "<YOUR_CLUSTER_ID>"
     controller:
       controllerOnly: true
       zookeeperMigrationMode: true  # keep on until all brokers are flipped
     broker:
       zookeeperMigrationMode: false  # brokers now run as pure KRaft brokers

Apply changes using Helm upgrade with version 3.4.3.
Ensure kafka-broker pods restart successfully.

Verify brokers are healthy:

kubectl exec -it kafka-controller-0 -- /bin/bash
kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 describe --status

Phase 3: Disable ZooKeeper

Finalize migration by disabling ZooKeeper:

kafka:
     kraft:
       enabled: true
     controller:
       controllerOnly: true
       zookeeperMigrationMode: false  # finalize migration
     broker:
       zookeeperMigrationMode: false
     zookeeper:
       enabled: false  # turn off ZooKeeper subchart

Apply changes using Helm upgrade with version 3.4.3
Verify kafka-broker and kafka-controller pods restart successfully and kafka-zookeeper pods are removed.

Phase 4: Upgrade to 3.4.4

Remove migration-specific configuration from your values.yaml. The configuration should now only include:

kafka:
     # kraft:
     #   enabled: true
     #   clusterId: "<YOUR_CLUSTER_ID>"
     # controller:
     #   controllerOnly: true
     #   zookeeperMigrationMode: false
     # zookeeper:
     #  enabled: false
     broker:
       # zookeeperMigrationMode: false
       <anything broker related you had>


	You do not need `kafka.kraft.*` anymore as it is stored as a secret. You also do not need controller-related configuration as `values.yaml` has the defaults already set.

Proceed with the standard upgrade to 3.4.4.

Post-upgrade Steps

Since the kafka-zookeeper is no longer deployed, you can remove its associated persistentVolumeClaims with the following command:

kubectl get pvc -n kfuse | grep kafka-zookeeper
kubectl delete pvc data-kafka-zookeeper-0 # Add the pvc names for all the kafka-zookeeper persistentVolumeClaims

3.4.3

Pre-upgrade Steps

If you plan to use GCP Stackdriver metrics and enrichment features, create a GCP service account secret before upgrading.

Follow the instructions at GCP Metrics Credentials to create a service account with the required permissions.

Create the secret in your Kubernetes cluster:

kubectl create secret generic kfuse-sd-secret \
     --from-file=key.json=<path-to-service-account-json>

Configure the secret name in your values.yaml:

global:
     gcpConfig:
       secretName: "kfuse-sd-secret"