# Upgrade Kloudfuse

## Upgrade command

1. Before performing an upgrade, validate that the upgrade won’t revert any customization on your cluster. See [Upgrade validation](https://docs.kloudfuse.com/platform/3.4.0.p1/upgrade-validation/)

2. To check which Kloudfuse version you have, run the following command:

```console
   helm list
   ```

3. Run the `upgrade` command.

```console
   helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
        -n kfuse \
        --version <VERSION> \
        -f custom-values.yaml
   ```

|     |     |
| --- | --- |
| **1** | `version`: Valid Kloudfuse release value; use the most recent one. |

## Upgrading to Latest Kloudfuse Releases

### 3.4.4 - p1

This guide covers upgrading to version 3.4.4-p1. This guide can be used for upgrading from 3.4.3 to 3.4.4-p1 and for upgrading from 3.4.4 to 3.4.4-p1; the only difference is in Phase 1 configuration.

#### Important Notes

- **Indentation Matters**: The `kafka-kraft` section must be at the same indentation level as the `kafka` section (root level), NOT under the `global` section.
- **Disk Size**: Always copy the persistence disk size from your existing kafka broker to the kafka-kraft broker configuration.
- **Version Consistency**: Use version `3.4.4-p1` for all helm upgrade commands throughout the process.

Scripts referenced in this guide are available at [GitHub scripts](https://github.com/kloudfuse-ext/customer/tree/main/scripts):

- `pause_consumption.sh` - Pauses Pinot consumption on all tables
- `resume_consumption.sh` - Resumes Pinot consumption on all tables
- `get_consuming_segments_info.sh` - Gets current status of consuming segments

#### Phase 1: Deploy Both Legacy Kafka and Kafka-Kraft

Deploy both legacy kafka and kafka-kraft services, but continue using legacy kafka for all operations.

1. Update `custom_values.yaml`

Add the three legacy flags under the `global.kafka` section:
   ```yaml
   global:
        kafka:
          deployLegacy: true
          useLegacy: true
          ingesterUseLegacy: true
   ```

2. Configure Kafka Services

Add the `kafka-kraft` section at the same indentation level as the `kafka` section (NOT under `global`).
   
   |     |     |
   | --- | --- |
   |  | Ensure that the existing `kafka.broker` disk size is copied to `kafka-kraft.broker`. For example, if your existing kafka has persistence of 200Gi, copy it to the kafka-kraft section. |

If upgrading from 3.4.3
   ```yaml
   kafka:
        broker:
          persistence:
            size: 200Gi

kafka-kraft:
        broker:
          persistence:
            size: 200Gi
   ```

If upgrading from 3.4.4

Make sure your kafka section has the zookeeper and the kraft flag both set as below. Also add the `kafka-kraft` section as shown:
   ```yaml
   kafka:
        image:
          registry: us.gcr.io/mvp-demo-301906
          repository: kfuse/kafka
          tag: 4.1.0-9cc8552
        broker:
          persistence:
            size: 200Gi
        zookeeper:
          enabled: false
        kraft:
          enabled: true

kafka-kraft:
        broker:
          persistence:
            size: 200Gi
   ```

3. Run Helm Upgrade

```console
   helm upgrade -n kfuse kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse -f custom_values.yaml --version 3.4.4-p1
   ```

4. Wait for Deployment

Wait for `kafka-kraft-broker` and `kafka-kraft-controller` pods to be up and running, and for the kafka topic creator job to finish.

#### Phase 2: Switch Ingester to Kafka-Kraft

Switch the ingester to use the new kafka-kraft by removing `ingesterUseLegacy` from `custom_values.yaml`.

1. Update `custom_values.yaml`

Remove only the `ingesterUseLegacy` flag from the `global.kafka` section. The `kafka` and `kafka-kraft` sections remain unchanged:
   ```yaml
   global:
        kafka:
          deployLegacy: true
          useLegacy: true
   ```

2. Run Helm Upgrade
   ```console
   helm upgrade -n kfuse kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse -f custom_values.yaml --version 3.4.4-p1
   ```

3. Check Kafka Consumer Lag

Check kafka consumer lag on `kafka-broker-0` by running the below code snippet. Output of the code will show multiple topics with multiple columns; once the values in the lag column is all zero, then move onto the next step.
   ```console
   kubectl exec -ti -n kfuse kafka-broker-0 -- bash
   unset JMX_PORT
   /opt/bitnami/kafka/bin/kafka-consumer-groups.sh \
        --bootstrap-server :9092 --describe --all-groups
   ```

4. Pause Pinot Consumption

Pause Pinot consumption by first port-forwarding to `pinot-controller-0` and then running the [`pause_consumption.sh`](https://github.com/kloudfuse-ext/customer/blob/main/scripts/pause_consumption.sh) script:
   ```console
   kubectl port-forward -n kfuse pinot-controller-0 9000:9000
   ```
   Then run the [`pause_consumption.sh`](https://github.com/kloudfuse-ext/customer/blob/main/scripts/pause_consumption.sh) script.

5. Wait for Segment Sealing

Run [`get_consuming_segments_info.sh`](https://github.com/kloudfuse-ext/customer/blob/main/scripts/get_consuming_segments_info.sh) (pinot-controller needs to be port forwarded) to get the current status. To continue with the upgrade the segments need to be sealed, which can be verified if the map for `_segmentToConsumingInfoMap` element doesn’t contain any element in `{}`.

Example output when segments are sealed:
   ```console
   ~/get_consuming_segments_info.sh
   Fetching realtime tables...
   Found tables:
   kf_events_REALTIME
   kf_logs_REALTIME
   kf_logs_views_REALTIME
   kf_metrics_REALTIME
   kf_metrics_rollup_REALTIME
   kf_rum_actions_REALTIME
   kf_rum_errors_REALTIME
   kf_rum_longtasks_REALTIME
   kf_rum_resources_REALTIME
   kf_rum_views_REALTIME
   kf_traces_REALTIME
   kf_traces_errors_REALTIME

Getting consuming segments info for: kf_events
        (from kf_events_REALTIME)
   {"serversFailingToRespond":0,
       "serversUnparsableRespond":0,
       "_segmentToConsumingInfoMap":{}} 
   ```

#### Phase 3: Switch All Services to Kafka-Kraft

Switch all other services to use `kafka-kraft`.

1. Update `custom_values.yaml`

Remove the `global.kafka` and `kafka` sections from `custom_values.yaml`. Only the `kafka-kraft` section is needed. The default helm configuration for 3.4.4-p1 already uses the new kafka for all services.

3. Re-enable Pinot Consumption

Once the setup-pinot job has completed, re-enable pinot consumption on all tables by running [`resume_consumption.sh`](https://github.com/kloudfuse-ext/customer/blob/main/scripts/resume_consumption.sh).

#### Post-upgrade Steps

1. Let the New Kafka-Kraft Bake for 24hrs

After successful migration and a waiting period of 24hrs, the legacy kafka-broker and kafka-zookeeper PVCs should be deleted:
   ```console
   kubectl get pvc -n kfuse | grep kafka-zookeeper
   # Add the pvc names for all kafka-zookeeper
   kubectl delete pvc data-kafka-zookeeper-0

kubectl get pvc -n kfuse | grep kafka-broker
   # Add the pvc names for all kafka-broker instances
   kubectl delete pvc data-kafka-broker-0
   ```

### 3.4.4

#### Pre-upgrade Steps

1. Update `kfuse-vector` Configuration

The `kfuse-vector` component has been renamed to `kfuse-archival-vector`. If your `values.yaml` contains a `kfuse-vector` section, you must rename it before upgrading to 3.4.4:
   ```yaml
   # Old configuration (3.4.3 and earlier)
   kfuse-vector:
     <your-configuration>

# New configuration (3.4.4 and later)
   kfuse-archival-vector:
     <your-configuration>
   ```

#### Kafka 4.1 Migration

|     |     |
| --- | --- |
|  | This release upgrades Kafka to version 4.1 with KRaft mode, which requires a multi-phase migration from ZooKeeper. Follow all phases carefully to ensure a successful migration. |

#### Phase 1: Enable KRaft Migration Mode

1. Record your ZooKeeper cluster ID:
   ```console
   kubectl exec -it <zk-pod> -- zkCli.sh get /cluster/id
   ```
   The output should look like: `{"version":"1","id":"gUC0M1nBRH2gkWHLSxD5lQ"}`

Note the cluster ID value (e.g., `gUC0M1nBRH2gkWHLSxD5lQ`)

2. Configure Kafka with KRaft migration mode in your `values.yaml`:
   ```yaml
   kafka:
        kraft:
          enabled: true
          clusterId: "<YOUR_CLUSTER_ID>"
        controller:
          controllerOnly: true
          zookeeperMigrationMode: true
        broker:
          zookeeperMigrationMode: true
   ```

3. Apply the changes using Helm upgrade with version 3.4.3

4. Verify migration is running by checking controller status:
   ```console
   kubectl exec -it kafka-controller-0 -- /bin/bash
   kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 describe --status
   ```

5. Verify migration completed:
   ```console
   kubectl logs kafka-controller-0 | grep -E "Completed migration of metadata from ZooKeeper to KRaft"
   ```

#### Phase 2: Disable Broker Migration Mode

1. Once migration reaches running state and brokers have caught up, turn off migration on brokers:
   ```yaml
   kafka:
        kraft:
          enabled: true
          clusterId: "<YOUR_CLUSTER_ID>"
        controller:
          controllerOnly: true
          zookeeperMigrationMode: true  # keep on until all brokers are flipped
        broker:
          zookeeperMigrationMode: false  # brokers now run as pure KRaft brokers
   ```

2. Apply changes using Helm upgrade with version 3.4.3.

3. Ensure kafka-broker pods restart successfully.

4. Verify brokers are healthy:
   ```console
   kubectl exec -it kafka-controller-0 -- /bin/bash
   kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 describe --status
   ```

#### Phase 3: Disable ZooKeeper

1. Finalize migration by disabling ZooKeeper:
   ```yaml
   kafka:
        kraft:
          enabled: true
        controller:
          controllerOnly: true
          zookeeperMigrationMode: false  # finalize migration
        broker:
          zookeeperMigrationMode: false
        zookeeper:
          enabled: false  # turn off ZooKeeper subchart
   ```

2. Apply changes using Helm upgrade with version 3.4.3

3. Verify `kafka-broker` and `kafka-controller` pods restart successfully and `kafka-zookeeper` pods are removed.

#### Phase 4: Upgrade to 3.4.4

1. Remove migration-specific configuration from your `values.yaml`. The configuration should now only include:
   ```yaml
   kafka:
        # kraft:
        #   enabled: true
        #   clusterId: "<YOUR_CLUSTER_ID>"
        # controller:
        #   controllerOnly: true
        #   zookeeperMigrationMode: false
        # zookeeper:
        #  enabled: false
        broker:
          # zookeeperMigrationMode: false
          <anything broker related you had>
   ```

|     |     |
| --- | --- |
|  | You do not need `kafka.kraft.*` anymore as it is stored as a secret. You also do not need controller-related configuration as `values.yaml` has the defaults already set. |

2. Proceed with the standard upgrade to 3.4.4.

#### Post-upgrade Steps

Since the `kafka-zookeeper` is no longer deployed, you can remove its associated `persistentVolumeClaims` with the following command:
   ```console
   kubectl get pvc -n kfuse | grep kafka-zookeeper
   kubectl delete pvc data-kafka-zookeeper-0 # Add the pvc names for all the kafka-zookeeper persistentVolumeClaims
   ```

### 3.4.3

#### Pre-upgrade Steps

If you plan to use GCP Stackdriver metrics and enrichment features, create a GCP service account secret before upgrading.

1. Follow the instructions at [GCP Metrics Credentials](https://docs.kloudfuse.com/platform/3.4.0.p1/cloud-service-gcp-metrics/#credentials) to create a service account with the required permissions.

2. Create the secret in your Kubernetes cluster:
   ```console
   kubectl create secret generic kfuse-sd-secret \
        --from-file=key.json=<path-to-service-account-json>
   ```

3. Configure the secret name in your `values.yaml`:
   ```yaml
   global:
        gcpConfig:
          secretName: "kfuse-sd-secret"
   ```