# Cross-AZ Recovery

Kloudfuse supports a cost-efficient multi-AZ deployment model that uses cloud storage (S3, GCS, or Azure Blob) as a synchronization mechanism between availability zones. This avoids continuous cross-AZ data transfer costs while still enabling fast failover and full data recovery after an AZ outage.

## Deployment Architecture

The design keeps each Kloudfuse cluster isolated within a single AZ, while sharing cloud storage across AZs at the region level.

Key characteristics:

- **Node groups** are deployed within a single AZ. Kloudfuse 2 on AZ2 can be pre-created with zero nodes running, keeping standby costs near zero.

- **Load balancer** is deployed in a multi-AZ configuration and routes all traffic to the active cluster. It is the only component that spans AZs in normal operation.

- **Cloud storage** (S3 / GCS / Azure Blob) is region-scoped and accessible from all AZs. It serves as the shared synchronization layer between the two clusters.

- **Pinot segments** are continuously backed up to cloud storage as they are created.

- **PostgreSQL** (dashboards, alerts, configuration) is periodically dumped to cloud storage.

- **AZ-Service** \- The `az-service` which backs up its configuration databases (`configdb`, `orchestratordb`) to cloud storage. Starting with release 3.4.2, `az-service` is enabled by default.

## Prerequisites

Before you begin:

- A **fresh installation for AZ-1 and AZ-2**. A Kloudfuse cluster cannot be upgraded to an AZ failover setup. The ingress controller (`ingress-nginx` **or**`envoy-gateway`) must **not be installed as part of the Kloudfuse cluster configuration**.

- **Cloud storage** must be set up and accessible from both AZs within the region.

- The ingress controller — either `ingress-nginx` or `envoy-gateway` — must be installed separately from Kloudfuse.

|     |     |
| --- | --- |
|  | For a cold stand-by, after the initial install, you can scale the number of nodes for AZ2 down to zero. This will reduce the costs of instances that you are not actually using. |

### Building the Kloudfuse Clusters

The Kloudfuse Clusters need to be both installed in the same Kubernetes Cluster. However, each Cluster needs to run on nodes that are tied to a specific AZ. So the cluster for AZ1 needs to have all the nodes it intends to run installed in AZ1 and the backup cluster in AZ2 needs to be associated with nodes that will run only in AZ2. To do this you need to add nodeAffinity/Tolerations in the global section of the `custom_values.yaml`

Refer to the documentation at [Node Affinity and Tolerations](https://docs.kloudfuse.com/platform/4.1.0/setup/advanced/node-affinity-tolerations/) on how to set this in the `custom_values.yaml`

### Ingress Installation

A dedicated ingress controller installed as a standalone Helm release, separate from the Kloudfuse chart. You need to disable the bundled controller in `custom-values.yaml`.

Pick whichever controller your deployment uses:

- **ingress-nginx** — see [Standalone Nginx Ingress](https://docs.kloudfuse.com/platform/4.1.0/setup/network/standalone/) for full configuration and installation instructions.

- **Envoy Gateway** — see [Standalone Envoy Gateway](https://docs.kloudfuse.com/platform/4.1.0/setup/network/envoy-standalone/) for full configuration, installation, and failover instructions. Envoy Gateway is required if your cross-AZ setup needs the same load balancer IP and DNS to survive a Kloudfuse cluster swap during failover (the standalone Envoy controller and its load balancer outlive any individual Kloudfuse install).

### AZ-Service

`az-service` reads its cloud storage configuration from `global.cloudStorage`, which is the recommended approach since all Kloudfuse components share the same storage account. The `zone` field is always required under `az-service.config`. An `az-service`-specific override is available if the service needs to write to a different bucket.

#### Cloud Storage Configuration (global.cloudStorage)

Set `global.cloudStorage` in your `custom-values.yaml`. `az-service` will use this configuration automatically alongside the rest of the Kloudfuse components.

```yaml
global:
  cloudStorage:
    type: <s3 | gcs | azure> (1)
    useSecret: true (2)
    secretName: cloud-storage-secret (3)
    s3:
      region: <region> (4)
      bucket: <bucket> (5)
    # gcs:
    #   bucket: <bucket>
    # azure:
    #   container: <container>

az-service:
  config:
    zone: "<zone-id>" (6)
    backupIntervalMinutes: 60 (7)
```

#### Cloud Storage Credentials

Create the Kubernetes secret before running the Helm install. The secret name must match `cloudStorage.secretName` in your configuration.

**S3** — the secret must contain `accessKey` and `secretKey`:

```console
kubectl create secret generic cloud-storage-secret \
  --from-literal=accessKey=<accessKey> \
  --from-literal=secretKey='<secretKey>'
```

**GCS** — the secret must contain the JSON credential file named `secretKey`:

```console
kubectl create secret generic cloud-storage-secret --from-file=./secretKey
```

**Azure** — the secret must contain the storage account `connectionString`:

```console
kubectl create secret generic cloud-storage-secret \
  --from-literal=connectionString='<connectionString>'
```

#### Overriding Cloud Storage for AZ-Service

If `az-service` needs to write backups to a different bucket than the one used by the rest of Kloudfuse, set `az-service.config.cloudStorage` explicitly. This takes precedence over `global.cloudStorage` for `az-service` only.

```yaml
az-service:
  config:
    zone: "<zone-id>"
    cloudStorage:
      type: s3
      useSecret: true
      secretName: az-service-storage-secret
      s3:
        region: <region>
        bucket: <az-service-bucket>
```

## Failover Steps

The following steps shift traffic from Kloudfuse 1 (AZ1) to Kloudfuse 2 (AZ2). These steps apply whether you are responding to a failure or executing a planned migration. The same steps are used in reverse to move back from AZ2 to AZ1.

### Switching steps

1. **Install secrets in Kloudfuse AZ2.**

Re-create all secrets from AZ1 in the AZ2 cluster before running the Helm install. The following are the most common required secrets.

- **Image pull credentials** (required to pull Kloudfuse container images)

- **Cloud storage secret** (required by `az-service` for backup and restore)

- **PostgreSQL credentials** (if using a cloud-managed database)

- **TLS certificate** (if HTTPS is enabled on the cluster)

- **SSO Metadata Secret** (required for SAML authentication)

2. **Scale up Node Group 2 on AZ2.**

If the node group was pre-created with zero nodes, scale it up to the required node count. This step make take some time as the Cloud Provider launches new nodes.

3. **Do a helm upgrade on the Cluster in AZ2**

Run the Helm upgrade command to install or update Kloudfuse on the AZ2 cluster. Ensure `custom-values.yaml` reflects the AZ2 configuration.

```console
helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
     -n kfuse \
     --version <VERSION> \ (1)
     -f custom-values.yaml
```

4. **Update Nginx Controller route Traffic to Kloudfuse AZ2.**

Update the load balancer to direct traffic to AZ2. From this point, Kloudfuse 2 is accessible for UI access and data ingestion. No data loss occurs at this step.

```console
helm upgrade --install <release-name> ingress-nginx/ingress-nginx \ (1)
     --namespace <release-name> --create-namespace \
     --version <version> \ (2)
     -f values.yaml
```

5. **Port-forward az-service on AZ1 and AZ2**

Open two terminal sessions — one for each cluster. Switch context, set the namespace, and forward the `az-service` port so the `curl` commands in the steps below can reach each cluster locally. Replace `<AZ1_CONTEXT>` and `<AZ2_CONTEXT>` with the context names from `kubectl config get-contexts`.

**AZ1:**

```console
kubectl config use-context <AZ1_CONTEXT>
kubectl config set-context --current --namespace kfuse
kubectl port-forward svc/az-service 8081:8080
```

**AZ2** (separate terminal):

```console
kubectl config use-context <AZ2_CONTEXT>
kubectl config set-context --current --namespace kfuse
kubectl port-forward svc/az-service 8082:8080
```

Keep both port-forward sessions running for the duration of the failover. The subsequent `curl` commands use `localhost:8081` for AZ1 and `localhost:8082` for AZ2.

6. **Pause Kloudfuse AZ1.**

Silences alerts and pauses Pinot consumption on AZ1, ensuring all in-flight data is flushed to Kafka before the segment state is transferred.

```console
curl -X POST http://localhost:8081/pause
```

To bypass the Kafka consumption check (for example, if AZ1 is unreachable):

```console
curl -X POST "http://localhost:8081/pause?force=true"
```

7. **Activate Kloudfuse AZ2.**

Restores PostgreSQL (`configdb`, `orchestratordb`) from the S3 backup, rehydrates missing Pinot segments from cloud storage, and resumes Pinot consumption. The `zone` parameter must match the zone ID used by AZ1’s `az-service`.

```console
curl -X POST "http://localhost:8082/activate?zone=<zone-id>"
```

To restore to a specific point in time, add the `latestTime` parameter:

```console
curl -X POST "http://localhost:8082/activate?zone=<zone-id>&latestTime=<timestamp>"
```

To monitor rehydration progress:

```console
curl http://localhost:8082/status
```

8. **Resume alerts on Kloudfuse AZ2.**

Unsilences alerts once activation is complete and you have verified that the cluster is healthy.

```console
curl -X POST http://localhost:8082/resume
```

### AZ-Service command reference

The `az-service` exposes four endpoints. All write operations use `POST`.

#### Pause

Silences alerts and pauses Pinot consumption. Use this on the cluster you are failing away from.

```console
curl -X POST https://<KFUSE_ENDPOINT>/pause
```

| Parameter | Required | Description |
| --- | --- | --- |
| `force` | No | When `true`, skips the Kafka consumption check. Use when the source cluster is unreachable. |

#### Activate

Restores `configdb` and `orchestratordb` from S3, rehydrates missing Pinot segments from cloud storage, and resumes Pinot consumption. Use this on the cluster you are failing over to.

```console
curl -X POST "https://<KFUSE_ENDPOINT>/activate?zone=<zone-id>"
```

| Parameter | Required | Description |
| --- | --- | --- |
| `zone` | Yes | Zone ID of the source backup. Must match the `zone` value configured in the source cluster’s `az-service`. |
| `latestTime` | No | Restore `configdb` and `orchestratordb` to this timestamp. Defaults to the most recent available backup. |
| `namespace` | No | Kubernetes namespace of the source cluster. When provided, `activate` calls `/pause` on the source cluster automatically. |
| `force` | No | Passed through to the source cluster pause call when `namespace` is set. |
| `batchSize` | No | Number of Pinot segments to rehydrate per batch. Defaults to the service configuration value. |
| `ignoreError` | No | When `true`, continues activation even if non-critical errors occur. |

#### Status

Returns the current Pinot rehydration progress. Poll this after `activate` to track completion.

```console
curl https://<KFUSE_ENDPOINT>/status
```

#### Resume

Unsilences alerts. Run this after `activate` completes and the cluster is verified healthy.

```console
curl -X POST https://<KFUSE_ENDPOINT>/resume
```