# Disaster Recovery on the Google Kubernetes Engine (GKE) Platform

Learn how to implement a Disaster Recovery plan for Google Kubernetes Engine (GKE) Platform. For more information, see Google documentation on [Backup for GKE](https://cloud.google.com/kubernetes-engine/docs/add-on/backup-for-gke/concepts/backup-for-gke).

## Google Cloud Backup

To handle region and availability zone failures, we recommend the following steps:

1. Create a GKE cluster for the primary region and availability zone; see Google documentation for [Creating a zonal cluster](https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-zonal-cluster).

2. Create another GKE cluster in the fail-over region and availability zone.

3. For both GKE clusters, complete the DNS/TLS setup prerequisites as described in [Configure HTTPS/TLS on the Kloudfuse Ingress](https://docs.kloudfuse.com/platform/3.2.5/protocol-configure-ingress/#prerequisites) or [Configure Kloudfuse Stack to Work with TLS Termination](https://docs.kloudfuse.com/platform/3.2.5/protocol-configure-termination/#prerequisites), depending on your approach.

4. Enable GKE backup and recovery for these clusters. See these instructions in Google documentation:

- [Enable Backup for GKE API](https://cloud.google.com/kubernetes-engine/docs/add-on/backup-for-gke/how-to/install)
   - [Enable Backup for GKE for a cluster](https://cloud.google.com/kubernetes-engine/docs/add-on/backup-for-gke/how-to/enable-gke-cluster)

5. Create a GCS bucket with cross-region and availability zone access.

6. Install the Kloudfuse helm chart in the primary GKE cluster; see [Install Kloudfuse Using Helm](https://docs.kloudfuse.com/platform/3.2.5/install/).

|     |     |
| --- | --- |
|  | Kloudfuse add the following labels for user-configured secrets and `config` maps, such as SSO/SAML setup, TLS certificates, and so on.<br>```<br>app.kubernetes.io/instance: kfuse<br>``` |

## Configure Backup Policy

Set up the GKE backup policy based on the [Recovery Point Objective (RPO)](https://docs.kloudfuse.com/platform/3.2.5/glossary/#rpo) and retention policy of your organization.

This automatically creates backups of Kloudfuse installations to fulfill the RPO.

GKE retains the backups for the specified period, and deletes them after.

GKE uses different configurations parameters, depending on the RPO:

- For RPOs of [60 minutes or longer](https://docs.kloudfuse.com/platform/3.2.5/disaster-recovery-gke/#target-rpo-minutes), use the option `target-rpo-minutes`.

- For RPOs [less than 60 minutes](https://docs.kloudfuse.com/platform/3.2.5/disaster-recovery-gke/#cron-schedule), use the option `cron-schedule`.

- RPO ≥ 60 minutes

- RPO < 60 minutes

Use the following example as a guide for setting up a GKE backup policy based on the `target-rpo-minutes` option.

```console
gcloud beta container backup-restore backup-plans create kloudfuse-backup-plan \
    --project=<projectID> \ (1)
    --location=<location of the primary GKE cluster> \ (2)
    --cluster=<name of the primary GKE cluster> \ (3)
    --selected-applications=<namespace>/kloudfuse \ (4)
    --include-secrets \ (5)
    --target-rpo-minutes=60 \ (6)
    --backup-retain-days=1 (7)
```

Copied!

|     |     |
| --- | --- |
| **1** | `project`: The ID of the Google Cloud project. |
| **2** | `location`: Location of the primary gke cluster — the region of the backup/restore plan; see Google documentation on [Available regions and zones](https://cloud.google.com/compute/docs/regions-zones#available). |
| **3** | `cluster`: The name of the primary GKE cluster to back up and restore. |
| **4** | `selected-applications`: A list of protected applications; see Google documentation on [Define custom backup and restore logic](https://cloud.google.com/kubernetes-engine/docs/add-on/backup-for-gke/how-to/protected-application). |
| **5** | `include-secrets`: An optional argument to include `Secret` resources if they are in the scope of the backup and restore policy. |
| **6** | `target-rpo-minutes`: Minimum RPO is 60 minutes. See [RPO < 60 minutes](https://docs.kloudfuse.com/platform/3.2.5/disaster-recovery-gke/#cron-schedule) for instructions on how to configure shorter RPOs. |
| **7** | `backup-retain-days`: Specify the data retention time, in days. |

Use the following example as a guide for setting up a GKE backup policy based on the `--cron-schedule` option.

The value mask is `mins hrs days months yrs`. The `*` (wildcard) character is equivalent to all possible values.

For example, the value of `10 3 * * *` creates a backup at 3:10 AM every day. All times are in UTC format.

```console
gcloud beta container backup-restore backup-plans create kloudfuse-backup-plan \
    --project=<project> \ (1)
    --location=<location of the primary gke cluster> \ (2)
    --cluster=<name of the primary gke cluster> \ (3)
    --selected-applications=<namespace>/kloudfuse \ (4)
    --include-secrets \ (5)
    --cron-schedule="10 3 * * *" \ (6)
    --backup-retain-days=1 (7)
```

Copied!

## Restore

1. Configure the Restore plan.

```console
gcloud beta container backup-restore restore-plans create kloudfuse-restore-plan \
       --project=<project> \ (1)
       --location=<location of the failover gke cluster> \ (2)
       --backup-plan=kloudfuse-backup-plan \ (3)
       --cluster=<name of the failover gke cluster> \ (4)
       --cluster-resource-conflict-policy=delete-and-restore \ (5)
       --selected-applications=<namespace>/kloudfuse \ (6)
       --volume-data-restore-policy=restore-volume-data-from-backup (7)
```

Copied!

|     |     |
| --- | --- |
| **1** | `project`: The ID of the Google Cloud project. |
| **2** | `location`: Location of the primary gke cluster — the region of the backup/restore plan; see Google documentation on [Available regions and zones](https://cloud.google.com/compute/docs/regions-zones#available). |
| **3** | `backup-plan`: The saved plan for Kloudfuse backups. |
| **4** | `cluster`: The name of the primary GKE cluster to back up and restore. |
| **5** | `cluster-resource-conflict-policy`: Define the behavior for resource conflicts; see Google documentation for [ClusterResourceConflictPolicy](https://cloud.google.com/kubernetes-engine/docs/add-on/backup-for-gke/reference/rest/v1/RestoreConfig#clusterresourceconflictpolicy). |
| **6** | `selected-applications`: A list of protected applications; see Google documentation on [Define custom backup and restore logic](https://cloud.google.com/kubernetes-engine/docs/add-on/backup-for-gke/how-to/protected-application). |
| **7** | Defines how to populate data for restored volumes. See Google documentation on how to [Create a restore plan](https://cloud.google.com/kubernetes-engine/docs/add-on/backup-for-gke/how-to/restore-plan#create_a_restore_plan). |

2. When a primary region and availability zone fails, the administrator can restore the Kloudfuse to the target/fail-over GKE cluster.

|     |     |
| --- | --- |
|  | We recommend that you set up an automatic detection and alerting mechanism. |

```console
gcloud beta container backup-restore restores create kloudfuse-restore \
       --project=<project> \ (1)
       --location=<location of the fail-over gke cluster> \ (2)
       --restore-plan=kloudfuse-restore-plan \ (3)
       --backup=<select the latest GKE backups for the kloudfuse> (4)
```

Copied!

|     |     |
| --- | --- |
| **1** | `project`: The ID of the Google Cloud project. |
| **2** | `location`: Location of the primary gke cluster — the region of the backup/restore plan; see Google documentation on [Available regions and zones](https://cloud.google.com/compute/docs/regions-zones#available). |
| **3** | `restore-plan`: The saved plan for Kloudfuse restores. |
| **4** | `backup`: The backup to restore; must be located in the backup plan used by the restore plan. |

3. When using a regional static IP address for the load balancer, complete these additional steps:

- In the Kloudfuse Ingress configuration, update the load balancer IP to the regional static IP for the fail-over region. See [Configure Helm Values for Kloudfuse Ingress](https://docs.kloudfuse.com/platform/3.2.5/protocol-configure-ingress/#helm).
   - Update the DNS record to point to this new static IP.

4. Alternatively, when using a global static IP address for the load balancer, be sure to "fence off" the failed GKE cluster. Otherwise, if the primary GKE cluster recovers, it can accidentally bind to the same static IP address.

## Backup Limitations

Backup works on Kubernetes resources and underlying persistent volumes **_only_**.

It **_does not work_** for:

- Cluster configuration information: Node configuration, node pools, initial cluster size, or enabled features.
- Container images referenced by a backup. If an image referenced by a workload manifest is removed from its image repository, subsequent restore of that configuration **_does not_** restore the workload.
- Configuration information or state of services outside the cluster: Cloud SQL or external load balancers.
- Other volume types, such as Filestore NFS or Google Cloud NetApp Volumes. However, you can use Backup for GKE to provide solutions for workloads that are backed by Filestore volumes. See [Handle Filestore volumes with Backup for GKE](https://cloud.google.com/kubernetes-engine/docs/add-on/backup-for-gke/how-to/handle-filestore-volumes).