Disaster Recovery - GKE :: Kloudfuse Docs

Disaster Recovery - GKE

Learn how to implement a disaster recovery plan for the Google Kubernetes Engine (GKE) platform. For more information, see the Google documentation on Backup for GKE.

Google Cloud Backup

To handle region and availability zone failures, Kloudfuse recommends the following steps:

  1. Create a GKE cluster for the primary region and availability zone. See the Google documentation for Creating a zonal cluster.

  2. Create another GKE cluster in the failover region and availability zone.

  3. For both GKE clusters, complete the DNS/TLS setup prerequisites as described in Https Configuration or TLS Host Based Routing, depending on your approach.

  4. Enable GKE backup and recovery for these clusters. See these instructions in the Google documentation:

  1. Create a GCS bucket with cross-region and availability zone access.

  2. Install the Kloudfuse Helm chart in the primary GKE cluster. See Installation Topics.
Kloudfuse adds the following labels to user-configured secrets and config maps, such as SSO/SAML setup, TLS certificates, and so on:
<br>app.kubernetes.io/instance: kfuse<br>

Configure Backup Policy

Set up the GKE backup policy based on the Recovery Point Objective and retention policy of your organization.

This automatically creates backups of Kloudfuse installations to fulfill the RPO.

GKE retains backups for the specified period and deletes them afterward.

GKE uses different configuration parameters depending on the RPO:

RPO ≥ 60 minutes

Use the following example as a guide for setting up a GKE backup policy based on the target-rpo-minutes option.

gcloud beta container backup-restore backup-plans create kloudfuse-backup-plan \
    --project=<projectID> \ (1)
    --location=<location of the primary GKE cluster> \ (2)
    --cluster=<name of the primary GKE cluster> \ (3)
    --selected-applications=<namespace>/kloudfuse \ (4)
    --include-secrets \ (5)
    --target-rpo-minutes=60 \ (6)
    --backup-retain-days=1 (7)

Copied!
1 project: The ID of the Google Cloud project.
2 location: Location of the primary gke cluster — the region of the backup/restore plan; see Google documentation on Available regions and zones.
3 cluster: The name of the primary GKE cluster to back up and restore.
4 selected-applications: A list of protected applications; see Google documentation on Define custom backup and restore logic.
5 include-secrets: An optional argument to include Secret resources if they are in the scope of the backup and restore policy.
6 target-rpo-minutes: Minimum RPO is 60 minutes. See RPO < 60 minutes for instructions on how to configure shorter RPOs.
7 backup-retain-days: Specify the data retention time, in days.

RPO < 60 minutes

Use the following example as a guide for setting up a GKE backup policy based on the --cron-schedule option.

The value mask is mins hrs days months yrs. The * (wildcard) character is equivalent to all possible values.

For example, the value 10 3 * * * creates a backup at 3:10 AM every day. All times are in UTC format.

gcloud beta container backup-restore backup-plans create kloudfuse-backup-plan \
    --project=<project> \ (1)
    --location=<location of the primary GKE cluster> \ (2)
    --cluster=<name of the primary GKE cluster> \ (3)
    --selected-applications=<namespace>/kloudfuse \ (4)
    --include-secrets \ (5)
    --cron-schedule="10 3 * * *" \ (6)
    --backup-retain-days=1 (7)

Copied!

Restore

  1. Configure the restore plan:
gcloud beta container backup-restore restore-plans create kloudfuse-restore-plan \
       --project=<project> \ (1)
       --location=<location of the failover GKE cluster> \ (2)
       --backup-plan=kloudfuse-backup-plan \ (3)
       --cluster=<name of the failover GKE cluster> \ (4)
       --cluster-resource-conflict-policy=delete-and-restore \ (5)
       --selected-applications=<namespace>/kloudfuse \ (6)
       --volume-data-restore-policy=restore-volume-data-from-backup (7)

Copied!
1 project: The ID of the Google Cloud project.
2 location: Location of the primary gke cluster — the region of the backup/restore plan; see Google documentation on Available regions and zones.
3 backup-plan: The saved plan for Kloudfuse backups.
4 cluster: The name of the failover GKE cluster.
5 cluster-resource-conflict-policy: Defines how conflicting cluster-scoped resources already present in the failover cluster are handled during restore; see Google documentation for ClusterResourceConflictPolicy.
6 selected-applications: A list of protected applications; see Google documentation on Define custom backup and restore logic.
7 Defines how to populate data for restored volumes. See Google documentation on how to Create a restore plan.
  1. When a primary region or availability zone fails, the administrator can restore Kloudfuse to the target failover GKE cluster.
Kloudfuse recommends setting up an automatic detection and alerting mechanism for zone failures. 
gcloud beta container backup-restore restores create kloudfuse-restore \
       --project=<project> \ (1)
       --location=<location of the failover GKE cluster> \ (2)
       --restore-plan=kloudfuse-restore-plan \ (3)
       --backup=<select the latest GKE backup for Kloudfuse> (4)

Copied!
1 project: The ID of the Google Cloud project.
2 location: Location of the primary gke cluster — the region of the backup/restore plan; see Google documentation on Available regions and zones.
3 restore-plan: The saved plan for Kloudfuse restores.
4 backup: The backup to restore; must be located in the backup plan used by the restore plan.
  1. When using a regional static IP address for the load balancer, complete these additional steps:
  1. When using a global static IP address for the load balancer, ensure that the failed GKE cluster is fenced off. If the primary GKE cluster recovers, it can accidentally bind to the same static IP address.

Backup Limitations

Backup works on Kubernetes resources and underlying persistent volumes only.

It does not cover: