# HA Multi-AZ Setup

If you need High Availability(HA) for a Kloudfuse cluster, you have the option of deploying it across multiple availability zones (multi-AZ). A multi-AZ setup minimizes downtime and maintains observability workflows in the event of a zone failure. Each zone must have at least 2 pods to run the supported components.

## Benefits of Multi-AZ Deployment

- Protects against zone-level failures
- Ensures service continuity
- Balances workload across zones
- Aligns with SRE best practices

## Prerequisites

Before you begin:

- **Kubernetes Cluster Infrastructure**

- A **_fresh installation_**. A Kloudfuse cluster cannot be upgraded to a Multi-AZ setup.
  - Nodes to be set up across **3 availability zones** (required)
  - Each availability zone must have an **equal number of nodes** (required)
  - The total number of nodes must be a multiple of 6 (required)
  - Nodes cannot have additional taints outside what is used by Kloudfuse (required) Additional taints will cause validations to fail and those will not be used

- **Cloud Managed PostgreSQL instance**

- Cloud Provider managed PostgreSQL instance:
    - AWS RDS
    - GCP Cloud SQL
    - Azure Database for PostgreSQL
    - PostgreSQL must be available to all 3 availability zones.
    - PostgreSQL version 14.11+
    - Kubernetes Secret with PostgreSQL credentials

- **AWS-only**

- A NLB and Kloudfuse DNS mapping
  - Use a NLB (Network Load Balancer) for Kloudfuse DNS mapping. Elastic IPs are AZ-specific, the DNS for the Kloudfuse endpoint must be a CNAME pointing to the NLB DNS to ensure traffic routing during a zone failure.

## Step 1: Configure Helm Values

In the `custom_values.yaml`, configure the following fields:

```yaml
global:
  cloudProvider: <aws | gcp | azure>
  numNodes: <Total number of nodes across all zones>
  multiAzDeployment:
    enabled: true
  configDB:
    host: <Postgres host for configDB>
  orchestratorDB:
    host: <Postgres host for orchestratorDB>

installKfusePgCredentials: false
```

This configuration ensures Kloudfuse uses external PostgreSQL and skips deploying its own credentials secret.

## Step 2: Disable Embedded PostgreSQL

To use cloud-managed PostgreSQL, disable the internal PostgreSQL services:

```yaml
ingester:
  postgresql:
    enabled: false

kfuse-configdb:
  enabled: false
```

## Step 3: Automatic Scaling and Anti-Affinity Rules

When `multiAzDeployment.enabled` is set to `true`, Kloudfuse will automatically:

- Adjusts `replicaCount` for services based on `global.numNodes`
- Applies pod anti-affinity rules to distribute replicas across availability zones

Do **not** manually set `replicaCount` for most services.

## Service Behavior in Multi-AZ Mode

These services will auto-scale based on the number of nodes that are being used in the cluster.

- advance-functions-service
- events-query-service
- ingester
- kafka
- llm-evaluation-service
- llm-query-service
- logs-parser
- logs-query-service
- logs-transformer
- kfuse-observability-agent
- kfuse-vector
- metrics-transformer
- pinot
- query-service
- rum-query-service
- trace-query-service
- trace-transformer
- zapper

Services that, by default, will have 3 replicas (1 per zone)

- beffe
- ingress-nginx
- kafka zookeeper
- kfuse-profiling-server (requires cloud storage)
- pinot zookeeper
- redis
- ui

Services that always use 1 replica

These components remain single-instance. On zone failure, Kubernetes reschedules the pod to a healthy node in another zone.

- az-service
- config-mgmt-service
- grafana
- hydration-service
- kfuse-auth
- kfuse-cloud-exporter (scraper and exporters)
- kfuse-profiler
- kfuse-saml
- rule-manager
- user-mgmt-service

## Additional Notes

- Replication factors will be automatically managed for Kafka and Pinot
- Do not override `replicaCount` unless given specific guidance to do so
- `global.numNodes` is used to influence replica scaling.

## References

- [Set PostgreSQL Credentials](https://docs.kloudfuse.com/platform/4.0.2/administration/authentication/database-manage-credentials/)