# Configure Envoy Ingress

## Overview

Envoy Gateway implements the Kubernetes Gateway API and replaces `ingress-nginx` as the ingress controller for Kloudfuse. This guide covers new cluster installation, zero-downtime migration from `ingress-nginx`, rollback procedures, and post-uninstall cleanup.

## Prerequisites

### Hardware Requirements

- **Kloudfuse version 4.0.0 or later** — Envoy Gateway is not compatible with earlier versions
- Kubernetes cluster version 1.27 or later
- `kubectl` configured with cluster access
- Helm 3.x installed

### Install Envoy CRDs

Install the CRDs before running `helm install` or `helm upgrade`. The version must match the Envoy Gateway version used by the Kloudfuse chart:

| Kloudfuse Version | Envoy Gateway CRD Version |
| --- | --- |
| 4.0.0 | v1.3.0 |
| 4.0.1+ | v1.7.1 |

```bash
helm show crds oci://docker.io/envoyproxy/gateway-helm \
  --version <VERSION> \
  | kubectl apply --server-side --force-conflicts -f -
```

|     |     |
| --- | --- |
|  | This command is idempotent — safe to run on clusters that already have the CRDs installed. The `--server-side` flag avoids annotation size limits on large CRDs. |

### Install or Upgrade cert-manager

cert-manager **v1.14+** with Gateway API support is required for automatic TLS certificate issuance.

|     |     |
| --- | --- |
|  | cert-manager is cluster-wide but runs in a single namespace. If cert-manager is already installed on your cluster, upgrade the existing installation in its current namespace rather than installing a second instance. |

```bash
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace <cert-manager-namespace> \ (1)
  --version v1.17.1 \
  --set crds.enabled=true \
  --set config.enableGatewayAPI=true
```

|     |     |
| --- | --- |
| **1** | Use the namespace where cert-manager is already installed (e.g., `cert-manager`), not the Kloudfuse namespace. Run `kubectl get pods -A | grep cert-manager` to find it. |

|     |     |
| --- | --- |
|  | The `config.enableGatewayAPI=true` flag is required. Without it, cert-manager only watches Ingress resources and will not create certificates for Gateway resources. |

Add the following TLS section to your `custom_values.yaml` to enable automatic certificate issuance:

```yaml
tls:
  enabled: true
  host: <your-hostname>
  email: <your-email>
  clusterIssuer: <namespace>-letsencrypt-prod
  createClusterIssuer: true
```

## New Install

For new clusters where `ingress-nginx` has never been installed, enable `envoy-gateway` and disable `ingress-nginx`:

- AWS (EKS)

- GCP (GKE)

```yaml
envoy-gateway:
  enabled: true
  installGatewayRoutes: true
  envoyService:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
      service.beta.kubernetes.io/aws-load-balancer-eip-allocations: <YOUR_EIP_ALLOC_IDS>
    patch:
      externalTrafficPolicy: Local
    external:
      enabled: true
    internal:
      enabled: true
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"

ingress-nginx:
  enabled: false
  installIngressRules: false
```

### Separate Internal and External Hosts

To route ingest traffic to an internal load balancer and query traffic to an external load balancer, configure the `tls.internalIngest` and `tls.externalQuery` fields:

```yaml
tls:
  host: <PRIMARY DNS HOST>
  internalIngest:
    hosts:
      - <INTERNAL INGEST DNS HOST>
    secretName: "<TLS secret for ingest hosts, or empty for ACM>"
    hostOnly: false
  externalQuery:
    hosts:
      - <EXTERNAL QUERY DNS HOST>
    secretName: "<TLS secret for query hosts, or empty for ACM>"
    hostOnly: false
```

| Field | Description |
| --- | --- |
| `tls.internalIngest.hosts` | DNS hostnames for internal ingestion routes (Gateway listeners and ingest HTTPRoutes). |
| `tls.internalIngest.secretName` | Kubernetes TLS secret for the ingest hosts. Leave empty when using ACM termination. |
| `tls.internalIngest.hostOnly` | If `true`, ingest routes only match `internalIngest.hosts` and exclude `tls.host`. |
| `tls.externalQuery.hosts` | DNS hostnames for external query/browser routes (Gateway listeners and query HTTPRoutes). |
| `tls.externalQuery.secretName` | Kubernetes TLS secret for the query hosts. Leave empty when using ACM termination. |
| `tls.externalQuery.hostOnly` | If `true`, query routes only match `externalQuery.hosts` and exclude `tls.host`. |

## Upgrade from Nginx to Envoy

This procedure migrates an existing Kloudfuse installation from `ingress-nginx` to `envoy-gateway` without recreating load balancers or changing IPs or DNS. This migration will result in zero-downtime.

The Nginx LoadBalancer Service is preserved throughout the migration. Instead of deleting it and creating a new Envoy LB, the Nginx service’s selector is repointed to Envoy proxy pods. The same NLB/LB keeps the same IPs and DNS.

### Step 1: Enable Envoy

1. Add the `envoy-gateway` section and the migration flag to your `custom_values.yaml`:
   
```yaml
global:
     envoyMigration:
       enabled: true
       external: false
       internal: false

envoy-gateway:
     enabled: true
     installGatewayRoutes: true
     envoyService:
       external:
         enabled: true
       # Enable if you have an internal LB (1)
       # internal:
       #   enabled: true
```

2. If your existing nginx configuration uses separate internal/external hosts, migrate the flat TLS fields to the new nested format. Add the `tls.internalIngest` and `tls.externalQuery` fields alongside your existing flat fields (which nginx still reads until it is disabled):

| Nginx (existing) | Envoy (add these) |
| --- | --- |
| `tls.ingestInternalHosts: [host]` | `tls.internalIngest.hosts: [host]` |
| `tls.ingestInternalSecretName: secret` | `tls.internalIngest.secretName: secret` |
| `tls.ingestInternalHostOnly: true` | `tls.internalIngest.hostOnly: true` |
| `tls.queryHosts: [host]` | `tls.externalQuery.hosts: [host]` |
| `tls.queryHostSecretName: secret` | `tls.externalQuery.secretName: secret` |
| `tls.queryHostOnly: true` | `tls.externalQuery.hostOnly: true` |

## Verification

After completing the migration steps, verify:

```bash
export NAMESPACE="<namespace>"

# Nginx controller pod should be gone; Envoy pods should be running
kubectl get pods -n $NAMESPACE | grep -E "nginx|envoy"

# Nginx LB service should still exist; selector and targetPorts should point to Envoy
kubectl get svc kfuse-ingress-nginx-controller -n $NAMESPACE \
  -o jsonpath='selector: {.spec.selector}{"\n"}targetPorts: {.spec.ports[*].targetPort}{"\n"}'

# Gateway should show Programmed: False
kubectl get gateway -n $NAMESPACE

# All HTTPRoutes should show Accepted and Resolved
kubectl get httproute -n $NAMESPACE

# HTTPS should return 401 (Envoy auth challenge) — not 302 (Nginx redirect)
curl -sk https://your-hostname/ -o /dev/null -w "%{http_code}\n"
```

## Uninstall

When running `helm delete` (or `helm uninstall`), the envoy-gateway controller is removed but the resources it created at runtime are **not deleted by Helm**. These orphaned resources must be cleaned up manually.

### Cleanup Steps

After `helm delete kfuse -n <namespace>`:

```bash
# 1. Delete controller-managed proxy Deployments and Services
export NAMESPACE="<namespace>"
kubectl delete deploy -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl delete svc -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl delete configmap -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE

# 2. Remove GatewayClass finalizers and delete them (cluster-scoped)
for gc in $(kubectl get gatewayclass -o name | grep "$NAMESPACE"); do
  kubectl patch "$gc" --type=merge -p '{"metadata":{"finalizers":[]}}'
  kubectl delete "$gc"
done

# 3. Verify no orphaned resources remain
kubectl get all -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl get gatewayclass | grep $NAMESPACE
```

**Shared clusters** — Do **NOT** delete Gateway API or Envoy Gateway CRDs on clusters where other namespaces also use envoy-gateway. CRDs are cluster-scoped; deleting them removes all Gateway, HTTPRoute, and SecurityPolicy resources across **all** namespaces. Only delete the GatewayClass resources for your own namespace.

### Troubleshooting

#### Kubernetes Gateway Programmed

If you are doing an upgrade, the envoy-gateway will says Programmed is False. you’re reusing the Nginx LB instead of creating a new one — so the Gateway object never gets an address assigned to it directly. This is expected and harmless.

### Multi-Tenant Clusters

When multiple Kloudfuse instances share the same cluster, each namespace must scope its `envoy-gateway` controller:

```yaml
envoy-gateway:
  config:
    envoyGateway:
      provider:
        kubernetes:
          watch:
            type: Namespaces
            namespaces:
              - <your-namespace>
```

The chart automatically uses a namespace-specific `controllerName` to prevent cross-namespace interference.

## References

- [Envoy Gateway Documentation](https://gateway.envoyproxy.io/docs/)
- [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io/)