# Configure Envoy Ingress

## Overview

Envoy Gateway implements the Kubernetes Gateway API and replaces `ingress-nginx` as the ingress controller for Kloudfuse. This guide covers new cluster installation, zero-downtime migration from `ingress-nginx`, rollback procedures, and post-uninstall cleanup.

## Prerequisites

### Hardware Requirements

- **Kloudfuse version 4.0.0 or later** — Envoy Gateway is not compatible with earlier versions

- Kubernetes cluster version 1.27 or later

- `kubectl` configured with cluster access

- Helm 3.x installed

### Install Envoy CRDs

Install the CRDs before running `helm install` or `helm upgrade`. The version must match the Envoy Gateway version used by the Kloudfuse chart:

| Kloudfuse Version | Envoy Gateway CRD Version |
| --- | --- |
| 4.0.0 | v1.3.0 |
| 4.0.1+ | v1.7.1 |

```bash
helm show crds oci://docker.io/envoyproxy/gateway-helm \
  --version <VERSION> \
  | kubectl apply --server-side --force-conflicts -f -
```

### Install or Upgrade cert-manager

cert-manager **v1.14+** with Gateway API support is required for automatic TLS certificate issuance.

```bash
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace <cert-manager-namespace> \
  --version v1.17.1 \
  --set crds.enabled=true \
  --set config.enableGatewayAPI=true
```

Add the following TLS section to your `custom_values.yaml` to enable automatic certificate issuance:

```yaml
tls:
  enabled: true
  host: <your-hostname>
  email: <your-email>
  clusterIssuer: <namespace>-letsencrypt-prod
  createClusterIssuer: true
```

## New Install

For new clusters where `ingress-nginx` has never been installed, enable `envoy-gateway` and disable `ingress-nginx`:

```yaml
envoy-gateway:
  enabled: true
  installGatewayRoutes: true
  envoyService:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
      service.beta.kubernetes.io/aws-load-balancer-eip-allocations: <YOUR_EIP_ALLOC_IDS>
    patch:
      externalTrafficPolicy: Local
    external:
      enabled: true
    internal:
      enabled: true
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"

ingress-nginx:
  enabled: false
  installIngressRules: false
```

To route ingest traffic to an internal load balancer and query traffic to an external load balancer, configure the `tls.internalIngest` and `tls.externalQuery` fields:

```yaml
tls:
  host: <PRIMARY DNS HOST>
  internalIngest:
    hosts:
      - <INTERNAL INGEST DNS HOST>
    secretName: "<TLS secret for ingest hosts, or empty for ACM>"
    hostOnly: false
  externalQuery:
    hosts:
      - <EXTERNAL QUERY DNS HOST>
    secretName: "<TLS secret for query hosts, or empty for ACM>"
    hostOnly: false
```

## Upgrade from Nginx to Envoy

This procedure migrates an existing Kloudfuse installation from `ingress-nginx` to `envoy-gateway` without recreating load balancers or changing IPs or DNS. This migration will result in zero-downtime.

| Step | Change Values | Traffic Served By | What Happens |
| --- | --- | --- | --- |
| **D0** (current) | None | Nginx | Starting state |
| **Step 1** (prepare) | Enable Envoy + `envoyMigration.enabled` | Nginx (unchanged) | Envoy starts alongside nginx. Nginx LB gets `resource-policy: keep` annotation. Envoy creates `ClusterIP` services (no new LB). |
| **Step 2** (switch) | Set `envoyMigration.external/internal: true` | **Envoy** | Nginx LB selector switches to Envoy pods. TargetPorts change to 10080/10443. Traffic flows through Envoy via the same LB. |
| **Step 3** (cleanup) | Set `ingress-nginx.enabled: false` | Envoy | Remove Nginx Controller. Nginx LB Service is kept (resource-policy annotation). |

### Step 1: Enable Envoy

1. Add the `envoy-gateway` section and the migration flag to your `custom_values.yaml`:

```yaml
global:
     envoyMigration:
       enabled: true
       external: false
       internal: false

envoy-gateway:
     enabled: true
     installGatewayRoutes: true
     envoyService:
       external:
         enabled: true
       # Enable if you have an internal LB (1)
       # internal:
       #   enabled: true
```

2. If your existing nginx configuration uses separate internal/external hosts, migrate the flat TLS fields to the new nested format.

### Step 2: Switch Traffic to Envoy

1. Change `global.envoyMigration.external` to `true` in your `custom_values.yaml`.

```yaml
global:
     envoyMigration:
       enabled: true
       external: true
       # Set to true if you have an internal Nginx LB to switch (1)
       internal: false
```

2. Run `helm upgrade`:

```console
helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
     -n kfuse \
     --version <VERSION> \
     -f custom-values.yaml
```

### Step 3: Disable Nginx

1. Set `ingress-nginx.enabled` to `false` in your `custom_values.yaml`:

```yaml
ningress-nginx:
     enabled: false
     installIngressRules: false
```

2. Run `helm upgrade`:

### Step 4: Verification

```bash
export NAMESPACE="<namespace>"

# Nginx controller pod should be gone; Envoy pods should be running
kubectl get pods -n $NAMESPACE | grep -E "nginx|envoy"

# Nginx LB service should still exist; selector and targetPorts should point to Envoy
kubectl get svc kfuse-ingress-nginx-controller -n $NAMESPACE \
  -o jsonpath='selector: {.spec.selector}{"\n"}targetPorts: {.spec.ports[*].targetPort}{"\n"}'

# Gateway should show Programmed: False
kubectl get gateway -n $NAMESPACE

# All HTTPRoutes should show Accepted and Resolved
kubectl get httproute -n $NAMESPACE

# HTTPS should return 401 (Envoy auth challenge) — not 302 (Nginx redirect)
curl -sk https://your-hostname/ -o /dev/null -w "%{http_code}\n"
```

## Rollback to Nginx

1. Remove the `helm.sh/resource-policy` annotations from the Nginx Load Balancer:

```bash
export NAMESPACE="<namespace>"
kubectl annotate svc kfuse-ingress-nginx-controller \
     helm.sh/resource-policy- -n $NAMESPACE
kubectl annotate svc kfuse-ingress-nginx-controller-internal \
     helm.sh/resource-policy- -n $NAMESPACE  # if internal LB exists
```

2. Update the `custom_values.yaml` to disable Envoy and re-enable Nginx:

```yaml
global:
     envoyMigration:
       enabled: false
       external: false
       internal: false

envoy-gateway:
     enabled: false
     installGatewayRoutes: false

ingress-nginx:
     enabled: true
     installIngressRules: true
```

3. Run `helm upgrade`:

4. Clean up orphaned Envoy resources:

```bash
export NAMESPACE="<namespace>"

# Delete controller-managed proxy Deployments and Services
kubectl delete deploy -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl delete svc -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE

# Remove GatewayClass finalizers and delete them
for gc in $(kubectl get gatewayclass -o name | grep "$NAMESPACE"); do
     kubectl patch "$gc" --type=merge -p '{"metadata":{"finalizers":[]}}'
     kubectl delete "$gc"
done
```

## Uninstall

When running `helm delete` (or `helm uninstall`), the envoy-gateway controller is removed but the resources it created at runtime are **not deleted by Helm**. These orphaned resources must be cleaned up manually.

### Cleanup Steps

After `helm delete kfuse -n <namespace>`:

```bash
# 1. Delete controller-managed proxy Deployments and Services
export NAMESPACE="<namespace>"
kubectl delete deploy -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl delete svc -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl delete configmap -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE

# 2. Remove GatewayClass finalizers and delete them (cluster-scoped)
for gc in $(kubectl get gatewayclass -o name | grep "$NAMESPACE"); do
  kubectl patch "$gc" --type=merge -p '{"metadata":{"finalizers":[]}}'
  kubectl delete "$gc"
done

# 3. Verify no orphaned resources remain
kubectl get all -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl get gatewayclass | grep $NAMESPACE
```

## Troubleshooting

### Kubernetes Gateway Programmed

If you are doing an upgrade, the envoy-gateway will says Programmed is False. you’re reusing the Nginx LB instead of creating a new one — so the Gateway object never gets an address assigned to it directly. This is expected and harmless.

### Failed to install CRDS

Error Message:

```
Error: failed to install CRD crds/gatewayapi-crds.yaml: 10 errors occurred:
        * customresourcedefinitions.apiextensions.k8s.io "gatewayclasses.gateway.networking.k8s.io" is forbidden: ValidatingAdmissionPolicy 'safe-upgrades.gateway.networking.k8s.io' with binding 'safe-upgrades.gateway.networking.k8s.io' denied request: Installing CRDs with version before v1.5.0 is prohibited by default. Uninstall ValidatingAdmissionPolicy safe-upgrades.gateway.networking.k8s.io to install older versions.
```

### AWS NLB Hairpin

On AWS, cert-manager’s HTTP-01 self-check may fail during **initial certificate issuance** because pods cannot reach the external NLB from inside the cluster. This only affects the first-time cert setup — not ongoing operations. Not applicable to GCP.

If the certificate takes too long to issue, add a `hostAliases` entry to cert-manager that maps your domain to the envoy ClusterIP inside the cluster:

```bash
export NAMESPACE="kfuse"
export HOSTNAME="observe.example.com"

ENVOY_IP=$(kubectl get svc envoy -n $NAMESPACE \
  -o jsonpath='{.spec.clusterIP}')

helm upgrade cert-manager jetstack/cert-manager \
  --namespace $NAMESPACE \
  --set "hostAliases[0].ip=$ENVOY_IP" \
  --set "hostAliases[0].hostnames[0]=$HOSTNAME"
```

### Multi-Tenant Clusters

When multiple Kloudfuse instances share the same cluster, each namespace must scope its `envoy-gateway` controller:

```yaml
envoy-gateway:
  config:
    envoyGateway:
      provider:
        kubernetes:
          watch:
            type: Namespaces
            namespaces:
              - <your-namespace>
```

## References

- [Envoy Gateway Documentation](https://gateway.envoyproxy.io/docs/)
- [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io/)