Configure Envoy Ingress

Overview

Envoy Gateway implements the Kubernetes Gateway API and replaces ingress-nginx as the ingress controller for Kloudfuse. This guide covers new cluster installation, zero-downtime migration from ingress-nginx, rollback procedures, and post-uninstall cleanup.

Prerequisites

Hardware Requirements

• Kloudfuse version 4.0.0 or later — Envoy Gateway is not compatible with earlier versions
• Kubernetes cluster version 1.27 or later
• kubectl configured with cluster access
• Helm 3.x installed

Install Envoy CRDs

Install the CRDs before running helm install or helm upgrade. The version must match the Envoy Gateway version used by the Kloudfuse chart:

helm show crds oci://docker.io/envoyproxy/gateway-helm \
  --version <VERSION> \
  | kubectl apply --server-side --force-conflicts -f -

Install or Upgrade cert-manager

cert-manager v1.14+ with Gateway API support is required for automatic TLS certificate issuance.

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace <cert-manager-namespace> \ (1)
  --version v1.17.1 \
  --set crds.enabled=true \
  --set config.enableGatewayAPI=true

Add the following TLS section to your custom_values.yaml to enable automatic certificate issuance:

tls:
  enabled: true
  host: <your-hostname>
  email: <your-email>
  clusterIssuer: <namespace>-letsencrypt-prod
  createClusterIssuer: true

New Install

For new clusters where ingress-nginx has never been installed, enable envoy-gateway and disable ingress-nginx:

• AWS (EKS)

• GCP (GKE)

envoy-gateway:
  enabled: true
  installGatewayRoutes: true
  envoyService:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
      service.beta.kubernetes.io/aws-load-balancer-eip-allocations: <YOUR_EIP_ALLOC_IDS>
    patch:
      externalTrafficPolicy: Local
    external:
      enabled: true
    internal:
      enabled: true
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"

ingress-nginx:
  enabled: false
  installIngressRules: false

Separate Internal and External Hosts

To route ingest traffic to an internal load balancer and query traffic to an external load balancer, configure the tls.internalIngest and tls.externalQuery fields:

tls:
  host: <PRIMARY DNS HOST>
  internalIngest:
    hosts:
      - <INTERNAL INGEST DNS HOST>
    secretName: "<TLS secret for ingest hosts, or empty for ACM>"
    hostOnly: false
  externalQuery:
    hosts:
      - <EXTERNAL QUERY DNS HOST>
    secretName: "<TLS secret for query hosts, or empty for ACM>"
    hostOnly: false

Upgrade from Nginx to Envoy

This procedure migrates an existing Kloudfuse installation from ingress-nginx to envoy-gateway without recreating load balancers or changing IPs or DNS. This migration will result in zero-downtime.

The Nginx LoadBalancer Service is preserved throughout the migration. Instead of deleting it and creating a new Envoy LB, the Nginx service’s selector is repointed to Envoy proxy pods. The same NLB/LB keeps the same IPs and DNS.

Step 1: Enable Envoy

1. Add the envoy-gateway section and the migration flag to your custom_values.yaml:

global:
     envoyMigration:
       enabled: true
       external: false
       internal: false

envoy-gateway:
     enabled: true
     installGatewayRoutes: true
     envoyService:
       external:
         enabled: true
       # Enable if you have an internal LB (1)
       # internal:
       #   enabled: true

2. If your existing nginx configuration uses separate internal/external hosts, migrate the flat TLS fields to the new nested format. Add the tls.internalIngest and tls.externalQuery fields alongside your existing flat fields (which nginx still reads until it is disabled):

Verification

After completing the migration steps, verify:

export NAMESPACE="<namespace>"

# Nginx controller pod should be gone; Envoy pods should be running
kubectl get pods -n $NAMESPACE | grep -E "nginx|envoy"

# Nginx LB service should still exist; selector and targetPorts should point to Envoy
kubectl get svc kfuse-ingress-nginx-controller -n $NAMESPACE \
  -o jsonpath='selector: {.spec.selector}{"\n"}targetPorts: {.spec.ports[*].targetPort}{"\n"}'

# Gateway should show Programmed: False
kubectl get gateway -n $NAMESPACE

# All HTTPRoutes should show Accepted and Resolved
kubectl get httproute -n $NAMESPACE

# HTTPS should return 401 (Envoy auth challenge) — not 302 (Nginx redirect)
curl -sk https://your-hostname/ -o /dev/null -w "%{http_code}\n"

Uninstall

When running helm delete (or helm uninstall), the envoy-gateway controller is removed but the resources it created at runtime are not deleted by Helm. These orphaned resources must be cleaned up manually.

Cleanup Steps

After helm delete kfuse -n <namespace>:

# 1. Delete controller-managed proxy Deployments and Services
export NAMESPACE="<namespace>"
kubectl delete deploy -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl delete svc -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl delete configmap -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE

# 2. Remove GatewayClass finalizers and delete them (cluster-scoped)
for gc in $(kubectl get gatewayclass -o name | grep "$NAMESPACE"); do
  kubectl patch "$gc" --type=merge -p '{"metadata":{"finalizers":[]}}'
  kubectl delete "$gc"
done

# 3. Verify no orphaned resources remain
kubectl get all -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl get gatewayclass | grep $NAMESPACE

Shared clusters — Do NOT delete Gateway API or Envoy Gateway CRDs on clusters where other namespaces also use envoy-gateway. CRDs are cluster-scoped; deleting them removes all Gateway, HTTPRoute, and SecurityPolicy resources across all namespaces. Only delete the GatewayClass resources for your own namespace.

Troubleshooting

Kubernetes Gateway Programmed

If you are doing an upgrade, the envoy-gateway will says Programmed is False. you’re reusing the Nginx LB instead of creating a new one — so the Gateway object never gets an address assigned to it directly. This is expected and harmless.

Multi-Tenant Clusters

When multiple Kloudfuse instances share the same cluster, each namespace must scope its envoy-gateway controller:

envoy-gateway:
  config:
    envoyGateway:
      provider:
        kubernetes:
          watch:
            type: Namespaces
            namespaces:
              - <your-namespace>

The chart automatically uses a namespace-specific controllerName to prevent cross-namespace interference.

References

• Envoy Gateway Documentation
• Kubernetes Gateway API