Configure Envoy Ingress

Overview

Envoy Gateway implements the Kubernetes Gateway API and replaces ingress-nginx as the ingress controller for Kloudfuse. This guide covers new cluster installation, zero-downtime migration from ingress-nginx, rollback procedures, and post-uninstall cleanup.

Prerequisites

Hardware Requirements

• Kloudfuse version 4.0.0 or later — Envoy Gateway is not compatible with earlier versions

• Kubernetes cluster version 1.27 or later

• kubectl configured with cluster access

• Helm 3.x installed

Install Envoy CRDs

Install the CRDs before running helm install or helm upgrade. The version must match the Envoy Gateway version used by the Kloudfuse chart:

helm show crds oci://docker.io/envoyproxy/gateway-helm \
  --version <VERSION> \
  | kubectl apply --server-side --force-conflicts -f -

Install or Upgrade cert-manager

cert-manager v1.14+ with Gateway API support is required for automatic TLS certificate issuance.

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace <cert-manager-namespace> \
  --version v1.17.1 \
  --set crds.enabled=true \
  --set config.enableGatewayAPI=true

Add the following TLS section to your custom_values.yaml to enable automatic certificate issuance:

tls:
  enabled: true
  host: <your-hostname>
  email: <your-email>
  clusterIssuer: <namespace>-letsencrypt-prod
  createClusterIssuer: true

New Install

For new clusters where ingress-nginx has never been installed, enable envoy-gateway and disable ingress-nginx:

envoy-gateway:
  enabled: true
  installGatewayRoutes: true
  envoyService:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
      service.beta.kubernetes.io/aws-load-balancer-eip-allocations: <YOUR_EIP_ALLOC_IDS>
    patch:
      externalTrafficPolicy: Local
    external:
      enabled: true
    internal:
      enabled: true
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"

ingress-nginx:
  enabled: false
  installIngressRules: false

To route ingest traffic to an internal load balancer and query traffic to an external load balancer, configure the tls.internalIngest and tls.externalQuery fields:

tls:
  host: <PRIMARY DNS HOST>
  internalIngest:
    hosts:
      - <INTERNAL INGEST DNS HOST>
    secretName: "<TLS secret for ingest hosts, or empty for ACM>"
    hostOnly: false
  externalQuery:
    hosts:
      - <EXTERNAL QUERY DNS HOST>
    secretName: "<TLS secret for query hosts, or empty for ACM>"
    hostOnly: false

Upgrade from Nginx to Envoy

This procedure migrates an existing Kloudfuse installation from ingress-nginx to envoy-gateway without recreating load balancers or changing IPs or DNS. This migration will result in zero-downtime.

Step 1: Enable Envoy

1. Add the envoy-gateway section and the migration flag to your custom_values.yaml:

global:
     envoyMigration:
       enabled: true
       external: false
       internal: false

envoy-gateway:
     enabled: true
     installGatewayRoutes: true
     envoyService:
       external:
         enabled: true
       # Enable if you have an internal LB (1)
       # internal:
       #   enabled: true

2. If your existing nginx configuration uses separate internal/external hosts, migrate the flat TLS fields to the new nested format.

Step 2: Switch Traffic to Envoy

1. Change global.envoyMigration.external to true in your custom_values.yaml.

global:
     envoyMigration:
       enabled: true
       external: true
       # Set to true if you have an internal Nginx LB to switch (1)
       internal: false

2. Run helm upgrade:

helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
     -n kfuse \
     --version <VERSION> \
     -f custom-values.yaml

Step 3: Disable Nginx

1. Set ingress-nginx.enabled to false in your custom_values.yaml:

ningress-nginx:
     enabled: false
     installIngressRules: false

2. Run helm upgrade:

Step 4: Verification

export NAMESPACE="<namespace>"

# Nginx controller pod should be gone; Envoy pods should be running
kubectl get pods -n $NAMESPACE | grep -E "nginx|envoy"

# Nginx LB service should still exist; selector and targetPorts should point to Envoy
kubectl get svc kfuse-ingress-nginx-controller -n $NAMESPACE \
  -o jsonpath='selector: {.spec.selector}{"\n"}targetPorts: {.spec.ports[*].targetPort}{"\n"}'

# Gateway should show Programmed: False
kubectl get gateway -n $NAMESPACE

# All HTTPRoutes should show Accepted and Resolved
kubectl get httproute -n $NAMESPACE

# HTTPS should return 401 (Envoy auth challenge) — not 302 (Nginx redirect)
curl -sk https://your-hostname/ -o /dev/null -w "%{http_code}\n"

Rollback to Nginx

1. Remove the helm.sh/resource-policy annotations from the Nginx Load Balancer:

export NAMESPACE="<namespace>"
kubectl annotate svc kfuse-ingress-nginx-controller \
     helm.sh/resource-policy- -n $NAMESPACE
kubectl annotate svc kfuse-ingress-nginx-controller-internal \
     helm.sh/resource-policy- -n $NAMESPACE  # if internal LB exists

2. Update the custom_values.yaml to disable Envoy and re-enable Nginx:

global:
     envoyMigration:
       enabled: false
       external: false
       internal: false

envoy-gateway:
     enabled: false
     installGatewayRoutes: false

ingress-nginx:
     enabled: true
     installIngressRules: true

3. Run helm upgrade:

4. Clean up orphaned Envoy resources:

export NAMESPACE="<namespace>"

# Delete controller-managed proxy Deployments and Services
kubectl delete deploy -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl delete svc -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE

# Remove GatewayClass finalizers and delete them
for gc in $(kubectl get gatewayclass -o name | grep "$NAMESPACE"); do
     kubectl patch "$gc" --type=merge -p '{"metadata":{"finalizers":[]}}'
     kubectl delete "$gc"
done

Uninstall

When running helm delete (or helm uninstall), the envoy-gateway controller is removed but the resources it created at runtime are not deleted by Helm. These orphaned resources must be cleaned up manually.

Cleanup Steps

After helm delete kfuse -n <namespace>:

# 1. Delete controller-managed proxy Deployments and Services
export NAMESPACE="<namespace>"
kubectl delete deploy -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl delete svc -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl delete configmap -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE

# 2. Remove GatewayClass finalizers and delete them (cluster-scoped)
for gc in $(kubectl get gatewayclass -o name | grep "$NAMESPACE"); do
  kubectl patch "$gc" --type=merge -p '{"metadata":{"finalizers":[]}}'
  kubectl delete "$gc"
done

# 3. Verify no orphaned resources remain
kubectl get all -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl get gatewayclass | grep $NAMESPACE

Troubleshooting

Kubernetes Gateway Programmed

If you are doing an upgrade, the envoy-gateway will says Programmed is False. you’re reusing the Nginx LB instead of creating a new one — so the Gateway object never gets an address assigned to it directly. This is expected and harmless.

Failed to install CRDS

Error Message:

Error: failed to install CRD crds/gatewayapi-crds.yaml: 10 errors occurred:
        * customresourcedefinitions.apiextensions.k8s.io "gatewayclasses.gateway.networking.k8s.io" is forbidden: ValidatingAdmissionPolicy 'safe-upgrades.gateway.networking.k8s.io' with binding 'safe-upgrades.gateway.networking.k8s.io' denied request: Installing CRDs with version before v1.5.0 is prohibited by default. Uninstall ValidatingAdmissionPolicy safe-upgrades.gateway.networking.k8s.io to install older versions.

AWS NLB Hairpin

On AWS, cert-manager’s HTTP-01 self-check may fail during initial certificate issuance because pods cannot reach the external NLB from inside the cluster. This only affects the first-time cert setup — not ongoing operations. Not applicable to GCP.

If the certificate takes too long to issue, add a hostAliases entry to cert-manager that maps your domain to the envoy ClusterIP inside the cluster:

export NAMESPACE="kfuse"
export HOSTNAME="observe.example.com"

ENVOY_IP=$(kubectl get svc envoy -n $NAMESPACE \
  -o jsonpath='{.spec.clusterIP}')

helm upgrade cert-manager jetstack/cert-manager \
  --namespace $NAMESPACE \
  --set "hostAliases[0].ip=$ENVOY_IP" \
  --set "hostAliases[0].hostnames[0]=$HOSTNAME"

Multi-Tenant Clusters

When multiple Kloudfuse instances share the same cluster, each namespace must scope its envoy-gateway controller:

envoy-gateway:
  config:
    envoyGateway:
      provider:
        kubernetes:
          watch:
            type: Namespaces
            namespaces:
              - <your-namespace>

References

• Envoy Gateway Documentation
• Kubernetes Gateway API