Configure Envoy Ingress

Overview

Envoy Gateway implements the Kubernetes Gateway API and replaces ingress-nginx as the ingress controller for Kloudfuse. This guide covers new cluster installation, zero-downtime migration from ingress-nginx, rollback procedures, and post-uninstall cleanup.

X-Request-ID Forwarding

The Kloudfuse chart configures Envoy to forward a request-correlation header to Kloudfuse. The default values already enable this, so a standard install needs no action — but if you customize the ClientTrafficPolicy or run your own Envoy Gateway, keep the setting below in place.

envoy-gateway:
  clientTrafficPolicy:
    headers:
      preserveXRequestID: true

Prerequisites

Hardware Requirements

• Kloudfuse version 4.0.0 or later — Envoy Gateway is not compatible with earlier versions
• Kubernetes cluster version 1.27 or later
• kubectl configured with cluster access
• Helm 3.x installed

Install Envoy CRDs

Install the CRDs before running helm install or helm upgrade. The version must match the Envoy Gateway version used by the Kloudfuse chart:

helm show crds oci://docker.io/envoyproxy/gateway-helm \
  --version <VERSION> \
  | kubectl apply --server-side --force-conflicts -f -

Install or Upgrade cert-manager

cert-manager v1.14+ with Gateway API support is required for automatic TLS certificate issuance.

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace <cert-manager-namespace> \ (1)
  --version v1.17.1 \
  --set crds.enabled=true \
  --set config.enableGatewayAPI=true

Add the following TLS section to your custom_values.yaml to enable automatic certificate issuance:

tls:
  enabled: true
  host: <your-hostname>
  email: <your-email>
  clusterIssuer: <namespace>-letsencrypt-prod
  createClusterIssuer: true

New Install

For new clusters where ingress-nginx has never been installed, enable envoy-gateway and disable ingress-nginx:

envoy-gateway:
  enabled: true
  installGatewayRoutes: true
  envoyService:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
      service.beta.kubernetes.io/aws-load-balancer-eip-allocations: <YOUR_EIP_ALLOC_IDS>
    patch:
      externalTrafficPolicy: Local
    external:
      enabled: true
    internal:
      enabled: true
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"

ingress-nginx:
  enabled: false
  installIngressRules: false

Separate Internal and External Hosts

To route ingest traffic to an internal load balancer and query traffic to an external load balancer, configure the tls.internalIngest and tls.externalQuery fields:

tls:
  host: <PRIMARY DNS HOST>
  internalIngest:
    hosts:
      - <INTERNAL INGEST DNS HOST>
    secretName: "<TLS secret for ingest hosts, or empty for ACM>"
    hostOnly: false
  externalQuery:
    hosts:
      - <EXTERNAL QUERY DNS HOST>
    secretName: "<TLS secret for query hosts, or empty for ACM>"
    hostOnly: false

Upgrade from Nginx to Envoy

This procedure migrates an existing Kloudfuse installation from ingress-nginx to envoy-gateway without recreating load balancers or changing IPs or DNS. This migration will result in zero-downtime.

Step 1: Enable Envoy

1. Add the envoy-gateway section and the migration flag to your custom_values.yaml:

global:
     envoyMigration:
       enabled: true
       external: false
       internal: false

envoy-gateway:
     enabled: true
     installGatewayRoutes: true
     envoyService:
       external:
         enabled: true
       # Enable if you have an internal LB (1)
       # internal:
       #   enabled: true

2. If your existing nginx configuration uses separate internal/external hosts, migrate the flat TLS fields to the new nested format. Add the tls.internalIngest and tls.externalQuery fields alongside your existing flat fields.

3. Run helm upgrade:

helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
     -n kfuse \
     --version <VERSION> \ (1)
     -f custom-values.yaml

Step 2: Switch Traffic to Envoy

1. Change global.envoyMigration.external to true in your custom_values.yaml.

global:
     envoyMigration:
       enabled: true
       external: true
       # Set to true if you have an internal Nginx LB to switch (1)
       internal: false

2. Run helm upgrade:

Step 3: Disable Nginx

1. Set ingress-nginx.enabled to false in your custom_values.yaml:

 ing... 2. Run `helm upgrade`:

### Verification

After completion, verify:

```bash
# Envoy pods should be running
kubectl get pods -n <namespace> | grep envoy

# Gateway should be Programmed: False
kubectl get gateway -n <namespace>

# Nginx should still be serving (302 redirect to login)
curl -sk https://your-hostname/ -o /dev/null -w "%{http_code}"

# Nginx LB should have keep annotation
kubectl get svc kfuse-ingress-nginx-controller -n <namespace> \
  -o jsonpath='{.metadata.annotations.helm\.sh/resource-policy}'

Rollback to Nginx

To revert from Envoy back to Nginx:

1. Only if the original Nginx LB Service still exists (i.e. you ran the in-place migration), remove those annotations so Helm can manage the Service again:

export NAMESPACE="<namespace>"
kubectl annotate svc kfuse-ingress-nginx-controller \
     helm.sh/resource-policy- -n $NAMESPACE

2. Update the custom_values.yaml to disable Envoy and re-enable Nginx with the full controller config that was in place before the Envoy migration.

global:
  envoyMigration:
    enabled: false
    external: false
    internal: false

envoy-gateway:
  enabled: false
  installGatewayRoutes: false

ingress-nginx:
  enabled: true
  installIngressRules: true
  controller:
    ingressClassResource:
      enabled: true
      name: kfuse-ingress
    ingressClass: kfuse-ingress
    watchIngressWithoutClass: false
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
        service.beta.kubernetes.io/aws-load-balancer-eip-allocations: <YOUR_EIP_ALLOC_IDS>  (1)
      externalTrafficPolicy: Local
      external:
        enabled: true
      internal:
        enabled: true
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-internal: "true"

3. Run helm upgrade:

4. Clean up orphaned Envoy resources:

export NAMESPACE="<namespace>"

# Delete controller-managed proxy Deployments and Services
kubectl delete deploy -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl delete svc -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE

# Remove GatewayClass finalizers and delete them
for gc in $(kubectl get gatewayclass -o name | grep "$NAMESPACE"); do
     kubectl patch "$gc" --type=merge -p '{"metadata":{"finalizers":[]}}'
     kubectl delete "$gc"
done

5. Verify the rollback:

export NAMESPACE="<namespace>"

# Nginx controller pod should be running; no Envoy proxy pods should remain
kubectl get pods -n $NAMESPACE | grep -E "nginx|envoy"

# Nginx LB selector should point back to Nginx pods; targetPorts should be 80/443
kubectl get svc kfuse-ingress-nginx-controller -n $NAMESPACE \
     -o jsonpath='selector: {.spec.selector}{"\n"}targetPorts: {.spec.ports[*].targetPort}{"\n"}'

# HTTPS should return 302 (Nginx login redirect) — not 401
curl -sk https://your-hostname/ -o /dev/null -w "%{http_code}\n"

Uninstall

When running helm delete (or helm uninstall), the envoy-gateway controller is removed but the resources it created at runtime are not deleted by Helm. These orphaned resources must be cleaned up manually.

Cleanup Steps

After helm delete kfuse -n <namespace>:

# 1. Delete controller-managed proxy Deployments and Services
export NAMESPACE="<namespace>"
kubectl delete deploy -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl delete svc -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE

# 2. Remove GatewayClass finalizers and delete them (cluster-scoped)
for gc in $(kubectl get gatewayclass -o name | grep "$NAMESPACE"); do
  kubectl patch "$gc" --type=merge -p '{"metadata":{"finalizers":[]}}'
  kubectl delete "$gc"
done

Troubleshooting

Kubernetes Gateway Programmed

If you are doing an upgrade, the envoy-gateway will says Programmed is False. you’re reusing the Nginx LB instead of creating a new one — so the Gateway object never gets an address assigned to it directly. This is expected and harmless.

Failed to install CRDS

Error Message:

Error: failed to install CRD crds/gatewayapi-crds.yaml: 10 errors occurred:
        * customresourcedefinitions.apiextensions.k8s.io "gatewayclasses.gateway.networking.k8s.io" is forbidden: ValidatingAdmissionPolicy 'safe-upgrades.gateway.networking.k8s.io' with binding 'safe-upgrades.gateway.networking.k8s.io' denied request: Installing CRDs with version before v1.5.0 is prohibited by default. Uninstall ValidatingAdmissionPolicy safe-upgrades.gateway.networking.k8s.io to install older versions.

AWS NLB Hairpin

On AWS, cert-manager’s HTTP-01 self-check may fail during initial certificate issuance because pods cannot reach the external NLB from inside the cluster. This only affects the first-time cert setup — not ongoing operations. Not applicable to GCP.

export NAMESPACE="kfuse"
export HOSTNAME="observe.example.com"

ENVOY_IP=$(kubectl get svc envoy -n $NAMESPACE \
  -o jsonpath='{.spec.clusterIP}')

helm upgrade cert-manager jetstack/cert-manager \
  --namespace $NAMESPACE \
  --set "hostAliases[0].ip=$ENVOY_IP" \
  --set "hostAliases[0].hostnames[0]=$HOSTNAME"

Multi-Tenant Clusters

When multiple Kloudfuse instances share the same cluster, each namespace must scope its envoy-gateway controller:

envoy-gateway:
  config:
    envoyGateway:
      provider:
        kubernetes:
          watch:
            type: Namespaces
            namespaces:
              - <your-namespace>

References

• Envoy Gateway Documentation
• Kubernetes Gateway API