Node Affinity and Tolerations

Node affinity and tolerations control which Kubernetes nodes Kloudfuse pods are allowed to schedule onto. Use these settings to constrain the entire Kloudfuse deployment to a specific node group, availability zone, or set of nodes that match your infrastructure requirements.

Scheduling Concepts

Node Affinity

Node affinity is a set of rules that the Kubernetes scheduler uses when placing pods. It extends nodeSelector with richer expression syntax and two enforcement modes:

Node affinity expressions use matchExpressions to filter nodes by label:

Tolerations

Nodes can be tainted to repel pods that do not explicitly tolerate the taint. A taint has three components: a key, a value, and an effect. The two most common effects are:

• NoSchedule — pods without a matching toleration are not scheduled onto the node.

• NoExecute — existing pods without a matching toleration are evicted, and new pods are not scheduled.

A toleration in the pod spec matches a taint and allows the pod to schedule onto that node despite the taint. When combined with node affinity, you can both attract pods to specific nodes and permit them to schedule onto tainted nodes in one configuration.

Managing Node Labels/Taints

Use the tabs below to apply labels and taints via your cloud provider CLI, or directly with kubectl on individual nodes.

• GCP (GKE)
• AWS (EKS)
• Azure (AKS)
• kubectl

# Label the node pool
gcloud container node-pools update <node-pool-name> \
  --cluster=<cluster-name> \
  --node-labels=ng_label=az2

# Taint the node pool
gcloud container node-pools update <node-pool-name> \
  --cluster=<cluster-name> \
  --node-taints=ng_taint=az2:NoSchedule

Configure labels and taints when creating the managed node group, or apply them to existing nodes:

# Label nodes in the node group
kubectl label nodes -l eks.amazonaws.com/nodegroup=<ng-name> ng_label=az2

# Taint nodes in the node group
kubectl taint nodes -l eks.amazonaws.com/nodegroup=<ng-name> ng_taint=az2:NoSchedule

Apply labels and taints directly to individual nodes using kubectl:

# Add a label to a node
kubectl label node <node-name> ng_label=az2

# Remove a label from a node (append - to the key)
kubectl label node <node-name> ng_label-

# Add a NoSchedule taint to a node
kubectl taint node <node-name> ng_taint=az2:NoSchedule

Kloudfuse

Global Affinity

Set global.affinity to a standard Kubernetes affinity object. The following example uses a hard node affinity requirement:

global:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: <node-label-key>
                operator: In
                values:
                  - <node-label-value>

Global Tolerations

Set global.tolerations to a list of standard Kubernetes toleration objects:

global:
  tolerations:
    - key: "<taint-key>"
      operator: "Equal"
      value: "<taint-value>"
      effect: "NoSchedule"

Per-Service Overrides

If a service has local affinity or tolerations values set in custom_values.yaml, those values take priority over the global settings. Use this to place an individual component on a different set of nodes:

redis:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: ng_label
                operator: In
                values:
                  - memory-optimized
  tolerations:
    - key: "workload"
      operator: "Equal"
      value: "memory-optimized"
      effect: "NoSchedule"

Example:

A common use case is deploying Kloudfuse onto a dedicated node group within a larger shared cluster, where the node group is labeled and tainted to prevent general workloads from running there.

In the example below, all Kloudfuse pods are constrained to nodes labeled ng_label=az2. The toleration allows pods to schedule onto nodes that carry the ng_taint=az2:NoSchedule taint:

global:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: ng_label
                operator: In
                values:
                  - az2
  tolerations:
    - key: "ng_taint"
      operator: "Equal"
      value: "az2"
      effect: "NoSchedule"

Before applying this configuration, ensure that the target nodes are labeled and tainted accordingly:

Helm Validation

The Kloudfuse Helm chart runs a pre-install validation that inspects the live cluster to determine how many nodes are available to Kloudfuse. This validation drives replica sizing warnings and pre-flight checks. Its evaluation of global.affinity has two limitations.

Only requiredDuringSchedulingIgnoredDuringExecution is evaluated. The validation parses nodeSelectorTerms under requiredDuringSchedulingIgnoredDuringExecution to filter the node list. If you set preferredDuringSchedulingIgnoredDuringExecution instead, the entire affinity block is skipped and all cluster nodes are counted as available Kloudfuse nodes. The resulting replica sizing warnings may be inaccurate. Use requiredDuringSchedulingIgnoredDuringExecution whenever possible to ensure the validation reflects your actual node topology.

Only In and NotIn operators are evaluated. The validation evaluates matchExpressions entries with the In and NotIn operators. Entries using Exists, DoesNotExist, Gt, or Lt are silently ignored — nodes are not filtered based on those expressions, so the validation may again over-count available nodes.

Neither limitation affects runtime scheduling. Kubernetes evaluates all affinity modes and operators correctly when placing pods; only the Helm pre-install validation is affected.

Node Pool Separation

global.affinity and global.tolerations apply uniformly to every Kloudfuse pod. They are the right choice when you want to contain the entire Kloudfuse deployment within a specific node group or availability zone.

For splitting Kloudfuse workloads across multiple dedicated node pools by workload type (ingestion, query, and control plane), use Workload Isolation instead.

GKE Limitation: kube-system Pods and Node Taints

On GKE, kube-system pods (such as kube-dns, metrics-server, and GKE add-ons) do not carry tolerations for user-defined taints. If you taint every node in the cluster with a custom NoSchedule taint, these system pods cannot schedule and will be evicted from their current nodes, potentially breaking cluster DNS and monitoring.

To avoid this, ensure at least one node pool in your cluster remains untainted so that kube-system pods have nodes available to run on.