# Role Isolation

Role Isolation deploys Kloudfuse services onto dedicated Kubernetes node pools based on the role each service plays in the data pipeline — ingestion (write path), query (read path), or control plane. Each role runs on its own pool, providing resource isolation, improved stability, and independent scaling.

In this context, a **role** refers to the functional area of the Kloudfuse pipeline that a service implements. It is unrelated to Kubernetes RBAC roles or end-user roles. Every Kloudfuse service has a default role assignment (its `kfRole` field), and the `global.kfRoles` configuration maps each role to a node pool.

Role Isolation is recommended for any production deployment where ingestion load or query load could affect overall cluster stability. It is the first step in a scaled-out topology.

|     |     |
| --- | --- |
|  | Role Isolation and Stream Isolation are compatible and can be enabled together. When both are active, Pinot pods receive node affinity rules from both — one for role-based placement (`kfRoles`) and one for stream-specific placement (`kf_stream`). |

## Benefits

- **Resource isolation** — Ingestion spikes do not impact query performance, and vice-versa
- **Independent scaling** — Scale ingestion, query, and control plane node pools separately based on workload
- **Improved stability** — Noisy-neighbor effects are eliminated between write and read paths
- **Cost optimization** — Right-size node pools for each workload type

## Overview

Kloudfuse services are grouped into three roles:

| Role | Description |
| --- | --- |
| `ingestion` | Write path — data collection, parsing, and transformation |
| `query` | Read path — serving queries and the user interface |
| `control` | Control plane — cluster coordination and metadata |

Each service has a default role assignment configured via its `kfRole` field in `values.yaml`.

DaemonSet workloads (such as the kfuse-observability-agent) run on **all** node pools and automatically receive merged tolerations and affinity rules.

## Prerequisites

- A Kubernetes cluster with **three separate node pools** — one for each role (ingestion, query, control).
- Each node pool must have nodes **labeled** and **tainted** with the configured key and value.
- Each role must have at least one node (`numNodes` must be greater than 0 for all three roles).

## Step 1: Configure Node Pools

Create three node pools in your cloud provider. Each node pool must have a Kubernetes label and taint applied.

The default configuration uses the label key `kf_role` with values `ingestion`, `query`, and `control`.

- GCP
- AWS
- Azure

For each node pool:

1. Go to **Kubernetes Engine** \> **Clusters** \> your cluster > **Node Pools**.
2. Create or edit a node pool.
3. In the **Metadata** section, add a **Kubernetes label**:
   - Key: `kf_role`
   - Value: `ingestion`, `query`, or `control`
4. In the **Taints** section, add a taint:
   - Key: `kf_role`
   - Value: `ingestion`, `query`, or `control`
   - Effect: `NoSchedule`

## Step 2: Configure Helm Values

In your `custom_values.yaml`, enable node pool separation and configure the number of nodes in each pool:

```yaml
global:
  kfRoles:
    enabled: true
    ingestion:
      numNodes: 3  (1)
    query:
      numNodes: 3
    control:
      numNodes: 3
```

|     |     |
| --- | --- |
| **1** | `numNodes` — The number of nodes in this pool. Used for replica count calculations. Must be greater than 0. |

When `kfRoles.enabled` is `true`:

- Each service is automatically assigned to its designated node pool using node affinity rules.
- Tolerations are automatically generated so pods can schedule onto tainted nodes.
- `global.nodeSelector` is ignored (node affinity is used instead).
- `numNodes` per pool replaces `global.numNodes` for replica count calculations.

## Step 3: Install or Upgrade

Run the standard Helm install or upgrade command with your custom values:

```console
helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
  -n kfuse \
  --version <VERSION> \ (1)
  -f custom-values.yaml
```

|     |     |
| --- | --- |
| **1** | Replace `<VERSION>` with a valid Kloudfuse release value; use the most recent one. |

## Verify Node Placement

After deployment, verify that pods are scheduled onto the correct node pools:

```console
kubectl get pods -o wide -n kfuse
```

Check that:
- Ingestion services (kafka, ingester, logs-transformer, etc.) run on nodes labeled `kf_role=ingestion`.
- Query services (query-service, beffe, UI, etc.) run on nodes labeled `kf_role=query`.
- Control plane services (Kafka KRaft controller, ZooKeeper) run on nodes labeled `kf_role=control`.
- DaemonSets run on nodes across all pools.

You can list nodes by role:

```console
kubectl get nodes -l kf_role=ingestion
kubectl get nodes -l kf_role=query
kubectl get nodes -l kf_role=control
```

## Advanced Configuration

### Customizing node Key and Value

The `nodeKey` and `nodeValue` fields default to `kf_role` and `ingestion`/`query`/`control` respectively, but can be overridden for advanced use cases.

```yaml
global:
  kfRoles:
    enabled: true
    ingestion:
      nodeValue: "ingestion-az1"
      numNodes: 3
    query:
      nodeValue: "query-az1"
      numNodes: 3
    control:
      nodeValue: "control-az1"
      numNodes: 3
```

In this case, nodes must be labeled and tainted with the corresponding values (e.g., `kf_role=ingestion-az1`).

### Overriding a Service’s Role

Each service has a default role assignment. You can override the role for any individual service in your `custom_values.yaml`:

```yaml
redis:
  kfRole: query  (1)
```

|     |     |
| --- | --- |
| **1** | Moves Redis from its default `ingestion` pool to the `query` pool. |

### Per-Service nodeSelector, Affinity, and Tolerations

If a service has local `nodeSelector`, `affinity`, or `tolerations` configured, those take priority over the auto-generated values from `kfRoles`. This allows fine-grained placement control for specific services:

```yaml
query-service:
  nodeSelector:
    custom-label: "special-node"
  tolerations:
    - key: "custom-label"
      operator: "Equal"
      value: "special-node"
      effect: "NoSchedule"
```

### Combining Roles on a Shared Node Pool

If you only need to isolate one workload type, you can combine the remaining roles onto a single shared node pool by pointing their `nodeValue` to the same label. For example, to separate only query services while running ingestion and control plane services together:

1. Create **two** node pools instead of three:

- A shared pool for ingestion and control, labeled and tainted with `kf_role=ingestion-control`.
   - A dedicated pool for query, labeled and tainted with `kf_role=query`.

2. Configure `kfRoles` so that both `ingestion` and `control` use the same `nodeValue`:

```yaml
global:
     kfRoles:
       enabled: true
       ingestion:
         nodeValue: "ingestion-control"  (1)
         numNodes: 4
       query:
         numNodes: 3
       control:
         nodeValue: "ingestion-control"  (1)
         numNodes: 4  (2)
```

|     |     |
| --- | --- |
| **1** | Both `ingestion` and `control` point to the same `nodeValue`, so their pods schedule onto the same node pool. |
| **2** | Set `numNodes` to the same value for roles that share a node pool, reflecting the actual number of nodes in that shared pool. |

This pattern works for any combination. For example, to isolate only ingestion and combine query with control:

```yaml
global:
  kfRoles:
    enabled: true
    ingestion:
      numNodes: 4
    query:
      nodeValue: "query-control"
      numNodes: 3
    control:
      nodeValue: "query-control"
      numNodes: 3
```

### Combining with Stream Isolation

Node pool separation works alongside Pinot Stream Isolation. When both features are enabled, Pinot pods receive both `kfRole`-based affinity (for node pool placement) and `kf_stream`-based affinity (for stream-specific node placement).