# Relabel Rules During ingestion, the Kloudfuse platform supports the rewriting the labels for Metrics, Events, Logs, and Traces data. This page describes configuration of relabel rules for Metrics, Events, and Traces. For configuration of relabel rules for Logs, see [Parsing logs](https://docs.kloudfuse.com/platform/3.5.0/logs-parsing/). The configuration follows the Prometheus `relabel config` format, see [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) for more details. You can use helm chart values in the relevant `relabel_configs` section of the `custom_values.yaml` file. ## Relabel rules in the Ingester ```yaml ingester: config: aggregator: metric: # relabelConfigs - Configure relabel rules for metric labels. # Format follows Prometheus relabel_config syntax. # Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config relabelConfigs: [] events: # relabelConfigs - Configure relabel rules for events labels. # Format follows Prometheus relabel_config syntax. # Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config relabelConfigs: [] traces: # relabelConfigs - Configure relabel rules for traces labels. # Format follows Prometheus relabel_config syntax. # Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config relabelConfigs: [] ``` ## Common Example of Relabel Rules The examples below only show a single relabel rule for each scenario. Multiple rules can be provided, and they are evaluated in order. Specifically, each rule is evaluated against the current state of the labels after application of the previous rules. ### Add New Label Based on Current Label Value In a common scenario, you can enrich some metrics with new labels, depending on the existing label values. For example, for the label `aws_account` with value `123456`, you can add `aws_account_name` label with `AWS_account_name_for_123456` value by inserting the following section at the end of the `custom_values.yaml` file. ```yaml ingester: config: aggregator: metric: # relabelConfigs - Configure relabel rules for metric labels. # Format follows Prometheus relabel_config syntax. # Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config relabelConfigs: - ... - regex: "123456" replacement: AWS_account_name_for_123456 source_labels: - aws_account target_label: aws_account_name - ... ``` Run the helm upgrade to apply the changes. ### Replace Label Value Based on Current Label Value In a common scenario, you can update the label value, based on regex grouping. Consider the following example: A label `span_name` always comes with prefix `span_`. You can add a regex pattern that strips the prefix `span_`. Note that in the example below, the `replacement` value supports group references, e.g., `${1}`. ```yaml ingester: config: aggregator: metric: # relabelConfigs - Configure relabel rules for metric labels. # Format follows Prometheus relabel_config syntax. # Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config relabelConfigs: - ... - regex: "span_(.*)" replacement: ${1} source_labels: - span_name target_label: span_name - ... ``` Run the helm upgrade to apply the changes. ### Conditionally Replace Label Value By utilizing `source_labels` and `regex` you can conditionally replace the label value. Continuing with the example above, you can remove the prefix of `span_` only if the `service` label is `my-service`. By default the relabel rules concatenate the label values of the `source_labels` with the `;` separator. The separator used can be changed by explicitly providing the `separator` parameter. ```yaml ingester: config: aggregator: metric: # relabelConfigs - Configure relabel rules for metric labels. # Format follows Prometheus relabel_config syntax. # Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config relabelConfigs: - ... - regex: "my-service;span_(.*)" replacement: ${1} source_labels: - service - span_name target_label: span_name - ... ``` Run the helm upgrade to apply the changes. ### Add New Label Based on Multiple Labels Continuing with the example above, you can create a new label `service_span` by combining value of the `service` and `span_name` labels. You can also utilize regex grouping to extract a part of the value. In the example below, the `service_span` label is created by combining the value of the `service` label and a portion of the `span_name` label. ```yaml ingester: config: aggregator: metric: # relabelConfigs - Configure relabel rules for metric labels. # Format follows Prometheus relabel_config syntax. # Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config relabelConfigs: - ... - regex: "(.*);span_(.*)" replacement: ${1}-${2} source_labels: - service - span_name target_label: service_span - ... ``` Run the helm upgrade to apply the changes. ### Drop labels Labels can be dropped from the data by setting the `action` to `labeldrop` and setting the `regex` to a pattern that matches the label name. In the example below, any label that starts with `span_` will be dropped from the data. ```yaml ingester: config: aggregator: metric: # relabelConfigs - Configure relabel rules for metric labels. # Format follows Prometheus relabel_config syntax. # Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config relabelConfigs: - ... - regex: "span_(.*)" action: labeldrop - ... ``` Run the helm upgrade to apply the changes. ### Keep labels and Drop Everything Else You can retain only the labels in your data while removing all other labels by setting the action to `labelkeep` and specifying a regex pattern that matches the desired label names. In the following example, any label that does not begin with `span_` will be excluded from the data. ```yaml ingester: config: aggregator: metric: # relabelConfigs - Configure relabel rules for metric labels. # Format follows Prometheus relabel_config syntax. # Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config relabelConfigs: - ... - regex: "span_(.*)" action: labelkeep - ... ``` Run the helm upgrade to apply the changes. ### Drop or Keep Series with Matching Label Values In business scenarios, some series have to be dropped based on existing label values. For example, to stop storing data for the label `aws_account` with value `123456`, you must add `aws_account_name` label with `AWS_account_name_for_123456` value in the `relabel config` with action `drop`. To keep certain series, use the `keep` action. ```yaml ingester: config: aggregator: metric: # relabelConfigs - Configure relabel rules for metric labels. # Format follows Prometheus relabel_config syntax. # Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config relabelConfigs: - ... - regex: "123456" action: drop source_labels: - aws_account - ... ``` Run the helm upgrade to apply the changes. ### Drop or Keep Series with Matching Metric Name In Metrics stream, `name` is a special label that contains the name of the metric. For example, to stop storing any metric with name that starts with `aws_ec2`, you must add `name` label with `aws_ec2.*` value in the `relabel config` with action `drop`. To only keep these metrics, use the `keep` action. ```yaml ingester: config: aggregator: metric: # relabelConfigs - Configure relabel rules for metric labels. # Format follows Prometheus relabel_config syntax. # Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config relabelConfigs: - ... - regex: "aws_ec2.*" action: drop source_labels: - __name__ - ... ``` Run the helm upgrade to apply the changes.