Datadog Agent Processing :: Kloudfuse Docs

Datadog Agent Processing

Overview

The Datadog Agent provides log processing rules and check-level configuration to reshape telemetry data before it is forwarded to Kloudfuse. This page covers common log and metric processing patterns using log_processing_rules for logs, and OpenMetrics check configuration and DogStatsD mapper profiles for metrics.

Unlike a pipeline-based collector, Datadog Agent configuration is split across two file types:

Each example below shows both files where relevant, with comments indicating which file each block belongs to.

Log Processing

Mask PII Data

Use log_processing_rules with type: mask_sequences to replace sensitive values before logs leave the agent. Each rule applies a regex pattern and replaces any match with the value of replace_placeholder.

log_processing_rules can be set per log source in a check config file, or globally in datadog.yaml to apply to all log sources.

conf.d/my-app.d/conf.yaml — mask PII in a specific log source

logs:
  - type: file
    path: /var/log/my-app/*.log
    service: my-service
    source: my-app
    log_processing_rules:
      - type: mask_sequences
        name: mask_passwords
        replace_placeholder: "***"
        pattern: "password=[^\\s&]+"
      - type: mask_sequences
        name: mask_emails
        replace_placeholder: "***@***.***"
        pattern: "\\b[A-Za-z0-9._%+\\-]+@[A-Za-z0-9.\\-]+\\.[A-Za-z]{2,}\\b"
      - type: mask_sequences
        name: mask_credit_cards
        replace_placeholder: "****-****-****-****"
        pattern: "\\b\\d{4}[\\s\\-]?\\d{4}[\\s\\-]?\\d{4}[\\s\\-]?\\d{4}\\b"

To apply masking globally across all log sources, add the same rules under logs_config in datadog.yaml:

datadog.yaml — mask PII across all log sources

logs_config:
  processing_rules:
    - type: mask_sequences
      name: mask_passwords
      replace_placeholder: "***"
      pattern: "password=[^\\s&]+"
    - type: mask_sequences
      name: mask_emails
      replace_placeholder: "***@***.***"
      pattern: "\\b[A-Za-z0-9._%+\\-]+@[A-Za-z0-9.\\-]+\\.[A-Za-z]{2,}\\b"
Test patterns against representative log samples. mask_sequences applies regex substitution on the raw string value of the log body before any parsing occurs.

Add Tags to Logs

Tags set on a log source are attached to every log record collected by that source. Set them in the check config file:

conf.d/my-app.d/conf.yaml — add tags to a specific log source

logs:
  - type: file
    path: /var/log/my-app/*.log
    service: my-service
    source: my-app
    tags:
      - env:production
      - team:platform

To add tags to all metrics, logs, and events collected by the agent, set them globally in datadog.yaml:

datadog.yaml — add tags to all collected data

tags:
  - env:production
  - team:platform
  - region:us-east-1

Parse JSON Logs

The Datadog Agent automatically detects and parses log bodies that are valid JSON. When a log body is a JSON object, each top-level key is promoted to a log attribute and is available as a queryable label in Kloudfuse — no additional configuration is required.

For example, if a log record arrives with this body:

{"level":"error","msg":"connection refused","service":"payments","trace_id":"abc123"}

The agent extracts the following attributes, each of which becomes a searchable label:

Attribute key Value
level error
msg connection refused
service payments
trace_id abc123

If your logs are not being parsed automatically, confirm that JSON auto-detection is enabled in datadog.yaml:

datadog.yaml — confirm JSON auto-detection is enabled

logs_config:
  auto_multi_line_detection: false   (1)
  use_http: true
1 auto_multi_line_detection is for multi-line aggregation and is separate from JSON parsing. JSON detection is always active when logs_enabled: true.

Filter Out Unwanted Logs

Use log_processing_rules with type: exclude_at_match to drop log records whose body matches a regex. Records that match the pattern are dropped and not forwarded to Kloudfuse:

conf.d/my-app.d/conf.yaml — drop health check and readiness probe logs

logs:
  - type: file
    path: /var/log/my-app/*.log
    service: my-service
    source: my-app
    log_processing_rules:
      - type: exclude_at_match
        name: exclude_healthchecks
        pattern: "GET /(healthz|readyz|ping)"
      - type: exclude_at_match
        name: exclude_debug
        pattern: "\\[DEBUG\\]"

To keep only logs that match a pattern and drop everything else, use type: include_at_match.

conf.d/my-app.d/conf.yaml — forward only ERROR and FATAL logs

logs:
  - type: file
    path: /var/log/my-app/*.log
    service: my-service
    source: my-app
    log_processing_rules:
      - type: include_at_match
        name: errors_only
        pattern: "ERROR|FATAL|CRITICAL"   (1)
1 Only log records whose body contains ERROR, FATAL, or CRITICAL are forwarded. All other records are dropped.

Metric Processing

Add Tags to All Metrics

Tags set in datadog.yaml are attached to every metric, log, and event collected by the agent. Use this to add environment, region, or team labels that apply across your entire infrastructure:

datadog.yaml — add global tags to all metrics

tags:
  - env:production
  - team:platform
  - region:us-east-1

To add tags only to metrics collected by a specific OpenMetrics endpoint, set them in the check config:

conf.d/openmetrics.d/conf.yaml — add tags to a specific metrics source

instances:
  - openmetrics_endpoint: http://my-app:8080/metrics
    namespace: my_app
    metrics:
      - ".*"
    tags:
      - env:production
      - team:platform

Drop High-Cardinality Labels from Metrics

When collecting metrics from a Prometheus or OpenMetrics endpoint, use exclude_labels to remove label keys that create an unbounded number of time series — such as request IDs, user IDs, or raw URLs. The listed labels are stripped from every metric collected by that check instance:

conf.d/openmetrics.d/conf.yaml — drop high-cardinality labels

instances:
  - openmetrics_endpoint: http://my-app:8080/metrics
    namespace: my_app
    metrics:
      - ".*"
    exclude_labels:
      - request_id
      - user_id
      - trace_id

Rename Metrics

For OpenMetrics endpoints, rename individual metrics by providing a mapping in the metrics list. Each entry maps the scraped metric name to the desired output name:

conf.d/openmetrics.d/conf.yaml — rename specific metrics

instances:
  - openmetrics_endpoint: http://my-app:8080/metrics
    namespace: my_app
    metrics:
      - http_server_duration: http.server.request.duration   (1)
      - process_cpu_usage: process.cpu.utilization
      - ".*"                                                 (2)
1 The scraped metric http_server_duration is forwarded to Kloudfuse as my_app.http.server.request.duration (the namespace is prepended automatically).
2 Collect all remaining metrics under their original names.

For DogStatsD metrics, use dogstatsd_mapper_profiles in datadog.yaml to rename metrics by matching a pattern:

datadog.yaml — rename DogStatsD metrics

dogstatsd_mapper_profiles:
  - name: my_app
    prefix: "my_app."
    mappings:
      - match: "my_app.http_server_duration"
        name: "my_app.http.server.request.duration"
      - match: "my_app.process_cpu_*"
        name: "my_app.process.cpu.$1"
        match_type: wildcard

Drop Metrics by Name or Pattern

For OpenMetrics endpoints, use exclude_metrics to drop metrics by exact name or regex pattern. Each entry is a drop condition — any metric whose name matches is not forwarded to Kloudfuse:

conf.d/openmetrics.d/conf.yaml — drop metrics by exact name

instances:
  - openmetrics_endpoint: http://my-app:8080/metrics
    namespace: my_app
    metrics:
      - ".*"
    exclude_metrics:
      - go_gc_duration_seconds
      - process_open_fds
      - process_max_fds

To drop all metrics whose names match a regex pattern:

conf.d/openmetrics.d/conf.yaml — drop metrics matching a pattern

instances:
  - openmetrics_endpoint: http://my-app:8080/metrics
    namespace: my_app
    metrics:
      - ".*"
    exclude_metrics:
      - "go_.*"                   (1)
      - "process_runtime_.*"
1 Drops all metrics whose names start with go_. Any metric matching any pattern in exclude_metrics is dropped.

To forward only metrics that match a specific pattern and drop everything else, list only the desired pattern in metrics rather than using ".*":

conf.d/openmetrics.d/conf.yaml — collect only metrics matching a pattern

instances:
  - openmetrics_endpoint: http://my-app:8080/metrics
    namespace: my_app
    metrics:
      - "http_.*"    (1)
1 Only metrics whose names start with http_ are collected. All other metrics exposed by the endpoint are ignored.

External References