# Datadog Agent for Kubernetes

See the following documentation:

- [Datadog Agent Requirements](https://docs.kloudfuse.com/platform/3.5.0/agent-datadog-kubernetes-requirements/)
- [Install Datadog Agent on Kubernetes](https://docs.kloudfuse.com/platform/3.5.0/agent-datadog-kubernetes-install/)
- [Configure Tags through Datadog Agent on Kubernetes](https://docs.kloudfuse.com/platform/3.5.0/agent-datadog-kubernetes-tags/)

## Collect high cardinality tags

The Datadog Agent is installed with cardinality set to **orchestrator** level, allowing granular (both pod and container-level) tagging of metrics.

For more information, see Datadog documentation on [Assigning Tags](https://docs.datadoghq.com/getting_started/tagging/assigning_tags/?tab=noncontainerizedenvironments#tags-cardinality).

The default setting in the Datadog Agent is set to **low**, tagging only at host level.

## Install Command

If you haven’t before, add datadog helm repo:

```yaml
helm repo add datadog https://helm.datadoghq.com
helm repo update
```

Install the datadog agent with updated values (cmd below assumes the file is called `dd-values-kfuse.yaml`. Replace that argument with the filename that’s relevant to your scenario).

```yaml
# Create separate namespace for the agent to be installed (if required)
kubectl create namespace kfuse-agent
helm upgrade --install kfuse-agent -f dd-values-kfuse.yaml datadog/datadog -n kfuse-agent --version 3.65.0
```

## Using Existing Datadog Agent Install

If you have an existing datadog agent, then you can use the same agent to send your data streams to the Kloudfuse platform.

## Streaming to Kloudfuse in addition to Datadog

Datadog Agent supports dual shipping. To send the data to Kloudfuse in addition to the Datadog, add the following in the datadog agent helm values.

|     |     |
| --- | --- |
|  | Change for each stream, as necessary. |

1. Under the `agents.customAgentConfig` key, specify:

```console
additional_endpoints:
     "http://<kfuse-ingress-external-ip>/ingester": (1)
   - <api_key>
use_v2_api:
series: true
logs_config:
...
logs_no_ssl: true
use_v2_api: false
additional_endpoints:
    - api_key: <api_key>
      Host: <kfuse-ingress-external-ip>
      Port: 80
      use_compression: true
apm_config:
...
additional_endpoints:
    "https://kfuse-ingress-external-ip>":
       - <api_key>
process_config:
...
additional_endpoints:
    "https://kfuse-ingress-external-ip>":
       - <api_key>
metadata_providers:
  - name: host
    interval: 300
```

2. Under the `clusterAgent.datadog_cluster_yaml` key, specify:

```console
additional_endpoints:
     "http://<kfuse-ingress-external-ip>/ingester":
   - <api_key>
use_v2_api:
series: true
process_config:
...
additional_endpoints:
    "https://kfuse-ingress-external-ip>":
       - <api_key>
orchestrator_explorer:
...
additional_endpoints:
    "https://kfuse-ingress-external-ip>":
       - <api_key>
```

|     |     |
| --- | --- |
| **1** | Use `http://` instead of `https://` when referencing the Kloudfuse ingester. |

## Adding Custom Tags

You can add custom tags to the agent, ensuring that all metrics that your agent collects has the correct tag.

To add custom tags, follow these steps:

1. Update `dd-values-kfuse.yaml` file to include the following code to enable custom tags.

|     |     |
| --- | --- |
|  | Each tag must be in the form `key:value`, a string separated by a column (`:`). |

```yaml
datadog:
     tags:
    - custom_tag_name:custom_tag_value
```

2. The Kloudfuse helm `custom-values.yaml` file must list the custom tags.

```yaml
ingester:
     config:
       hostTagIncludes:
       ...
    - custom_tag_name (1)
    ...
```

|     |     |
| --- | --- |
| **1** | `custom_tag_name`: Add all custom tag names; otherwise, the system may overwrite existing configurations. |

3. Alternatively, if you don’t plan to maintain a long list of possible custom tags, set `allowAllHostTags` to `true`. This effectively adds all extracted custom tags to the allow list.

```yaml
ingester:
     config:
       allowAllHostTags: true (1)
```

|     |     |
| --- | --- |
| **1** | `allowAllHostTags`: Adds all custom tag names to the allow list; otherwise, the system may overwrite existing configurations. |

## Enabling Pods to be detected by Prometheus Autodiscovery

In addition to enabling `prometheusScrape` in the Datadog `values.yaml`, the pods must have the following annotations.

|     |     |
| --- | --- |
|  | If you deploy application pods through helm, the helm values<br>support a `podAnnotations` section. |

```yaml
prometheus.io/path: <specify prometheus endpoint path, e.g., /metrics>
prometheus.io/port: <SPECIFY prometheus endpoint here, e.g., "9090">
prometheus.io/scrape: "true"
```

## Metadata for Metrics collected using the OpenMetrics check

If you use the preceding configuration, then the openmetrics check is enabled in the agent.

For more details, see Datadog documentation on [Kubernetes Prometheus and OpenMetrics metrics collection](https://docs.kloudfuse.com/platform/3.5.0/agent-datadog-kubernetes/#:https://docs.datadoghq.com/containers/kubernetes/prometheus/?tab=kubernetesadv1).

The Kloudfuse agent installed using the provided values file uses a custom check, `kf_openmetrics`, does not collect metadata by default. You can:

1. Configure `kf_openmetrics` to collect metadata by annotating the required sources.
2. Enable the **Autodiscovery agent** for these pods.

See the Datadog documentation section [Kubernetes and Integrations](https://docs.datadoghq.com/containers/kubernetes/integrations/?tab=kubernetesadv22).
3. Run the custom check.

## Kubernetes environment

To enable metrics metadata collection from a Kubernetes pod, as discussed in Datadog documentation section [Kubernetes and Integrations: Configuration](https://docs.datadoghq.com/containers/kubernetes/integrations/?tab=annotations#configuration), add the following code.

|     |     |
| --- | --- |
|  | You can add this to the helm and each deployment. |

```yaml
apiVersion: v1
kind: Pod
# (...)
metadata:
  name: '<POD_NAME>'
annotations:
    ad.datadoghq.com/<CONTAINER_IDENTIFIER>.check_names: '[