Prometheus Agent Remote Write Config

Using an Existing Prometheus Server

1. Determine the ingress IP for the Kloudfuse stack.

   kubectl get svc | grep kfuse-ingress-nginx-controller

You can use either the EXTERNAL-IP address, or DNS name.

2. Add a remote_write endpoint to use the Prometheus server. Choose the correct option, depending on your network protocol (http or https):

   http://<REPLACE KFUSE ADDRESS>/write as the remote_write
   
   https://<REPLACE KFUSE ADDRESS>/write as the remote_write
   

   	Starting with Kloudfuse 3.4.3 and Prometheus 2.40+, native histograms are supported. To enable sending native histograms, add send_native_histograms: true to your remote write configuration. See Native and Exponential Histograms for more details.

3. If the Prometheus server is installed using the standalone Prometheus helm chart, add the following helm values and run helm upgrade:

   server:
        # For Prometheus 2.40+ and Kloudfuse 3.4.3+, enable native histograms feature:
        extraArgs:
          enable-feature: native-histograms
   
   serverFiles:
        prometheus.yml:
          remote_write:
         - url: http://<REPLACE KFUSE ADDRESS>/write
           queue_config:
             max_samples_per_send: 4000
           # For Prometheus 2.40+ and Kloudfuse 3.4.3+, enable native histograms:
           send_native_histograms: true
   

Prometheus Operator

Helm Chart Values

Use these steps if the Prometheus server is installed using Prometheus Operator or kube-prometheus-stack helm chart:

1. Add the following helm values:

   prometheus:
        prometheusSpec:
          # For Prometheus 2.40+ and Kloudfuse 3.4.3+, enable native histograms feature:
          enableFeatures:
         - native-histograms
   remoteWrite:
       - queueConfig:
           maxSamplesPerSend: 4000
         url: http://<REPLACE KFUSE ADDRESS>/write
         # For Prometheus 2.40+ and Kloudfuse 3.4.3+, enable native histograms:
         sendNativeHistograms: true
   

2. Run helm upgrade.

Prometheus CRD

Alternatively,configure the remote write through Prometheus CRD.

Update the Prometheus CRD (kind: Prometheus) using the following configuration:

# For Prometheus 2.40+ and Kloudfuse 3.4.3+, enable native histograms feature:
enableFeatures:
  - native-histograms

remoteWrite:
  - queueConfig:
      maxSamplesPerSend: 4000
    url: http://<REPLACE KFUSE ADDRESS>/write
    # For Prometheus 2.40+ and Kloudfuse 3.4.3+, enable native histograms:
    sendNativeHistograms: true

Replacing the Prometheus Server

You can also replace the Prometheus server with Grafana Agent, which is lighter-weight and can be configured to scrape and emit write metrics.

Use the following helm chart to install the agent. Use the following helm values:

config:
  prometheus:
    remoteWrite:
      url: http://<REPLACE KfUSE ADDRESS>/write
    configs:
      <copy existing scrape configs here>