VictoriaMetrics Agent :: Kloudfuse Docs

VictoriaMetrics Agent

The VictoriaMetrics Agent (vmagent) is a lightweight metrics collection agent that scrapes Prometheus-compatible endpoints. It forwards the results to Kloudfuse using the Prometheus remote write protocol. It is a drop-in alternative to a Prometheus server for pure metric collection, with lower memory usage and built-in sharding support for large environments.

Overview

vmagent scrapes targets defined in a scrape_configs block — the same format as Prometheus — and writes the collected samples to one or more remote write endpoints. Kloudfuse accepts remote write at /ingester/write, making it compatible with any vmagent deployment without additional configuration.

When data reaches the Kloudfuse ingester it is stored in the time-series store and becomes immediately queryable via PromQL, dashboards, and alerts.

Prerequisites

A running Kloudfuse installation with its external hostname (for example, kloudfuse.example.com).
Helm 3 installed on your workstation.
If ingestion authentication is enabled on your cluster, an API key — see Ingestion Authentication with API Key.

Install with Helm

The official vmagent Helm chart is published to the GitHub Container Registry.

Create a values.yaml file with the Kloudfuse remote write endpoint:

   remoteWrite:
     - url: https://<kloudfuse-hostname>/ingester/write

If ingestion authentication is enabled, add the Authorization header:

   remoteWrite:
     - url: https://<kloudfuse-hostname>/ingester/write
       headers:
         - "Authorization: Bearer <token>"

Install the agent — replace <namespace> with the Kubernetes namespace where you want to deploy it:

   helm upgrade --install vmagent \
        oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-agent \
        -f values.yaml \
        -n <namespace> \
        --create-namespace

Confirm the pod is running:

   kubectl get pods -n <namespace> -l app.kubernetes.io/name=victoria-metrics-agent

Configure Scraping

vmagent uses the same scrape_configs format as Prometheus. Add targets to your values.yaml under config.scrape_configs:

config:
  scrape_configs:
    - job_name: my-app
      static_configs:
        - targets:
            - my-app-service.my-namespace.svc.cluster.local:8080

- job_name: node-exporter
      kubernetes_sd_configs:
        - role: node
      relabel_configs:
        - source_labels: [__address__]
          regex: (.+):(.+)
          target_label: __address__
          replacement: ${1}:9100

For Kubernetes service discovery, grant vmagent the required RBAC permissions. The Helm chart creates a ClusterRole and ClusterRoleBinding by default when rbac.create: true (the default).

Multiple Remote Write Endpoints

vmagent supports writing to multiple endpoints simultaneously. To send metrics to Kloudfuse and retain a local Prometheus instance at the same time:

remoteWrite:
  - url: https://<kloudfuse-hostname>/ingester/write
    headers:
      - "Authorization: Bearer <token>"
  - url: http://prometheus.monitoring.svc.cluster.local:9090/api/v1/write

Verify Metrics Are Arriving

After installation, confirm metrics are arriving in the Kloudfuse UI:

Click the Metrics tab, then select Explorer from the drop-down menu.
Set the time range to the last 15 minutes using the interval picker in the top-right corner.
In the metric selector, type the name of a metric scraped by vmagent (for example, node_cpu_seconds_total if scraping node-exporter, or vmagent_remotewrite_requests_total to check `vmagent’s own self-monitoring metrics).
Select the metric from the suggestions list and confirm a chart appears with data points.
Use the label filters to narrow results by job or instance — for example, filter by job="node-exporter" to isolate metrics from a specific scrape target.

Troubleshooting

No Metrics in Kloudfuse

If PromQL queries return no data:

Check vmagent pod logs for remote write errors:

   kubectl logs -n <namespace> -l app.kubernetes.io/name=victoria-metrics-agent --tail=50

Look for cannot send data or connection refused messages.

Confirm the remote write URL is reachable from inside the cluster by checking vmagent’s own remote write target status. Port-forward to the vmagent web UI and open the targets page in a browser:

   kubectl port-forward -n <namespace> <vmagent-pod-name> 8429:8429

Then open http://localhost:8429/targets in a browser. Each configured remoteWrite URL is listed there with its current status and last error, if any.

Check that the ingester endpoint path is exactly /ingester/write — the /ingester/ prefix is required when routing through the Kloudfuse ingress.

400 or 401 Errors in Logs

If vmagent logs show HTTP 400 or 401 responses from the remote write endpoint:

Confirm ingestion authentication is enabled on your cluster — if it is, the Authorization: Bearer <token> header is required.
Verify the token is valid via Admin > Settings > Auth key labels in the Kloudfuse UI.
Check that the header is formatted exactly as Authorization: Bearer <token> with no extra whitespace.

Metrics Are Delayed or Backlogged

If metrics appear in Kloudfuse with significant delay, or vmagent_remotewrite_pending_data_bytes is growing:

Increase the number of concurrent remote write workers:

   extraArgs:
       remoteWrite.queues: "8"

Increase remoteWrite.maxSamplesPerBlock to reduce the number of round trips to the ingester:

   extraArgs:
       remoteWrite.maxSamplesPerBlock: "4000"

Check Kloudfuse ingester availability — a temporary outage causes backpressure that vmagent drains automatically once connectivity is restored.