Filebeat :: Kloudfuse Docs

Filebeat

Overview

Filebeat is a lightweight log shipper from the Elastic stack. It monitors log files and locations you specify, collects log events, and forwards them to a destination using output plugins.

Kloudfuse exposes an ingester endpoint compatible with the Filebeat Elasticsearch output:

https://<kloudfuse-hostname>/ingester/api/v1/filebeat

Redirecting the Elasticsearch output to this URL is the only configuration change required.

Prerequisites

Configure Filebeat

Elasticsearch Output

Set the output.elasticsearch hosts value in your filebeat.yml to the Kloudfuse ingester endpoint:

setup.ilm.enabled: false
setup.template.enabled: false

output.elasticsearch:
  hosts: ["https://<kloudfuse-hostname>:443/ingester/api/v1/filebeat"]

setup.ilm.enabled: false and setup.template.enabled: false are required — Kloudfuse does not implement the Elasticsearch index management APIs.
Include :443 explicitly in the hosts URL.
Filebeat’s Elasticsearch output does not infer the port from the https:// scheme — it defaults to port 9200 if no port is specified, which will fail.

If ingestion authentication is enabled, add the API key as a header:

setup.ilm.enabled: false
setup.template.enabled: false

output.elasticsearch:
  hosts: ["https://<kloudfuse-hostname>:443/ingester/api/v1/filebeat"]
  headers:
    Kf-Api-Key: <token>

Helm Installation

Filebeat is commonly deployed as a Kubernetes DaemonSet using the elastic/filebeat Helm chart.

  1. Add the Elastic Helm repository:
helm repo add elastic https://helm.elastic.co
helm repo update
  1. Create a custom-values.yaml file with the Filebeat configuration:
daemonset:
     # Disable default Elasticsearch TLS cert and credential mounts — not needed for Kloudfuse
     secretMounts: []
     extraEnvs: []

filebeatConfig:
       filebeat.yml: |
         filebeat.inputs:
        - type: container
          paths:
            - /var/log/containers/*.log
          processors:
            - add_kubernetes_metadata:
                host: ${NODE_NAME}
                matchers:
                  - logs_path:
                      logs_path: "/var/log/containers/"

setup.ilm.enabled: false
      setup.template.enabled: false

output.elasticsearch:
        hosts: ["https://<kloudfuse-hostname>:443/ingester/api/v1/filebeat"]
        headers:
          Kf-Api-Key: <token>   # omit if auth is not enabled
Copy across any existing filebeatConfig you already have — the filebeat.yml key is a full file replacement, so anything not included will be lost.
secretMounts: [] and extraEnvs: [] are required when using the elastic/filebeat Helm chart.
The chart’s defaults mount an Elasticsearch TLS certificate secret (elasticsearch-master-certs) and inject
Elasticsearch credentials as environment variables (elasticsearch-master-credentials).
Neither secret exists when pointing Filebeat at Kloudfuse, so the pods will fail to start without these overrides.
  1. Install or upgrade Filebeat:
helm upgrade --install filebeat elastic/filebeat \
     -f custom-values.yaml \
     --namespace logging \
     --create-namespace

Use deployment instead of daemonset if Filebeat is configured to run as a Deployment rather than a DaemonSet.

Kubernetes Metadata Enrichment

The add_kubernetes_metadata processor enriches each log event with pod name, namespace, labels, and container name. Add it to any filebeat.inputs block collecting container logs:

filebeat.inputs:
  - type: container
    paths:
      - /var/log/containers/*.log
    processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

Kloudfuse automatically uses the kubernetes.container.name field as the log source when it is present.

Ingest Logs from an AWS S3 Bucket

Filebeat’s aws-s3 input type reads log files stored in S3 and forwards them to Kloudfuse. Run Filebeat as a Deployment (not a DaemonSet) for S3 ingestion.

daemonset:
  enabled: false

deployment:
  enabled: true
  filebeatConfig:
    filebeat.yml: |
      filebeat.inputs:
        - type: aws-s3
          number_of_workers: 5
          bucket_arn: "arn:aws:s3:::<your-bucket-name>"
          access_key_id: "<access-key-id>"         (1)
          secret_access_key: "<secret-access-key>" (2)
          endpoint: amazonaws.com
          default_region: "${AWS_REGION:us-east-1}"
          fields:
            source: <log-source-label>             (3)

setup.ilm.enabled: false
      setup.template.enabled: false

output.elasticsearch:
        hosts: ["https://<kloudfuse-hostname>:443/ingester/api/v1/filebeat"]
        headers:
          Kf-Api-Key: <token>   # omit if auth is not enabled
1 Not required when using a node IAM role or IRSA.
2 Not required when using a node IAM role or IRSA.
3 Logs appear under this source name in Kloudfuse.

IAM Permissions for S3

When using a node IAM role or IRSA instead of static credentials, attach the following permissions to the role:

{
  "Effect": "Allow",
  "Action": [\
    "s3:ListAllMyBuckets",\
    "s3:GetBucketLocation",\
    "s3:ListBucket",\
    "s3:GetObject"\
  ],
  "Resource": [\
    "arn:aws:s3:::<your-bucket-name>",\
    "arn:aws:s3:::<your-bucket-name>/*"\
  ]
}

Verify Logs Are Arriving

After restarting Filebeat with the updated configuration, generate some log output, then confirm logs are arriving in the Kloudfuse UI:

  1. Click the Logs tab to open the Logs search view.
  2. In the Filters panel on the left, locate the Source facet and select filebeat, or type filebeat in the search bar to filter by source.
  3. Set the time range to the last 15 minutes using the interval picker in the top-right corner.
  4. Confirm that log entries appear in the list.
  5. Click any log row to open the Detail view. The Facets section displays parsed fields, and the Kubernetes Labels section shows enriched metadata such as namespace, pod name, and container name when the add_kubernetes_metadata processor is enabled.
  6. To narrow results by Kubernetes namespace or container, click the relevant value in the Kubernetes Labels section and choose Filter By.

Troubleshooting

No Logs Appearing in Kloudfuse

If source="filebeat" returns no results after restarting Filebeat:

  1. Check the Filebeat logs for output errors:
# Systemd:
journalctl -u filebeat --since "5 minutes ago" | grep -iE "error|warn|output"

# Kubernetes:
kubectl logs -n logging -l app=filebeat --tail=50
  1. Confirm the output host is correct — the path must be exactly /ingester/api/v1/filebeat. A missing /ingester/ prefix returns a 404.
  2. Verify the Kloudfuse hostname is reachable from the Filebeat host or pod:
curl -sf https://<kloudfuse-hostname>/health/ && echo "reachable"
  1. Confirm setup.ilm.enabled: false and setup.template.enabled: false are set. Without these, Filebeat attempts to configure Elasticsearch index management at startup, which fails and may prevent log delivery.

Filebeat Connecting to Port 9200 Instead of 443

If Filebeat logs show connection errors to <kloudfuse-hostname>:9200, the hosts URL is missing an explicit port. Filebeat’s Elasticsearch output does not infer the port from the https:// scheme — it defaults to 9200 when no port is given.

Change:

hosts: ["https://<kloudfuse-hostname>/ingester/api/v1/filebeat"]

To:

hosts: ["https://<kloudfuse-hostname>:443/ingester/api/v1/filebeat"]

Helm Pods Failing with "secret not found"

If pods deployed via the elastic/filebeat Helm chart fail with errors like:

secret "elasticsearch-master-certs" not found
secret "elasticsearch-master-credentials" not found

Add the following to your Helm values to disable the chart’s default Elasticsearch credential mounts:

daemonset:
  secretMounts: []
  extraEnvs: []

401 Unauthorized Errors

If Filebeat logs show HTTP 401 responses from the ingester:

ILM or Template Errors at Startup

If Filebeat logs contain errors like error creating index template or ILM not supported:

S3 Input Not Reading Files

If S3 logs are not appearing in Kloudfuse:

External References