GCP Log Integration :: Kloudfuse Docs

GCP Log Integration

Kloudfuse ingests GCP logs by pulling messages from a Cloud Pub/Sub subscription. A Cloud Logging log sink routes log entries to a Pub/Sub topic, and Kloudfuse continuously pulls from the corresponding subscription.

For an overview of how this fits into the broader GCP integration architecture, see GCP Integration Architecture.

Overview

Any GCP service that writes to Cloud Logging can be ingested — including Compute Engine, GKE, Cloud SQL, App Engine, Cloud Functions, and custom application logs forwarded via the Cloud Logging API.

Data flows as follows:

GCP resources write log entries to Cloud Logging.
A Log Sink routes matching log entries to a Pub/Sub topic.
Kloudfuse maintains a pull subscription on that topic and continuously retrieves batches of log messages.
Logs appear in the Kloudfuse Log Analytics interface within 30–60 seconds.

Common use cases:

Security monitoring — Detect unusual API patterns, unauthorized access, or privilege escalation from Cloud Audit Logs.
Compliance reporting — Maintain a record of all configuration changes to GCP resources.
Incident investigation — Correlate GCP infrastructure changes with application errors using Kloudfuse’s unified log and metrics view.

Prerequisites

A GCP project with Cloud Logging and Pub/Sub APIs enabled.
The gcloud CLI installed and authenticated, or access to the Google Cloud Console.
A running Kloudfuse deployment on GKE.
The Kloudfuse ingestion namespace (default: kfuse).

Step 1: Create a Pub/Sub Topic and Subscription

Create the Topic

gcloud pubsub topics create kloudfuse-logs \
  --project=<project-id>

Create the Pull Subscription

gcloud pubsub subscriptions create kloudfuse-gcp-subscription \
  --topic=kloudfuse-logs \
  --project=<project-id> \
  --message-retention-duration=1d \
  --expiration-period=never

The subscription ID kloudfuse-gcp-subscription matches the default value in the Helm configuration. Use a different name if needed and update the subscriptionId field accordingly.

Step 2: Create a Log Sink

A log sink routes Cloud Logging entries to the Pub/Sub topic.

Route All Log Entries

To forward all log entries from the project:

gcloud logging sinks create kloudfuse-logs-sink \
  pubsub.googleapis.com/projects/<project-id>/topics/kloudfuse-logs \
  --project=<project-id>

Route Filtered Log Entries

To forward only specific log entries (for example, Cloud Audit Logs):

gcloud logging sinks create kloudfuse-audit-sink \
  pubsub.googleapis.com/projects/<project-id>/topics/kloudfuse-logs \
  --project=<project-id> \
  --log-filter='logName=~"cloudaudit.googleapis.com"'

Grant the Sink Writer Permission

After creating the sink, retrieve the sink’s service account and grant it publish rights on the topic:

SINK_SERVICE_ACCOUNT=$(gcloud logging sinks describe kloudfuse-logs-sink \
  --project=<project-id> \
  --format='value(writerIdentity)')

gcloud pubsub topics add-iam-policy-binding kloudfuse-logs \
  --project=<project-id> \
  --member="$SINK_SERVICE_ACCOUNT" \
  --role="roles/pubsub.publisher"

Step 3: Configure a Service Account

Kloudfuse needs a service account with the roles/pubsub.subscriber role to pull messages from the subscription.

Create the Service Account

gcloud iam service-accounts create kloudfuse-gcp \
  --display-name="Kloudfuse GCP Integration" \
  --project=<project-id>

Assign Roles

gcloud projects add-iam-policy-binding <project-id> \
  --member="serviceAccount:kloudfuse-gcp@<project-id>.iam.gserviceaccount.com" \
  --role="roles/pubsub.subscriber"


	If you also want to collect metrics, assign `roles/monitoring.viewer` and `roles/compute.viewer` to the same service account and reuse the same Kubernetes secret for both pipelines. See GCP Metrics — Service Account setup.

Create the JSON Key

gcloud iam service-accounts keys create kloudfuse-gcp-credentials.json \
  --iam-account=kloudfuse-gcp@<project-id>.iam.gserviceaccount.com

Create the Kubernetes Secret


	The credentials file must be named `credentials.json` inside the secret.

kubectl create secret generic kfuse-gcp-credentials \
  --from-file=credentials.json=kloudfuse-gcp-credentials.json \
  --namespace <kloudfuse-namespace>

Replace <kloudfuse-namespace> with your Kloudfuse installation namespace (kfuse by default).

Step 4: Configure Kloudfuse via Helm

Add the following to your custom-values.yaml:

global:
  enrichmentEnabled:
    - gcp
  gcpConfig:
    secretName: "kfuse-gcp-credentials" (1)
    projectId: "<project-id>" (2)
    pubsub:
      enabled: true
      subscriptions: (3)
        - projectId: "<project-id>" (4)
          subscriptionId: "kloudfuse-gcp-subscription" (5)

Consume Logs from Multiple Subscriptions

A single secret can consume logs from more than one Pub/Sub subscription, including subscriptions that belong to different GCP projects. Ensure the service account has the roles/pubsub.subscriber role in each project (or on each subscription) you list.

gcpConfig:
  pubsub:
    subscriptions:
      - projectId: "gcp-project-a-481726"
        subscriptionId: "kloudfuse-logs-sub-1"
      - projectId: "gcp-project-b-573204"
        subscriptionId: "kloudfuse-logs-sub-2"

Then apply the Helm upgrade:

helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
  -n kfuse \
  --version <VERSION> \ (1)
  -f custom-values.yaml

Verify the Integration

Confirm the Subscription is Receiving Messages

Check that messages are flowing into the Pub/Sub subscription:

gcloud pubsub subscriptions describe kloudfuse-gcp-subscription \
  --project=<project-id> \
  --format='table(name, messageRetentionDuration, expirationPolicy)'

To pull a single test message and confirm the subscription is active:

gcloud pubsub subscriptions pull kloudfuse-gcp-subscription \
  --project=<project-id> \
  --limit=1 \
  --auto-ack

Confirm Logs are Arriving in Kloudfuse

In the Kloudfuse UI, click the Logs tab and select Search from the drop-down.
Set the time picker to the last 15 minutes.
In the search bar, click Advanced Search and enter the following FuseQL query to confirm GCP logs are flowing:

source="gcp"

Any result confirms that GCP log records are reaching Kloudfuse.

To filter by GCP log name:

logName="projects/<project-id>/logs/cloudaudit.googleapis.com%2Factivity"

To count log volume by GCP resource type:

source="gcp" | count by resource.type

Troubleshooting

No Messages in the Pub/Sub Subscription

The subscription is not receiving messages from Cloud Logging.

Confirm the log sink was created and is active:

gcloud logging sinks describe kloudfuse-logs-sink \
     --project=<project-id> \
     --format='table(name, destination, writerIdentity)'

Confirm the sink’s writer identity has the roles/pubsub.publisher role on the topic.
Confirm logs are being written to Cloud Logging by checking the Logs Explorer in the Google Cloud Console.
If using an inclusion filter on the sink, verify the filter matches logs that are actually being emitted.

Kloudfuse Pod Cannot Pull from Pub/Sub

If Kloudfuse logs show authentication or permission errors when pulling from Pub/Sub:

Confirm the Kubernetes secret exists and contains credentials.json:

kubectl get secret kfuse-gcp-credentials -n <kloudfuse-namespace> -o jsonpath='{.data}' | jq 'keys'

Confirm the service account has the roles/pubsub.subscriber role:

gcloud projects get-iam-policy <project-id> \
     --flatten="bindings[].members" \
     --filter="bindings.members:kloudfuse-gcp@" \
     --format="table(bindings.role)"

Check the Kloudfuse ingester pod logs for specific error messages:

kubectl logs -n <kloudfuse-namespace> -l app=ingester --tail=100 | grep -i "pubsub\|gcp\|error"

Logs Appear in Pub/Sub but Not in Kloudfuse

Confirm pubsub.enabled: true and the correct subscriptionId are set in custom-values.yaml.
Confirm enrichmentEnabled includes gcp and gcpConfig.secretName matches the Kubernetes secret name.
Restart the ingester pod to pick up configuration changes:

kubectl rollout restart deployment -n <kloudfuse-namespace> -l app=ingester

Pub/Sub Subscription Lag is Growing

The Kloudfuse puller is falling behind the rate of incoming log messages.

Check the subscription’s undelivered message count in the Google Cloud Console under Pub/Sub > Subscriptions.
If lag is consistently growing, increase the puller concurrency or the number of ingester replicas via Helm.
Consider adding an exclusion filter to the log sink to reduce the volume of log entries forwarded.