GCP Logs Collection :: Kloudfuse Docs

GCP Logs Collection

Table of Contents

Kloudfuse integrates with the GCP cloud service to collect logs through the Pub/Sub console.

Complete these tasks to successfully collect logs:

Create a GCP Subscription

  1. Use an existing project, or create a new project in Cloud Pub-Sub console

Create New Project

  1. In the Cloud Pub-Sub console, create a new topic.

Create Topic

Specify Topic

  1. In the Pub/Sub Topics overview page, select Subscriptions in the left-hand navigation.

  2. Click Create Subscription.

Create Subscription

  1. Create a subscription with ID kloudfuse-gcp-subscription and select the topic you previously created, MyLogsCollector.

Click Create.

Specify Subscription

  1. Confirm that the new subscription exists.

New Subscription created successfully

  1. In the Logs Explorer page of the console, under More actions, choose Create sink from the drop-down.

Create Sink

  1. In Sink details, specify the Name and optional Description of the sink.

Click Next.

Sink details

  1. In Sink destination, in Select sink service, choose Cloud Pub/Sub. In Select a Cloud Pub/Sub Topic, select the topic that you created in a previous step.

Click Next.

Sink destination

  1. In the Choose logs to include in sink, create optional inclusion filters for the logs.

Click Preview logs to see the available logs.

Click Next.

Preview Logs

  1. In the Choose logs to filter out of sink, create optional exclusion filters for the logs.

Click Next.

  1. Click Create sink.

Create sink

  1. The confirmation message appears.

Confirmation

Configure Service Account for Pub/Sub Access

To allow Kloudfuse to consume logs from the GCP Pub/Sub subscription, you must configure a service account with the appropriate permissions and generate credentials.

Assign Pub/Sub Subscriber Role

  1. In the Google Cloud console, navigate to Menu > IAM & Admin > Service Accounts.

  2. Select your service account, or create a new one.

  3. Navigate to Menu > IAM & Admin > IAM.

  4. Click Grant Access to add a new principal, or edit an existing principal.

  5. In the New principals field, enter the service account email.

  6. In the Assign roles section, select Pub/Sub Subscriber (roles/pubsub.subscriber).

This role allows the service account to consume messages from the Pub/Sub subscription.

  1. Click Save.

Create the Service Account Key

  1. In the Google Cloud console, navigate to Menu > IAM & Admin > Service Accounts.

  2. Select the service account you configured with the Pub/Sub Subscriber role.

  3. Select Keys > Add key > Create new key.

  4. Select JSON, then click Create.

The JSON key file downloads to your local machine.

  1. Click Close.

Create Kubernetes Secret
The credentials file must be named credentials.json when creating the Kubernetes secret.

Create a Kubernetes secret from the downloaded JSON key file:

kubectl create secret generic kfuse-gcp-credentials \
  --from-file=credentials.json=<your-service-account-key>.json \
  -n kfuse

Copied!
This is the same secret used by GCP Metrics Collection. If you already created it for metrics, reuse it here — just ensure the service account also has the roles/pubsub.subscriber role.

Configure Kloudfuse to Consume Logs from the GCP Subscription

Add the following configuration to the custom-values.yaml file when installing Kloudfuse using Helm.

global:
  ...
  enrichmentEnabled:
    - gcp
  gcpConfig:
    secretName: "kfuse-gcp-credentials" (1)
    pubsub:
      enabled: true
      subscriptionId: "kloudfuse-gcp-subscription" (2)
  ...

yamlCopied!
1 The name of the Kubernetes secret containing the GCP service account credentials.
2 The Pub/Sub subscription ID created in the previous section.