GCP Logs Collection using Pub/Sub

Kloudfuse integrates with the GCP cloud service to collect logs through the Pub/Sub console.

Complete these tasks to successfully collect logs:

• Create a GCP Subscription
• Configure Kloudfuse to Consume Logs from the GCP Subscription

Create a GCP Subscription

1. Use an existing project, or create a new project in Cloud Pub-Sub console

Create New Project

2. In the Cloud Pub-Sub console, create a new topic.

Create Topic

Specify Topic

3. In the Pub/Sub Topics overview page, select Subscriptions in the left-hand navigation.

4. Click Create Subscription.

Create Subscription

5. Create a subscription with ID kloudfuse-gcp-subscription and select the topic you previously created, MyLogsCollector.

Click Create.

Specify Subscription

6. Confirm that the new subscription exists.

New Subscription created successfully

7. In the Logs Explorer page of the console, under More actions, choose Create sink from the drop-down.

Create Sink

8. In Sink details, specify the Name and optional Description of the sink.

Click Next.

Sink details

9. In Sink destination, in Select sink service, choose Cloud Pub/Sub. In Select a Cloud Pub/Sub Topic, select the topic that you created in a previous step.

Click Next.

Sink destination

10. In the Choose logs to include in sink, create optional inclusion filters for the logs.

Click Preview logs to see the available logs.

Click Next.

Preview Logs

11. In the Choose logs to filter out of sink, create optional exclusion filters for the logs.

Click Next.

12. Click Create sink.

Create sink

13. The confirmation message appears.

Confirmation

Configure Kloudfuse to Consume Logs from the GCP Subscription

Add the following configuration to the custom-values.yaml file when installing Kloudfuse using helm, and add the GCP pub/sub access key in the pubsubkey field

logs-parser:
  gcpConfig:
    enabled: true
    subscriptionId: "kloudfuse-gcp-subscription" (1)
    pubsubKey: "" (2)

yamlCopied!