GCP Metrics Collection

Kloudfuse integrates with the GCP cloud service to collect metrics.

Complete these tasks to successfully collect metrics:

• Create Credentials File

• Create Kubernetes Secret

• Configure Kloudfuse to Collect Metrics

• Enable Cloud Exporter

• Configure GCP Metrics Collection

• Method 1: Using Kubernetes Secret (Recommended for Production)

• Method 2: Using Direct secretKey (Quick Setup Only)

• Configure GCP Metrics Enrichment

• Prerequisites

• Enable GCP Metrics Enrichment

• Multi-Project Configuration

Create Credentials File

1. In the Google Cloud console, navigate to Menu > IAM & Admin > Service Accounts.

2. Select your service account, or create a new one.

Ensure that the service account has Monitoring Viewer permissions to collect GCP metrics.

3. Select Keys > Add key > Create new key.

4. Select JSON, then click Create.

5. Click Close.

Create Kubernetes Secret

Create a Kubernetes secret from the credentials file:

kubectl create secret generic kfuse-sd-secret --from-file=credentials.json -n kfuse

Configure Kloudfuse to Collect Metrics

Enable Cloud Exporter

First, enable the cloud exporter service in the global configuration:

global:
  cloud-exporter:
    enabled: true
  gcpConfig:
    secretName: "kfuse-sd-secret"

Configure GCP Metrics Collection

There are two methods to provide GCP credentials for metrics collection:

Method 1: Using Kubernetes Secret (Recommended for Production)

This is the recommended approach for production environments as it keeps sensitive credentials secure.

1. Update the helm values using the following configuration.

kfuse-cloud-exporter:
     prometheus-stackdriver-exporter:
       enabled: true
       stackdriver:
         httpTimeout: 30s
         maxRetries: 3
         projectId: "YOUR PROJECT ID" (1)
         ## projectId: "<project1-id>, <project2-id>, <project3-id>" (2)
         metrics:
           typePrefixes: "compute.googleapis.com"

Method 2: Using Direct secretKey (Quick Setup Only)

1. Convert the JSON credentials file to base64:

base64 -w 0 credentials.json

2. Update the helm values with the base64-encoded credentials:

kfuse-cloud-exporter:
     prometheus-stackdriver-exporter:
       enabled: true
       secretKey: "<BASE64_ENCODED_CREDENTIALS>" (1)
       stackdriver:
         httpTimeout: 30s
         maxRetries: 3
         projectId: "YOUR PROJECT ID"
         metrics:
           typePrefixes: "compute.googleapis.com"

3. Update the corresponding projectId of the GCP account and update the typePrefixes to collect the metrics from relevant services of GCP; see Google Cloud documentation on Google Cloud metrics.

Configure GCP Metrics Enrichment

Kloudfuse can enrich GCP metrics with custom labels and metadata from your GCP resources. This feature adds resource labels (like custom tags) from GCE instances to the corresponding metrics, providing better context for monitoring and alerting.

Prerequisites

• GCP credentials with appropriate permissions for resource enrichment

• The same Kubernetes secret (kfuse-sd-secret) created in the previous steps

Create IAM Role and Service Account

Create a service account in your GCP project with the required permissions for Kloudfuse to scrape additional labels from GCP resources.

# Create service account
gcloud iam service-accounts create kfuse-gcp-enrichment \
    --display-name="Kloudfuse GCP Enrichment" \
    --description="Service account for Kloudfuse GCP metrics enrichment"

# Assign Monitoring Viewer role for metrics collection
gcloud projects add-iam-policy-binding YOUR-PROJECT-ID \
    --member="serviceAccount:kfuse-gcp-enrichment@YOUR-PROJECT-ID.iam.gserviceaccount.com" \
    --role="roles/monitoring.viewer"

# Assign Compute Viewer role for GCE instance label enrichment
gcloud projects add-iam-policy-binding YOUR-PROJECT-ID \
    --member="serviceAccount:kfuse-gcp-enrichment@YOUR-PROJECT-ID.iam.gserviceaccount.com" \
    --role="roles/compute.viewer"

# Create JSON key file
gcloud iam service-accounts keys create kfuse-gcp-credentials.json \
    --iam-account=kfuse-gcp-enrichment@YOUR-PROJECT-ID.iam.gserviceaccount.com

Required IAM Role Permissions

Enable GCP Metrics Enrichment

To enable automatic enrichment of GCP metrics with resource labels, add the following configuration to your helm values:

Global Configuration

Add the GCP configuration under the global section:

global:
  enrichmentEnabled: (1)
    - gcp
  gcpConfig:
    secretName: "kfuse-sd-secret" (2)

Ingester Configuration

Configure the ingester service to scrape and enrich GCP resources:

ingester:
  config:
    gcpScrapeIntervalMinutes: 30 (1)
    gcpProjectConfigs:
      - projectId: "YOUR-PROJECT-ID" (2)
        zones: (3)
          - "us-west1-b"
          - "us-central1-a"
          - "europe-west1-b"
        services: (4)
          - "GCE"
    gcpZones: (5)
      - "us-west1-b"
      - "us-central1-a"
      - "europe-west1-b"
    gcpServices: (6)
      - "GCE"

Multi-Project Configuration

To enable enrichment across multiple GCP projects, configure multiple project entries under gcpProjectConfigs:

ingester:
  config:
    gcpScrapeIntervalMinutes: 30
    gcpProjectConfigs:
      - projectId: "project-1" (1)
        zones: ["us-west1-b"] (2)
        services: ["GCE"] (3)
      - projectId: "project-2" (4)
        zones: ["europe-west1-b"] (5)
        services: ["GCE"] (6)