# GCP Metrics Collection

Kloudfuse integrates with the GCP cloud service to collect metrics.

Complete these tasks to successfully collect metrics:

- [Create Credentials File](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/gcp/metrics/#credentials)
- [Create Kubernetes Secret](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/gcp/metrics/#secret)
- [Configure Kloudfuse to Collect Metrics](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/gcp/metrics/#collect)
  - [Enable Cloud Exporter](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/gcp/metrics/#enable-cloud-exporter)
  - [Configure GCP Metrics Collection](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/gcp/metrics/#configure-gcp-metrics-collection)
    - [Method 1: Using Kubernetes Secret (Recommended for Production)](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/gcp/metrics/#method-1-using-kubernetes-secret-recommended-for-production)
    - [Method 2: Using Direct secretKey (Quick Setup Only)](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/gcp/metrics/#method-2-using-direct-secretkey-quick-setup-only)
- [Configure GCP Metrics Enrichment](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/gcp/metrics/#enrichment)
  - [Prerequisites](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/gcp/metrics/#prerequisites)
  - [Enable GCP Metrics Enrichment](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/gcp/metrics/#enable-gcp-metrics-enrichment)
  - [Multi-Project Configuration](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/gcp/metrics/#multi-project-configuration)

## Create Credentials File

1. In the **Google Cloud** console, navigate to **Menu > IAM & Admin > Service Accounts**.
2. Select your service account, or create a new one.
   
   Ensure that the service account has **_Monitoring Viewer_** permissions to collect GCP metrics.  
3. Select **Keys > Add key > Create new key**.
4. Select JSON, then click **Create**.
5. Click **Close**.

## Create Kubernetes Secret

|     |     |
| --- | --- |
|     | The credentials file must be named `credentials.json` when creating the Kubernetes secret. |

Create a Kubernetes secret from the credentials file:

```console
kubectl create secret generic kfuse-gcp-credentials --from-file=credentials.json -n kfuse
```

|     |     |
| --- | --- |
|     | This is the same secret used by [GCP Logs Collection](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/gcp/logs/#create-kubernetes-secret). If you already created it for Pub/Sub log ingestion, reuse it here — just ensure the service account also has the `roles/monitoring.viewer` and `roles/compute.viewer` roles. |

## Configure Kloudfuse to Collect Metrics

### Enable Cloud Exporter

First, enable the cloud exporter service in the global configuration:

```yaml
global:
  cloud-exporter:
    enabled: true
  gcpConfig:
    secretName: "kfuse-gcp-credentials"
```

### Configure GCP Metrics Collection

There are two methods to provide GCP credentials for metrics collection:

### Method 1: Using Kubernetes Secret (Recommended for Production)

This is the recommended approach for production environments as it keeps sensitive credentials secure.

#### Single Project Configuration

1. Update the helm values using the following configuration for a single project.

```yaml
kfuse-cloud-exporter:
     prometheus-stackdriver-exporter:
       enabled: true
       stackdriver:
         httpTimeout: 30s
         maxRetries: 3
         projectId: "YOUR-PROJECT-ID" (1)
         metrics:
           typePrefixes: "compute.googleapis.com"
```

|     |     |
| --- | --- |
| **1** | `projectId`: GCP project ID to monitor |

#### Multi-Project Configuration

For monitoring multiple GCP projects, use one of the following approaches:

**Recommended Approach (Clean Array Format)**

```yaml
kfuse-cloud-exporter:
  prometheus-stackdriver-exporter:
    enabled: true
    stackdriver:
      httpTimeout: 30s
      maxRetries: 3
      metrics:
        typePrefixes: "compute.googleapis.com"
    config:
      project_ids: (1)
        - project-1
        - project-2
        - project-3
```

|     |     |
| --- | --- |
| **1** | `project_ids`: Array of GCP project IDs to monitor (recommended for multiple projects) |

**Alternative Approach (Backward Compatible)**

```yaml
kfuse-cloud-exporter:
  prometheus-stackdriver-exporter:
    enabled: true
    stackdriver:
      httpTimeout: 30s
      maxRetries: 3
      projectId: "project-1,project-2,project-3" (1)
      metrics:
        typePrefixes: "compute.googleapis.com"
```

|     |     |
| --- | --- |
| **1** | `projectId`: Comma-separated list of GCP project IDs (backward compatible format) |

### Method 2: Using Direct secretKey (Quick Setup Only)

|     |     |
| --- | --- |
|     | This method embeds credentials directly in the configuration file. Only use this for quick testing or proof-of-concept setups. Never use this approach in production environments. |

1. Convert the JSON credentials file to base64:

```console
base64 -w 0 credentials.json
```

2. Update the helm values with the base64-encoded credentials:

**Single Project**

```yaml
kfuse-cloud-exporter:
     prometheus-stackdriver-exporter:
       enabled: true
       secretKey: "<BASE64_ENCODED_CREDENTIALS>" (1)
       stackdriver:
         httpTimeout: 30s
         maxRetries: 3
         projectId: "YOUR-PROJECT-ID" (2)
         metrics:
           typePrefixes: "compute.googleapis.com"
```

|     |     |
| --- | --- |
| **1** | `secretKey`: Base64-encoded service account JSON credentials (not recommended for production) |
| **2** | `projectId`: GCP project ID to monitor |

## Configure GCP Metrics Enrichment

Kloudfuse can enrich GCP metrics with custom labels and metadata from your GCP resources. This feature adds resource labels (like custom tags) from GCE instances to the corresponding metrics, providing better context for monitoring and alerting.

### Prerequisites

- GCP credentials with appropriate permissions for resource enrichment
- The same Kubernetes secret (`kfuse-gcp-credentials`) created in the previous steps

#### Create IAM Role and Service Account

Create a service account in your GCP project with the required permissions for Kloudfuse to scrape additional labels from GCP resources.

```console
# Create service account
gcloud iam service-accounts create kfuse-gcp-enrichment \
    --display-name="Kloudfuse GCP Enrichment" \
    --description="Service account for Kloudfuse GCP metrics enrichment"

# Assign Monitoring Viewer role for metrics collection
gcloud projects add-iam-policy-binding YOUR-PROJECT-ID \
    --member="serviceAccount:kfuse-gcp-enrichment@YOUR-PROJECT-ID.iam.gserviceaccount.com" \
    --role="roles/monitoring.viewer"

# Assign Compute Viewer role for GCE instance label enrichment
gcloud projects add-iam-policy-binding YOUR-PROJECT-ID \
    --member="serviceAccount:kfuse-gcp-enrichment@YOUR-PROJECT-ID.iam.gserviceaccount.com" \
    --role="roles/compute.viewer"

# Create JSON key file
gcloud iam service-accounts keys create kfuse-gcp-credentials.json \
    --iam-account=kfuse-gcp-enrichment@YOUR-PROJECT-ID.iam.gserviceaccount.com
```

### Required IAM Role Permissions

The following table lists the IAM roles required for the service account:

| IAM Role | Purpose | Required For |
| --- | --- | --- |
| `roles/monitoring.viewer` | Provides read-only access to GCP monitoring metrics | Metrics collection (required for all configurations) |
| `roles/compute.viewer` | Provides read-only access to GCE instance metadata and labels | Metrics enrichment (required only when enrichment is enabled) |

### Enable GCP Metrics Enrichment

To enable automatic enrichment of GCP metrics with resource labels, add the following configuration to your helm values:

#### Global Configuration

Add the GCP configuration under the `global` section:

```yaml
global:
  enrichmentEnabled: (1)
    - gcp
  gcpConfig:
    secretName: "kfuse-gcp-credentials" (2)
```

|     |     |
| --- | --- |
| **1** | `enrichmentEnabled`: Enable enrichment for cloud providers (include "gcp" to enable GCP metrics enrichment) |
| **2** | `secretName`: The Kubernetes secret containing GCP credentials (same as used for metrics collection) |

#### Ingester Configuration

Configure the ingester service to scrape and enrich GCP resources:

```yaml
ingester:
  config:
    gcpScrapeIntervalMinutes: 30 (1)
    gcpProjectConfigs:
      - projectId: "YOUR-PROJECT-ID" (2)
        zones: (3)
          - "us-west1-b"
          - "us-central1-a"
          - "europe-west1-b"
        services: (4)
          - "GCE"
    gcpZones: (5)
      - "us-west1-b"
      - "us-central1-a"
      - "europe-west1-b"
    gcpServices: (6)
      - "GCE"
```

|     |     |
| --- | --- |
| **1** | `gcpScrapeIntervalMinutes`: How often to scrape GCP resources for label enrichment (default: 30 minutes, minimum: 5 minutes) |
| **2** | `projectId`: GCP project ID to scrape resources from |
| **3** | `zones`: List of GCP zones/regions where your resources are deployed for this specific project |
| **4** | `services`: Services to enable enrichment for this specific project (currently supports "GCE" for Compute Engine) |
| **5** | `gcpZones`: Default list of zones used when zones are not specified for a project in gcpProjectConfigs |
| **6** | `gcpServices`: Default list of services used when services are not specified for a project in gcpProjectConfigs |

### Multi-Project Configuration

To enable enrichment across multiple GCP projects, configure multiple project entries under `gcpProjectConfigs`:

```yaml
ingester:
  config:
    gcpScrapeIntervalMinutes: 30
    gcpProjectConfigs:
      - projectId: "project-1" (1)
        zones: ["us-west1-b"] (2)
        services: ["GCE"] (3)
      - projectId: "project-2" (4)
        zones: ["europe-west1-b"] (5)
        services: ["GCE"] (6)
```

|     |     |
| --- | --- |
| **1** | First GCP project ID to scrape resources from |
| **2** | Zones to scrape in the first project |
| **3** | Services to enable for the first project |
| **4** | Second GCP project ID to scrape resources from |
| **5** | Zones to scrape in the second project |
| **6** | Services to enable for the second project |