# AWS CloudWatch Metrics Integration

To fully set up ingestion of metrics from AWS to Kloudfuse through CloudWatch, complete these tasks:

- [Configure AWS Kinesis Firehose](https://docs.kloudfuse.com/platform/3.4.0.p1/cloud-service-aws-cloudwatch-metrics/#firehose)
- [Configure AWS CloudWatch Metrics Stream](https://docs.kloudfuse.com/platform/3.4.0.p1/cloud-service-aws-cloudwatch-metrics/#stream)
- [Enable AutoScaling Group Metrics](https://docs.kloudfuse.com/platform/3.4.0.p1/cloud-service-aws-cloudwatch-metrics/#auto-scaling)
- [Enable Collection of Request Metrics in S3](https://docs.kloudfuse.com/platform/3.4.0.p1/cloud-service-aws-cloudwatch-metrics/#s3)
- [Enable Enrichment of AWS Metrics](https://docs.kloudfuse.com/platform/3.4.0.p1/cloud-service-aws-cloudwatch-metrics/#enrichment)
- [Reduce Cost of Metrics Ingestion](https://docs.kloudfuse.com/platform/3.4.0.p1/cloud-service-aws-cloudwatch-metrics/#cost)

## Configure AWS Kinesis Firehose

|     |     |
| --- | --- |
|  | Use different Firehose accounts for logs and metrics. |

Create a new delivery stream in the account that emits the metrics, in the Kinesis Firehose AWS console.

Specify the following attribute values:

**Source**  
Direct PUT  
**Destination**  
HTTP Endpoint  
**Destination settings**  
Provide the external-facing endpoint of the Kloudfuse cluster as part of the following URL address format:

```
https://<external facing endpoint of Kfuse cluster>/ingester/kinesis/metrics
```

**Access token key**  
Provide when required  
**Content encoding**  
`GZIP`

1. Provide an existing S3 bucket, or create a new one for storing Kinesis records as a backup.

Backing up only failed data should be sufficient.

2. Change the name of the stream, as necessary.

## Configure AWS CloudWatch Metrics Stream

In the account that emits the metrics, in the Cloudwatch AWS console, navigate to the **Metrics** section on the left side of the console, select **Streams**, and create a new metric stream.

1. Select the metric namespaces to send to the stream; the default is all metrics.

2. In the configuration section, select an existing Firehose owned by your account, and the select the Kinesis Firehose you created earlier.

3. Under **Change Output Format**, make sure to select JSON for the output format.

4. Change the name of the stream if necessary.

## Enable AutoScaling Group Metrics

Perform these steps in the account that emits the metrics.

1. Open the [Amazon EC2 console](https://console.aws.amazon.com/ec2/).

2. Choose **Auto Scaling Groups** from the navigation pane.

3. Enable the checkbox next to your Auto Scaling group.

A split pane opens up at the bottom of the page.

4. On the **Monitoring** tab, select the Auto Scaling group metrics collection, and enable the checkbox located under **Auto Scaling**, at the top of the page.

## Enable Collection of Request Metrics in S3

In the account that emits the metrics, follow the instructions in AWS documentation for [Creating a CloudWatch metrics configuration for all the objects in your bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configure-request-metrics-bucket.html).

## Enable Enrichment of AWS Metrics

The metrics sent by AWS CloudWatch to the Kinesis Firehose include minimal labels. Kloudfuse enables you to attach more labels and user-defined custom tags to the ingested metrics, from _within_ the AWS console, by scraping AWS.

To enable this enrichment of AWS metrics, follow these steps:

1. **Modify yaml** in the global section of the `custom-values.yaml` file:

```yaml
global:
     enrichmentEnabled:
    - aws
```

2. **Create IAM scraper role** in the AWS account where the services that emit the metrics run.

Attach the following policy, for Kloudfuse to scrape the additional labels from AWS. See AWS documentation [Define custom IAM permissions with customer managed policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html)

Create a scraper role with custom policies

```yaml
{"Version": "2012-10-17",
"Statement": [
{  
"Sid": "Statement1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNT-NUMBER:role/eksctl-XXXXX-nodegroup-ng-XXXXXX-NodeInstanceRole-XXXXXXXXXX"
},
"Action": "sts:AssumeRole"
}
]}
```

3. Modify the Trust Relationship for the policy of the scrape role ARN to add the node-group (Node IAM Role ARN), in which Kloudfuse is running on.

4. Ensure that the permissions map to the node pool that hosts the EKS cluster for Kloudfuse.

5. Enable Kloudfuse to consume the new role; there are two approaches: through [AWS credentials](https://docs.kloudfuse.com/platform/3.4.0.p1/cloud-service-aws-cloudwatch-metrics/#aws) or through [Role ARNs](https://docs.kloudfuse.com/platform/3.4.0.p1/cloud-service-aws-cloudwatch-metrics/#arn).

- AWS credentials
   - Role ARNs

**Add your AWS credentials as a secret, and use the secret in the `ingester config`**.

1. Retrieve your aws credentials; see [Configure tool authentication with AWS](https://docs.aws.amazon.com/powershell/latest/userguide/creds-idc.html).

2. In the Kloudfuse namespace, create a kube secret name `aws-access-key`, with keys `accessKey` and `secretKey`.

```console
kubectl create secret generic aws-access-key --from-literal=accessKey=<AWS_ACCESS_KEY_ID> --from-literal=secretKey=<AWS_SECRET_ACCESS_KEY>
```

3. Specify the `secretName` in the `custom-values.yaml` file.

```yaml
ingester:
  config:
    awsScraper:
      secretName: aws-access-key
```

4. By default, Kloudfuse attempts to scrape from all regions and all [AWS Services](https://docs.kloudfuse.com/platform/3.4.0.p1/cloud-service-aws-supported-services/). Customize this by adding the following configuration in the `custom-values.yaml` file:

```yaml
ingester:
  config:
    awsScraper:
      secretName: aws-access-key
      namespaces:
    - <add namespace>
regions:
  - <add region>
```

**Add Role ARNs in the ingester config**: This option enables you to scrape multiple AWS accounts.

1. Add the scraper Role ARNs that you created with the new permissions to the `awsRoleArns` list to your `custom-values.yaml` file.

```yaml
ingester:
  config:
    awsRoleArns:
  - role: <ADD ROLE ARN HERE>
```

2. By default, Kloudfuse attempts to scrape from all regions and all [AWS Namespaces](https://docs.kloudfuse.com/platform/3.4.0.p1/cloud-service-aws-supported-services/). Customize this by adding the following configuration in the `custom-values.yaml` file:

```yaml
ingester:
  config:
    awsRoleArns:
      role: <ADD ROLE ARN HERE>
      namespaces:
    - <add namespace>
regions:
  - <add region>
```

|     |     |
| --- | --- |
|  | For Global Services like AWS CloudFront, a scraper role in the `us-east-1` region is required.<br>Ensure that you create the scraper role in `us-east-1` and also configure a Firehose delivery stream and CloudWatch Metric stream in the `us-east-1` region for global services.<br>```yaml
ingester:
  config:
    awsRoleArns:
      - role: <ADD US-EAST-1 ROLE ARN HERE>
        regions:
          - us-east-1
``` |

1. To modify the node-group IAM role where Kloudfuse Platform runs, add the following permissions policy to the node-group (Node IAM Role ARN) to assume the role.

```yaml
{"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": <REPLACE SCRAPER ROLE ARN HERE>
}
]
}
```

2. Complete a helm upgrade to save the changes.

```console
helm upgrade --create-namespace --install kfuse . -f <custom_values.yaml>
```

## Reduce Cost of Metrics Ingestion

AWS CloudWatch metrics ingestion can be a high-cost operation. The driving factor here is the AWS `CW:MetricsStreamUsage` attribute, especially the `MetricsUpdate` statistical aggregate.

To reduce the cost of operating CloudWatch metrics ingestion, consider these factors:

- **Volume of Ingested Metrics**  
Control this by sending only the necessary Namespaces and metrics to the stream. Avoid selecting **_All Namespaces_** and **_All Metrics_** when configuring ingestion.

- **Data Retention**  
Modify the retention period of the CloudWatch metrics data by changing the retention setting for the log group of the firehose stream.

- **Sampling Frequency**  
The frequency of data sampling by CloudWatch is controlled internally by the AWS CloudWatch implementation.