# AWS CloudWatch Metrics Integration

To fully set up ingestion of metrics from AWS to Kloudfuse through CloudWatch, complete these tasks:

- [Configure AWS Kinesis Firehose](https://docs.kloudfuse.com/platform/3.3.6/cloud-service-aws-cloudwatch-metrics/#firehose)
- [Configure AWS CloudWatch Metrics Stream](https://docs.kloudfuse.com/platform/3.3.6/cloud-service-aws-cloudwatch-metrics/#stream)
- [Enable AutoScaling Group Metrics](https://docs.kloudfuse.com/platform/3.3.6/cloud-service-aws-cloudwatch-metrics/#auto-scaling)
- [Enable Collection of Request Metrics in S3](https://docs.kloudfuse.com/platform/3.3.6/cloud-service-aws-cloudwatch-metrics/#s3)
- [Enable Enrichment of AWS Metrics](https://docs.kloudfuse.com/platform/3.3.6/cloud-service-aws-cloudwatch-metrics/#enrichment)
- [Reduce Cost of Metrics Ingestion](https://docs.kloudfuse.com/platform/3.3.6/cloud-service-aws-cloudwatch-metrics/#cost)

## Configure AWS Kinesis Firehose

|     |     |
| --- | --- |
|  | Use different Firehose accounts for logs and metrics. |

Create a new delivery stream in the account that emits the metrics, in the Kinesis Firehose AWS console.

Specify the following attribute values:

- **Source**: Direct PUT
- **Destination**: HTTP Endpoint
- **Destination settings**: Provide the external-facing endpoint of the Kloudfuse cluster as part of the following URL address format:
  
  ```
  https://<external facing endpoint of Kfuse cluster>/ingester/kinesis/metrics
  ```
- **Access token key**: Provide when required
- **Content encoding**: `GZIP`

1. Provide an existing S3 bucket, or create a new one for storing Kinesis records as a backup. Backing up only failed data should be sufficient.
2. Change the name of the stream, as necessary.

## Configure AWS CloudWatch Metrics Stream

In the account that emits the metrics, in the Cloudwatch AWS console, navigate to the **Metrics** section on the left side of the console, select **Streams**, and create a new metric stream.

1. Select the metric namespaces to send to the stream; the default is all metrics.
2. In the configuration section, select an existing Firehose owned by your account, and select the Kinesis Firehose you created earlier.
3. Under **Change Output Format**, make sure to select JSON for the output format.
4. Change the name of the stream if necessary.

## Enable AutoScaling Group Metrics

Perform these steps in the account that emits the metrics.

1. Open the [Amazon EC2 console](https://console.aws.amazon.com/ec2/).
2. Choose **Auto Scaling Groups** from the navigation pane.
3. Enable the checkbox next to your Auto Scaling group.

A split pane opens up at the bottom of the page.

4. On the **Monitoring** tab, select the Auto Scaling group metrics collection, and enable the checkbox located under **Auto Scaling**, at the top of the page.

## Enable Collection of Request Metrics in S3

In the account that emits the metrics, follow the instructions in AWS documentation for [Creating a CloudWatch metrics configuration for all the objects in your bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configure-request-metrics-bucket.html).

## Enable Enrichment of AWS Metrics

The metrics sent by AWS CloudWatch to the Kinesis Firehose include minimal labels. Kloudfuse enables you to attach more labels and user-defined custom tags to the ingested metrics, from _within_ the AWS console, by scraping AWS.

To enable this enrichment of AWS metrics, follow these steps:

1. **Modify yaml** in the global section of the `custom-values.yaml` file:

```yaml
   global:
        enrichmentEnabled:
       - aws
   ```

2. **Create IAM scraper role** in the AWS account where the services that emit the metrics run.

Attach the following policy, for Kloudfuse to scrape the additional labels from AWS.

Create a scraper role with custom policies

```yaml
   "Action": [\
       "acm:ListCertificates",\
       "acm:ListTagsForCertificate",\
       "apigateway:GET",\
       "athena:ListWorkGroups",\
       "athena:ListTagsForResource",\
       "autoscaling:DescribeAutoScalingGroups",\
       "cloudwatch:ListMetrics",\
       "cloudwatch:GetMetricStatistics",\
       "dynamodb:ListTables",\
       "dynamodb:DescribeTable",\
       "dynamodb:ListTagsOfResource",\
       "ec2:DescribeInstances",\
       "ec2:DescribeInstanceStatus",\
       "ec2:DescribeSecurityGroups",\
       "ec2:DescribeNatGateways",\
       "ec2:DescribeVolumes",\
       "ecs:ListClusters",\
       "ecs:ListContainerInstances",\
       "ecs:ListServices",\
       "ecs:DescribeContainerInstances",\
       "ecs:DescribeServices",\
       "ecs:ListTagsForResource",\
       "elasticache:DescribeCacheClusters",\
       "elasticache:ListTagsForResource",\
       "elasticfilesystem:DescribeFileSystems",\
       "elasticfilesystem:DescribeBackupPolicy",\
       "elasticloadbalancing:DescribeTags",\
       "elasticloadbalancing:DescribeLoadBalancers",\
       "es:ListDomainNames",\
       "es:DescribeDomains",\
       "es:ListTags",\
       "events:ListRules",\
       "events:ListTagsForResource",\
       "events:ListEventBuses",\
       "firehose:DescribeDeliveryStream",\
       "firehose:ListDeliveryStreams",\
       "firehose:ListTagsForDeliveryStream",\
       "fsx:DescribeFileSystems",\
       "fsx:ListTagsForResource",\
       "glue:ListJobs",\
       "glue:GetTags",\
       "kafka:ListTagsForResource",\
       "kafka:ListClustersV2",\
       "kinesis:ListStreams",\
       "kinesis:ListTagsForStream",\
       "kinesis:DescribeStream",\
       "lambda:GetPolicy",\
       "lambda:List*",\
       "lambda:ListTags",\
       "logs:DescribeLogGroups",\
       "logs:ListTagsForResource",\
       "logs:ListTagsLogGroup",\
       "mq:ListBrokers",\
       "mq:DescribeBroker",\
       "rds:DescribeDBInstances",\
       "rds:ListTagsForResource",\
       "rds:DescribeEvents",\
       "redshift:DescribeClusters",\
       "redshift:DescribeTags",\
       "route53:ListHealthChecks",\
       "route53:ListTagsForResource",\
       "s3:ListAllMyBuckets",\
       "s3:GetBucketTagging",\
       "sns:ListTagsForResource",\
       "sns:ListTopics",\
       "sqs:ListQueues",\
       "sqs:ListQueueTags",\
       "wafv2:ListWebACLs",\
       "wafv2:ListRuleGroups",\
       "wafv2:ListTagsForResource"\
   ]
   ```

3. Modify the Trust Relationship for the policy of the scrape role ARN to add the node-group (Node IAM Role ARN), in which Kloudfuse is running on:

```yaml
   {
          "Version": "2012-10-17",
          "Statement": [\
              {\
                  "Sid": "Statement1",\
                  "Effect": "Allow",\
                  "Principal": {\
                      "AWS": "arn:aws:iam::ACCOUNT-NUMBER:role/eksctl-XXXXX-nodegroup-ng-XXXXXX-NodeInstanceRole-XXXXXXXXXX"\
                  },\
                  "Action": "sts:AssumeRole"\
              }\
          ]
   }
   ```

4. Ensure that the permissions map to the node pool that hosts the EKS cluster for Kloudfuse.
5. Enable Kloudfuse to consume the new role; there are two approaches: through [AWS credentials](https://docs.kloudfuse.com/platform/3.3.6/cloud-service-aws-cloudwatch-metrics/#aws) or through [Role ARNs](https://docs.kloudfuse.com/platform/3.3.6/cloud-service-aws-cloudwatch-metrics/#arn).

- AWS credentials
   
   - Role ARNs

**Add your AWS credentials as a secret, and use the secret in the `ingester config`**.

1. Retrieve your aws credentials; see [Configure tool authentication with AWS](https://docs.aws.amazon.com/powershell/latest/userguide/creds-idc.html).
2. In the Kloudfuse namespace, create a kube secret name `aws-access-key`, with keys `accessKey` and `secretKey`.

```console
   kubectl create secret generic aws-access-key --from-literal=accessKey=<AWS_ACCESS_KEY_ID> --from-literal=secretKey=<AWS_SECRET_ACCESS_KEY>
   ```

3. Specify the `secretName` in the `custom-values.yaml` file.

```yaml
   ingester:
     config:
       awsScraper:
         secretName: aws-access-key
   ```

4. By default, Kloudfuse attempts to scrape from all regions and all [AWS Services](https://docs.kloudfuse.com/platform/3.3.6/cloud-service-aws-supported-services/). Customize this by adding the following configuration in the `custom-values.yaml` file:

```yaml
   ingester:
     config:
       awsScraper:
         secretName: aws-access-key
         namespaces:
       - <add namespace>
   regions:
     - <add region>
   ```

**Add Role ARNs in the ingester config**: This option enables you to scrape multiple AWS accounts.

1. Add the scraper Role ARNs that you created with the new permissions to the `awsRoleArns` list to your `custom-values.yaml` file.

```yaml
   ingester:
     config:
       awsRoleArns:
     - role: <ADD ROLE ARN HERE>
   ```

2. By default, Kloudfuse attempts to scrape from all regions and all [AWS Namespaces](https://docs.kloudfuse.com/platform/3.3.6/cloud-service-aws-supported-services/). Customize this by adding the following configuration in the `custom-values.yaml` file:

```yaml
   ingester:
     config:
       awsRoleArns:
         role: <ADD ROLE ARN HERE>
         namespaces:
       - <add namespace>
   regions:
     - <add region>
   ```

6. To modify the node-group IAM role where Kloudfuse Platform runs, add the following permissions policy to the node-group (Node IAM Role ARN) to assume the role.

```yaml
   {
          "Version": "2012-10-17",
          "Statement": [\
              {\
                  "Effect": "Allow",\
                  "Action": "sts:AssumeRole",\
                  "Resource": <REPLACE SCRAPER ROLE ARN HERE>\
              }\
          ]
   }
   ```

7. Complete a helm upgrade to save the changes.

```console
   helm upgrade --create-namespace --install kfuse . -f <custom_values.yaml>
   ```

## Reduce Cost of Metrics Ingestion

AWS CloudWatch metrics ingestion can be a high-cost operation. The driving factor here is the AWS `CW:MetricsStreamUsage` attribute, especially the `MetricsUpdate` statistical aggregate.

To reduce the cost of operating CloudWatch metrics ingestion, consider these factors:

- **Volume of Ingested Metrics**: Control this by sending only the necessary Namespaces and metrics to the stream. In other words, avoid selecting **_All Namespaces_** and **_All Metrics_** when configuring ingestion. Some namespaces are very costly when deriving metrics. These include AWS NLB and AWS Lambda because they feature both a high volume of metrics, and multiple dimensions.
- **Data Retention**: Our research indicates that you should modify the retention period of the CloudWatch metrics data by changing the retention setting for the log group of the firehose stream.
- **Sampling Frequency**: The frequency of data sampling by CloudWatch is controlled internally by the AWS CloudWatch implementation.