# AWS CloudWatch Metrics Integration

To fully set up ingestion of metrics from AWS to Kloudfuse through CloudWatch, complete these tasks:

- [Configure AWS Kinesis Firehose](https://docs.kloudfuse.com/platform/3.5.0/cloud-service-aws-cloudwatch-metrics/#firehose)

- [Configure AWS CloudWatch Metrics Stream](https://docs.kloudfuse.com/platform/3.5.0/cloud-service-aws-cloudwatch-metrics/#stream)

- [Enable AutoScaling Group Metrics](https://docs.kloudfuse.com/platform/3.5.0/cloud-service-aws-cloudwatch-metrics/#auto-scaling)

- [Enable Collection of Request Metrics in S3](https://docs.kloudfuse.com/platform/3.5.0/cloud-service-aws-cloudwatch-metrics/#s3)

- [Enable Enrichment of AWS Metrics](https://docs.kloudfuse.com/platform/3.5.0/cloud-service-aws-cloudwatch-metrics/#enrichment)

- [Reduce Cost of Metrics Ingestion](https://docs.kloudfuse.com/platform/3.5.0/cloud-service-aws-cloudwatch-metrics/#cost)

## Configure AWS Kinesis Firehose

|     |     |
| --- | --- |
|  | Use different Firehose accounts for logs and metrics. |

Create a new delivery stream in the account that emits the metrics, in the Kinesis Firehose AWS console.

Specify the following attribute values:

Source

Direct PUT

Destination

HTTP Endpoint

Destination settings

Provide the external-facing endpoint of the Kloudfuse cluster as part of the following URL address format:

```
https://<external facing endpoint of Kfuse cluster>/ingester/kinesis/metrics
```

Access token key

Provide when required

Content encoding

`GZIP`

1. Provide an existing S3 bucket, or create a new one for storing Kinesis records as a backup.

Backing up only failed data should be sufficient.

2. Change the name of the stream, as necessary.

## Configure AWS CloudWatch Metrics Stream

In the account that emits the metrics, in the Cloudwatch AWS console, navigate to the **Metrics** section on the left side of the console, select **Streams**, and create a new metric stream.

1. Select the metric namespaces to send to the stream; the default is all metrics.

2. In the configuration section, select an existing Firehose owned by your account, and the select the Kinesis Firehose you created earlier.

3. Under **Change Output Format**, make sure to select JSON for the output format.

4. Change the name of the stream if necessary.

## Enable AutoScaling Group Metrics

Perform these steps in the account that emits the metrics.

1. Open the [Amazon EC2 console](https://console.aws.amazon.com/ec2/).

2. Choose **Auto Scaling Groups** from the navigation pane.

3. Enable the checkbox next to your Auto Scaling group.

A split pane opens up at the bottom of the page.

4. On the **Monitoring** tab, select the Auto Scaling group metrics collection, and enable the checkbox located under **Auto Scaling**, at the top of the page.

## Enable Collection of Request Metrics in S3

In the account that emits the metrics, follow the instructions in AWS documentation for [Creating a CloudWatch metrics configuration for all the objects in your bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configure-request-metrics-bucket.html).

## Enable Enrichment of AWS Metrics

The metrics sent by AWS CloudWatch to the Kinesis Firehose include minimal labels. Kloudfuse enables you to attach more labels and user-defined custom tags to the ingested metrics, from _within_ the AWS console, by scraping AWS.

To enable this enrichment of AWS metrics, follow these steps:

1. **Modify yaml** in the global section of the `custom-values.yaml` file:

```yaml
global:
     enrichmentEnabled:
    - aws
```

2. **Create IAM scraper role** in the AWS account where the services that emit the metrics run.

Attach the following policy, for Kloudfuse to scrape the additional labels from AWS. See AWS documentation [Define custom IAM permissions with customer managed policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html)

```yaml
			"Action": [\
				"acm:ListCertificates",\
				"acm:ListTagsForCertificate",\
				"apigateway:GET",\
				"athena:ListWorkGroups",\
				"athena:ListTagsForResource",\
				"autoscaling:DescribeAutoScalingGroups",\
				"bedrock:ListFoundationModels",\
				"bedrock:ListTagsForResource",\
				"cloudwatch:ListMetrics",\
				"cloudwatch:GetMetricStatistics",\
				"dynamodb:ListTables",\
				"dynamodb:DescribeTable",\
				"dynamodb:ListTagsOfResource",\
				"ec2:DescribeInstances",\
				"ec2:DescribeInstanceStatus",\
				"ec2:DescribeSecurityGroups",\
				"ec2:DescribeNatGateways",\
				"ec2:DescribeVolumes",\
				"ecs:ListClusters",\
				"ecs:ListContainerInstances",\
				"ecs:ListServices",\
				"ecs:DescribeContainerInstances",\
				"ecs:DescribeServices",\
				"ecs:ListTagsForResource",\
				"elasticache:DescribeCacheClusters",\
				"elasticache:DescribeServerlessCaches",\
				"elasticache:ListTagsForResource",\
				"elasticfilesystem:DescribeFileSystems",\
				"elasticfilesystem:DescribeBackupPolicy",\
				"elasticloadbalancing:DescribeTags",\
				"elasticloadbalancing:DescribeLoadBalancers",\
				"es:ListDomainNames",\
				"es:DescribeDomains",\
				"es:ListTags",\
				"events:ListRules",\
				"events:ListTagsForResource",\
				"events:ListEventBuses",\
				"firehose:DescribeDeliveryStream",\
				"firehose:ListDeliveryStreams",\
				"firehose:ListTagsForDeliveryStream",\
				"fsx:DescribeFileSystems",\
				"fsx:ListTagsForResource",\
				"glue:ListJobs",\
				"glue:GetTags",\
				"kafka:ListTagsForResource",\
				"kafka:ListClustersV2",\
				"kinesis:ListStreams",\
				"kinesis:ListTagsForStream",\
				"kinesis:DescribeStream",\
				"lambda:GetPolicy",\
				"lambda:List*",\
				"lambda:ListTags",\
				"logs:DescribeLogGroups",\
				"logs:ListTagsForResource",\
				"logs:ListTagsLogGroup",\
				"mq:ListBrokers",\
				"mq:DescribeBroker",\
				"mediaconvert:ListQueues",\
				"mediaconvert:ListTagsForResource",\
				"qbusiness:ListApplications",\
				"qbusiness:GetApplication",\
				"qbusiness:ListTagsForResource",\
				"rds:DescribeDBInstances",\
				"rds:DescribeDBClusters",\
				"rds:ListTagsForResource",\
				"rds:DescribeEvents",\
				"redshift:DescribeClusters",\
				"redshift:DescribeTags",\
				"route53:ListHealthChecks",\
				"route53:ListTagsForResource",\
				"s3:ListAllMyBuckets",\
				"s3:GetBucketTagging",\
				"ses:ListConfigurationSets",\
				"ses:GetConfigurationSet",\
				"ses:ListTagsForResource",\
				"sns:ListTagsForResource",\
				"sns:ListTopics",\
				"sqs:ListQueues",\
				"sqs:ListQueueTags",\
				"states:ListStateMachines",\
				"states:ListActivities",\
				"states:ListTagsForResource",\
				"timestream:ListDatabases",\
				"timestream:ListTables",\
				"timestream:DescribeDatabase",\
				"timestream:DescribeTable",\
				"timestream:ListTagsForResource",\
				"wafv2:ListWebACLs",\
				"wafv2:ListRuleGroups",\
				"wafv2:ListTagsForResource",\
				"cloudfront:ListDistributions",\
				"cloudfront:GetDistribution",\
				"cloudfront:ListTagsForResource"\
			]
```

3. Modify the Trust Relationship for the policy of the scrape role ARN to add the node-group (Node IAM Role ARN), in which Kloudfuse is running on, as the Principal on the Account.

```yaml
{
       "Version": "2012-10-17",
       "Statement": [\
           {\
               "Sid": "Statement1",\
               "Effect": "Allow",\
               "Principal": {\
                   "AWS": "arn:aws:iam::ACCOUNT-NUMBER:role/eksctl-XXXXX-nodegroup-ng-XXXXXX-NodeInstanceRole-XXXXXXXXXX"\
               },\
               "Action": "sts:AssumeRole"\
           }\
       ]
}
```

4. Ensure that the permissions map to the node pool that hosts the EKS cluster for Kloudfuse.

5. Enable Kloudfuse to consume the new role; there are two approaches: through [AWS credentials](https://docs.kloudfuse.com/platform/3.5.0/cloud-service-aws-cloudwatch-metrics/#aws) or through [Role ARNs](https://docs.kloudfuse.com/platform/3.5.0/cloud-service-aws-cloudwatch-metrics/#arn).

- AWS credentials
   - Role ARNs

**Add your AWS credentials as a secret, and use the secret in the `ingester config`**.

1. Retrieve your aws credentials; see [Configure tool authentication with AWS](https://docs.aws.amazon.com/powershell/latest/userguide/creds-idc.html).

2. In the Kloudfuse namespace, create a kube secret name `aws-access-key`, with keys `accessKey` and `secretKey`.

```console
kubectl create secret generic aws-access-key --from-literal=accessKey=<AWS_ACCESS_KEY_ID> --from-literal=secretKey=<AWS_SECRET_ACCESS_KEY>
```

3. Specify the `secretName` in the `custom-values.yaml` file.

```yaml
ingester:
  config:
    awsScraper:
      secretName: aws-access-key
```

4. By default, Kloudfuse attempts to scrape from all regions and all [AWS Services](https://docs.kloudfuse.com/platform/3.5.0/cloud-service-aws-supported-services/). Customize this by adding the following configuration in the `custom-values.yaml` file:

```yaml
ingester:
  config:
    awsScraper:
      secretName: aws-access-key
      namespaces:
    - <add namespace>
  regions:
    - <add region>
```

**Add Role ARNs in the ingester config**: This option enables you to scrape multiple AWS accounts.

1. Add the scraper Role ARNs that you created with the new permissions to the `awsRoleArns` list to your `custom-values.yaml` file.

```yaml
ingester:
  config:
    awsRoleArns:
    - role: <ADD ROLE ARN HERE>
```

2. By default, Kloudfuse attempts to scrape from all regions and all [AWS Namespaces](https://docs.kloudfuse.com/platform/3.5.0/cloud-service-aws-supported-services/). Customize this by adding the following configuration in the `custom-values.yaml` file:

```yaml
ingester:
  config:
    awsRoleArns:
      role: <ADD ROLE ARN HERE>
      namespaces:
    - <add namespace>
  regions:
    - <add region>
```

|     |     |
| --- | --- |
|  | For Global Services like AWS CloudFront, a scraper role in the `us-east-1` region is required.<br>Ensure that you create the scraper role in `us-east-1` and also configure a Firehose delivery stream and CloudWatch Metric stream in the `us-east-1` region for global services.<br>```yaml<br>ingester:<br>  config:<br>    awsRoleArns:<br>      - role: <ADD US-EAST-1 ROLE ARN HERE><br>        regions:<br>          - us-east-1<br>```<br>|

1. To modify the node-group IAM role where Kloudfuse Platform runs, add the following permissions policy to the node-group (Node IAM Role ARN) to assume the role.

```yaml
{
       "Version": "2012-10-17",
       "Statement": [\
           {\
               "Effect": "Allow",\
               "Action": "sts:AssumeRole",\
               "Resource": <REPLACE SCRAPER ROLE ARN HERE>\
           }\
       ]
}
```

2. Complete a helm upgrade to save the changes.

```console
helm upgrade --create-namespace --install kfuse . -f <custom_values.yaml>
```

## AWS Namespace Enrichment

Kloudfuse enriches metadata for metrics from the following AWS services. Each service includes all AWS tags plus the specific metadata fields listed below:

| AWS Service | Enriched Metadata Fields |
| --- | --- |
| AutoScaling | All AWS tags on the AutoScaling Group |
| Firehose | All AWS tags on the Delivery Stream |
| RDS | **For DB Instances:**<br>- `allocatedstoragegb`, `availability_zone`, `backupretentionperioddays`<br>  <br>- `dbinstancearn`, `dbinstanceidentifier`, `dbinstanceclass`, `dbiresourceid`, `dbname`<br>  <br>- `engine`, `engineversion`, `multiaz`, `networktype`, `publicly_accessible`<br>  <br>- `secondary_availability_zone`, `storagetype`<br>  <br>- `host` (DbiResourceId), `hostname` (Endpoint Address)<br>  <br>**For DB Clusters:**<br>- `allocatedstoragegb`, `availability_zones`, `backupretentionperioddays`<br>  <br>- `dbclusterarn`, `dbclusterresourceid`, `databasename`<br>  <br>- `engine`, `enginemode`, `engineversion`, `global_write_forwarding_status`<br>  <br>- `multiaz`, `networktype`, `storagetype`<br>  <br>- All AWS tags |
| EKS | - `arn`, `cluster_name`, `endpoint`, `platform_version`, `role_arn`, `status`, `kube_server_version`<br>  <br>- All AWS tags |
| EBS | - `availability_zone`, `multiattachenabled`, `outpostarn`, `size`, `snapshotid`, `state`<br>  <br>- `throughput`, `volumeid`, `volume_type`, `volume_name`<br>  <br>- `device` (if attached)<br>  <br>- All AWS tags |
| EC2 | **For Instances:**<br>- `availability_zone`, `image_id`, `instance_id`, `instance_type`, `kernel`<br>  <br>- `iam_profile` (ARN), `host` (instance ID), `autoscaling_group`, `service`<br>  <br>- All AWS tags<br>  <br>**For NAT Gateways:**<br>- `natgatewayid`<br>  <br>- All AWS tags |
| ELB | - `canonicalhostedzonename`, `canonicalhostedzonenameid`, `dnsname`, `loadbalancername`, `scheme`, `vpcid`<br>  <br>- `host` (CanonicalHostedZoneName), `hostname` (CanonicalHostedZoneName), `name` (LoadBalancerName)<br>  <br>- All AWS tags |
| MQ | - `brokerarn`, `brokerid`, `brokername`, `brokerstate`, `deploymentmode`<br>  <br>- `enginetype`, `engineversion`, `hostinstancetype`, `storagetype`<br>  <br>- All AWS tags |
| S3 | All AWS tags on the S3 Bucket |
| EFS | - `filesystemarn`, `name`<br>  <br>- `aws_elasticfilesystem_default_backup` (enabled/disabled)<br>  <br>- All AWS tags |
| ELBv2 | - `loadbalancerarn`, `name` (LoadBalancerName), `host` (DNSName)<br>  <br>- All AWS tags |
| ACM | All AWS tags on the Certificate |
| ElastiCache | **For ElastiCache Clusters:**<br>- `cache_node_type`, `name` (CacheClusterId), `engine`, `engine_version`<br>  <br>- `preferred_availability_zone`, `replication_group`<br>  <br>- All AWS tags<br>  <br>**For Serverless Caches:**<br>- `name` (ServerlessCacheName), `engine`, `status`, `create_time` |
| CloudFront | All AWS tags on Distributions |
| Route53 | All AWS tags on Health Checks |
| SNS | All AWS tags on Topics |
| Redshift | All AWS tags on Clusters |
| OpenSearch | - `elasticsearch_version` (EngineVersion), `name` (DomainName), `dedicated_master_enabled`<br>  <br>- `instance_type`, `zone_awareness_enabled`, `ebs_enabled`<br>  <br>- All AWS tags |
| SQS | All AWS tags on Queues |
| Lambda | - `function_arn`, `functionname`, `memory_size`, `runtime`<br>  <br>- `architecture` (first in list), `storage_size` (EphemeralStorage Size)<br>  <br>- All AWS tags |
| DynamoDB | All AWS tags on Tables |
| ApiGateway | - `apiid` (Id)<br>  <br>- All AWS tags |
| ApiGatewayV2 | - `apiname` (Name)<br>  <br>- All AWS tags |
| Glue | All AWS tags on Jobs |
| Athena | All AWS tags on WorkGroups |
| ECS | **For Clusters and Services:**<br>- All AWS tags |
| EventBridge | **For Rules and Event Buses:**<br>- All AWS tags |
| Kafka | All AWS tags on Clusters |
| Kinesis | All AWS tags on Streams |
| Logs | All AWS tags on Log Groups |
| WAF | **For Web ACLs and Rule Groups:**<br>- All AWS tags |
| FSx | - `generation` (1 or 2, for ONTAP only), `file_system_type`<br>  <br>- All AWS tags |
| Bedrock | **For Foundation Models:**<br>- `model_name`, `provider_name`<br>  <br>- All AWS tags |
| QBusiness | **For Applications:**<br>- `display_name`<br>  <br>- All AWS tags |
| MediaConvert | **For Queues:**<br>- `name`, `status`, `type`<br>  <br>- All AWS tags on the Queue |
| States (Step Functions) | **For State Machines:**<br>- `name`, `type`<br>  <br>- All AWS tags on the State Machine<br>  <br>**For Activities:**<br>- `name`<br>  <br>- All AWS tags on the Activity |
| Timestream | **For Databases:**<br>- `database_name`, `arn`<br>  <br>- All AWS tags on the Database<br>  <br>**For Tables:**<br>- `table_name`, `database_name`, `arn`<br>  <br>- All AWS tags on the Table |
| SES | **For Configuration Sets:**<br>- `name` (ConfigurationSetName)<br>  <br>- `tls_policy` (DeliveryOptions.TlsPolicy)<br>  <br>- `sending_enabled` (SendingOptions.SendingEnabled)<br>  <br>- All AWS tags on the Configuration Set |

## Reduce Cost of Metrics Ingestion

AWS CloudWatch metrics ingestion can be a high-cost operation. The driving factor here is the AWS `CW:MetricsStreamUsage` attribute, especially the `MetricsUpdate` statistical aggregate.

To reduce the cost of operating CloudWatch metrics ingestion, consider these factors:

- Volume of Ingested Metrics

Control this by sending only the necessary Namespaces and metrics to the stream.

In other words, avoid selecting **_All Namespaces_** and **_All Metrics_** when configuring ingestion.

Some namespaces are very costly when deriving metrics. These include AWS NLB and AWS Lambda because they feature both a high volume of metrics and multiple dimensions.

- Data Retention

Our research indicates that you should modify the retention period of the CloudWatch metrics data by changing the retention setting for the log group of the firehose stream.

- Sampling Frequency

The frequency of data sampling by CloudWatch is controlled internally by the AWS CloudWatch implementation.