# AWS CloudWatch Metrics Integration

To fully set up ingestion of metrics from AWS to Kloudfuse through CloudWatch, complete these tasks:

- [Configure AWS Kinesis Firehose](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/aws/cloudwatch-metrics/#firehose)
- [Configure AWS CloudWatch Metrics Stream](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/aws/cloudwatch-metrics/#stream)
- [Enable AutoScaling Group Metrics](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/aws/cloudwatch-metrics/#auto-scaling)
- [Enable Collection of Request Metrics in S3](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/aws/cloudwatch-metrics/#s3)
- [Enable Enrichment of AWS Metrics](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/aws/cloudwatch-metrics/#enrichment)
- [Reduce Cost of Metrics Ingestion](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/aws/cloudwatch-metrics/#cost)

## Configure AWS Kinesis Firehose

|     |     |
| --- | --- |
|  | Use different Firehose accounts for logs and metrics. |

Create a new delivery stream in the account that emits the metrics, in the Kinesis Firehose AWS console.

Specify the following attribute values:

Source

Direct PUT

Destination

HTTP Endpoint

Destination settings

Provide the external-facing endpoint of the Kloudfuse cluster as part of the following URL address format:

```
https://<external facing endpoint of Kfuse cluster>/ingester/kinesis/metrics
```

Access token key

Provide when required

Content encoding

`GZIP`

1. Provide an existing S3 bucket, or create a new one for storing Kinesis records as a backup.
    
   Backing up only failed data should be sufficient.

2. Change the name of the stream, as necessary.

## Configure AWS CloudWatch Metrics Stream

In the account that emits the metrics, in the Cloudwatch AWS console, navigate to the **Metrics** section on the left side of the console, select **Streams**, and create a new metric stream.

1. Select the metric namespaces to send to the stream; the default is all metrics.

2. In the configuration section, select an existing Firehose owned by your account, and the select the Kinesis Firehose you created earlier.

3. Under **Change Output Format**, make sure to select JSON for the output format.

4. Change the name of the stream if necessary.

## Enable AutoScaling Group Metrics

Perform these steps in the account that emits the metrics.

1. Open the [Amazon EC2 console](https://console.aws.amazon.com/ec2/).

2. Choose **Auto Scaling Groups** from the navigation pane.

3. Enable the checkbox next to your Auto Scaling group.

A split pane opens up at the bottom of the page.

4. On the **Monitoring** tab, select the Auto Scaling group metrics collection, and enable the checkbox located under **Auto Scaling**, at the top of the page.

## Enable Collection of Request Metrics in S3

In the account that emits the metrics, follow the instructions in AWS documentation for [Creating a CloudWatch metrics configuration for all the objects in your bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configure-request-metrics-bucket.html).

## Enable Enrichment of AWS Metrics

The metrics sent by AWS CloudWatch to the Kinesis Firehose include minimal labels. Kloudfuse enables you to attach more labels and user-defined custom tags to the ingested metrics, from _within_ the AWS console, by scraping AWS.

To enable this enrichment of AWS metrics, follow these steps:

1. **Modify yaml** in the global section of the `custom-values.yaml` file:

```yaml
global:
     enrichmentEnabled:
    - aws
```

2. **Create IAM scraper role** in the AWS account where the services that emit the metrics run.

Attach the following policy, for Kloudfuse to scrape the additional labels from AWS. See AWS documentation [Define custom IAM permissions with customer managed policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html)

Create a scraper role with custom policies

```yaml
            "Action": [\
                "acm:ListCertificates",\
                "acm:ListTagsForCertificate",\
                "apigateway:GET",\
                "athena:ListWorkGroups",\
                "athena:ListTagsForResource",\
                "autoscaling:DescribeAutoScalingGroups",\
                "bedrock:ListFoundationModels",\
                "bedrock:ListTagsForResource",\
                "cloudwatch:ListMetrics",\
                "cloudwatch:GetMetricStatistics",\
                "dynamodb:ListTables",\
                "dynamodb:DescribeTable",\
                "dynamodb:ListTagsOfResource",\
                "ec2:DescribeInstances",\
                "ec2:DescribeInstanceStatus",\
                "ec2:DescribeSecurityGroups",\
                "ec2:DescribeNatGateways",\
                "ec2:DescribeVolumes",\
                "ecs:ListClusters",\
                "ecs:ListContainerInstances",\
                "ecs:ListServices",\
                "ecs:DescribeContainerInstances",\
                "ecs:DescribeServices",\
                "ecs:ListTagsForResource",\
                "elasticache:DescribeCacheClusters",\
                "elasticache:DescribeServerlessCaches",\
                "elasticache:ListTagsForResource",\
                "elasticfilesystem:DescribeFileSystems",\
                "elasticfilesystem:DescribeBackupPolicy",\
                "elasticloadbalancing:DescribeTags",\
                "elasticloadbalancing:DescribeLoadBalancers",\
                "es:ListDomainNames",\
                "es:DescribeDomains",\
                "es:ListTags",\
                "events:ListRules",\
                "events:ListTagsForResource",\
                "events:ListEventBuses",\
                "firehose:DescribeDeliveryStream",\
                "firehose:ListDeliveryStreams",\
                "firehose:ListTagsForDeliveryStream",\
                "fsx:DescribeFileSystems",\
                "fsx:ListTagsForResource",\
                "glue:ListJobs",\
                "glue:GetTags",\
                "kafka:ListTagsForResource",\
                "kafka:ListClustersV2",\
                "kinesis:ListStreams",\
                "kinesis:ListTagsForStream",\
                "kinesis:DescribeStream",\
                "lambda:GetPolicy",\
                "lambda:List*",\
                "lambda:ListTags",\
                "logs:DescribeLogGroups",\
                "logs:ListTagsForResource",\
                "logs:ListTagsLogGroup",\
                "mq:ListBrokers",\
                "mq:DescribeBroker",\
                "mediaconvert:ListQueues",\
                "mediaconvert:ListTagsForResource",\
                "qbusiness:ListApplications",\
                "qbusiness:GetApplication",\
                "qbusiness:ListTagsForResource",\
                "rds:DescribeDBInstances",\
                "rds:DescribeDBClusters",\
                "rds:ListTagsForResource",\
                "rds:DescribeEvents",\
                "redshift:DescribeClusters",\
                "redshift:DescribeTags",\
                "route53:ListHealthChecks",\
                "route53:ListTagsForResource",\
                "s3:ListAllMyBuckets",\
                "s3:GetBucketTagging",\
                "ses:ListConfigurationSets",\
                "ses:GetConfigurationSet",\
                "ses:ListTagsForResource",\
                "sns:ListTagsForResource",\
                "sns:ListTopics",\
                "sqs:ListQueues",\
                "sqs:ListQueueTags",\
                "states:ListStateMachines",\
                "states:ListActivities",\
                "states:ListTagsForResource",\
                "timestream:ListDatabases",\
                "timestream:ListTables",\
                "timestream:DescribeDatabase",\
                "timestream:DescribeTable",\
                "timestream:ListTagsForResource",\
                "wafv2:ListWebACLs",\
                "wafv2:ListRuleGroups",\
                "wafv2:ListTagsForResource",\
                "cloudfront:ListDistributions",\
                "cloudfront:GetDistribution",\
                "cloudfront:ListTagsForResource"\
            ]
```

3. Modify the Trust Relationship for the policy of the scrape role ARN to add the node-group (Node IAM Role ARN), in which Kloudfuse is running on, as the Principal on the Account.

```yaml
{
       "Version": "2012-10-17",
       "Statement": [\
           {\
               "Sid": "Statement1",\
               "Effect": "Allow",\
               "Principal": {\
                   "AWS": "arn:aws:iam::ACCOUNT-NUMBER:role/eksctl-XXXXX-nodegroup-ng-XXXXXX-NodeInstanceRole-XXXXXXXXXX"\
               },\
               "Action": "sts:AssumeRole"\
           }\
       ]
}
```

4. Ensure that the permissions map to the node pool that hosts the EKS cluster for Kloudfuse.

5. Enable Kloudfuse to consume the new role; there are two approaches: through [AWS credentials](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/aws/cloudwatch-metrics/#aws) or through [Role ARNs](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/aws/cloudwatch-metrics/#arn).

AWS credentials

**Add your AWS credentials as a secret, and use the secret in the `ingester config`**.

1. Retrieve your aws credentials; see [Configure tool authentication with AWS](https://docs.aws.amazon.com/powershell/latest/userguide/creds-idc.html).

2. In the Kloudfuse namespace, create a kube secret name `aws-access-key`, with keys `accessKey` and `secretKey`.

```console
   kubectl create secret generic aws-access-key --from-literal=accessKey=<AWS_ACCESS_KEY_ID> --from-literal=secretKey=<AWS_SECRET_ACCESS_KEY>
   ```

3. Specify the `secretName` in the `custom-values.yaml` file.

```yaml
   ingester:
     config:
       awsScraper:
         secretName: aws-access-key
   ```

4. By default, Kloudfuse attempts to scrape from all regions and all [AWS Services](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/aws/supported-services/). Customize this by adding the following configuration in the `custom-values.yaml` file:

```yaml
   ingester:
     config:
       awsScraper:
         secretName: aws-access-key
         namespaces:
       - <add namespace>
   regions:
     - <add region>
   ```

Role ARNs

**Add Role ARNs in the ingester config**: This option enables you to scrape multiple AWS accounts.

1. Add the scraper Role ARNs that you created with the new permissions to the `awsRoleArns` list to your `custom-values.yaml` file.

```yaml
   ingester:
     config:
       awsRoleArns:
   - role: <ADD ROLE ARN HERE>
   ```

2. By default, Kloudfuse attempts to scrape from all regions and all [AWS Namespaces](https://docs.kloudfuse.com/platform/4.0.2/data-collection/cloud-services/aws/supported-services/). Customize this by adding the following configuration in the `custom-values.yaml` file:

```yaml
   ingester:
     config:
       awsRoleArns:
         role: <ADD ROLE ARN HERE>
         namespaces:
       - <add namespace>
   regions:
     - <add region>
   ```

|     |     |
| --- | --- |
|  | For Global Services like AWS CloudFront, a scraper role in the `us-east-1` region is required.<br>Ensure that you create the scraper role in `us-east-1` and also configure a Firehose delivery stream and CloudWatch Metric stream in the `us-east-1` region for global services.<br>```yaml<br>ingester:<br>  config:<br>    awsRoleArns:<br>      - role: <ADD US-EAST-1 ROLE ARN HERE><br>        regions:<br>          - us-east-1<br>```<br> |

1. To modify the node-group IAM role where Kloudfuse Platform runs, add the following permissions policy to the node-group (Node IAM Role ARN) to assume the role.

```yaml
{
       "Version": "2012-10-17",
       "Statement": [\
           {\
               "Effect": "Allow",\
               "Action": "sts:AssumeRole",\
               "Resource": <REPLACE SCRAPER ROLE ARN HERE>\
           }\
       ]
}
```

2. Complete a helm upgrade to save the changes.

```console
helm upgrade --create-namespace --install kfuse . -f <custom_values.yaml>
```

## AWS Namespace Enrichment

Kloudfuse enriches metadata for metrics from the following AWS services. Each service includes all AWS tags plus the specific metadata fields listed below:

| AWS Service | Enriched Metadata Fields |
| --- | --- |
| AutoScaling | All AWS tags on the AutoScaling Group |
| Firehose | All AWS tags on the Delivery Stream |
| RDS | **For DB Instances:**<br>- `allocatedstoragegb`, `availability_zone`, `backupretentionperioddays`<br>  <br>- `dbinstancearn`, `dbinstanceidentifier`, `dbinstanceclass`, `dbiresourceid`, `dbname`<br>  <br>- `engine`, `engineversion`, `multiaz`, `networktype`, `publicly_accessible`<br>  <br>- `secondary_availability_zone`, `storagetype`<br>  <br>- `host` (DbiResourceId), `hostname` (Endpoint Address)<br>  <br>- All AWS tags<br>  <br>**For DB Clusters:**<br>- `allocatedstoragegb`, `availability_zones`, `backupretentionperioddays`<br>  <br>- `dbclusterarn`, `dbclusterresourceid`, `databasename`<br>  <br>- `engine`, `enginemode`, `engineversion`, `global_write_forwarding_status`<br>  <br>- `multiaz`, `networktype`, `storagetype`<br>  <br>- All AWS tags |
| EKS | - `arn`, `cluster_name`, `endpoint`, `platform_version`, `role_arn`, `status`, `kube_server_version`<br>  <br>- All AWS tags |
| EBS | - `availability_zone`, `multiattachenabled`, `outpostarn`, `size`, `snapshotid`, `state`<br>  <br>- `throughput`, `volumeid`, `volume_type`, `volume_name`<br>  <br>- `device` (if attached)<br>  <br>- All AWS tags |
| EC2 | **For Instances:**<br>- `availability_zone`, `image_id`, `instance_id`, `instance_type`, `kernel`<br>  <br>- `iam_profile` (ARN), `host` (instance ID), `autoscaling_group`, `service`<br>  <br>- All AWS tags<br>  <br>**For NAT Gateways:**<br>- `natgatewayid`<br>  <br>- All AWS tags |
| ELB | - `canonicalhostedzonename`, `canonicalhostedzonenameid`, `dnsname`, `loadbalancername`, `scheme`, `vpcid`<br>  <br>- `host` (CanonicalHostedZoneName), `hostname` (CanonicalHostedZoneName), `name` (LoadBalancerName)<br>  <br>- All AWS tags |
| MQ | - `brokerarn`, `brokerid`, `brokername`, `brokerstate`, `deploymentmode`<br>  <br>- `enginetype`, `engineversion`, `hostinstancetype`, `storagetype`<br>  <br>- All AWS tags |
| S3 | All AWS tags on the S3 Bucket |
| EFS | - `filesystemarn`, `name`<br>  <br>- `aws_elasticfilesystem_default_backup` (enabled/disabled)<br>  <br>- All AWS tags |
| ELBv2 | - `loadbalancerarn`, `name` (LoadBalancerName), `host` (DNSName)<br>  <br>- All AWS tags |
| ACM | All AWS tags on the Certificate |
| ElastiCache | **For ElastiCache Clusters:**<br>- `cache_node_type`, `name` (CacheClusterId), `engine`, `engine_version`<br>  <br>- `preferred_availability_zone`, `replication_group`<br>  <br>- All AWS tags<br>  <br>**For Serverless Caches:**<br>- `name` (ServerlessCacheName), `engine`, `status`, `create_time` |
| CloudFront | All AWS tags on Distributions |
| Route53 | All AWS tags on Health Checks |
| SNS | All AWS tags on Topics |
| Redshift | All AWS tags on Clusters |
| OpenSearch | - `elasticsearch_version` (EngineVersion), `name` (DomainName), `dedicated_master_enabled`<br>  <br>- `instance_type`, `zone_awareness_enabled`, `ebs_enabled`<br>  <br>- All AWS tags |
| SQS | All AWS tags on Queues |
| Lambda | - `function_arn`, `functionname`, `memory_size`, `runtime`<br>  <br>- `architecture` (first in list), `storage_size` (EphemeralStorage Size)<br>  <br>- All AWS tags |
| DynamoDB | All AWS tags on Tables |
| ApiGateway | - `apiid` (Id)<br>  <br>- All AWS tags |
| ApiGatewayV2 | - `apiname` (Name)<br>  <br>- All AWS tags |
| Glue | All AWS tags on Jobs |
| Athena | All AWS tags on WorkGroups |
| ECS | **For Clusters and Services:**<br>- All AWS tags |
| EventBridge | **For Rules and Event Buses:**<br>- All AWS tags |
| Kafka | All AWS tags on Clusters |
| Kinesis | All AWS tags on Streams |
| Logs | All AWS tags on Log Groups |
| WAF | **For Web ACLs and Rule Groups:**<br>- All AWS tags |
| FSx | - `generation` (1 or 2, for ONTAP only), `file_system_type`<br>  <br>- All AWS tags |
| Bedrock | **For Foundation Models:**<br>- `model_name`, `provider_name`<br>  <br>- All AWS tags |
| QBusiness | **For Applications:**<br>- `display_name`<br>  <br>- All AWS tags |
| MediaConvert | **For Queues:**<br>- `name`, `status`, `type`<br>  <br>- All AWS tags on the Queue |
| States (Step Functions) | **For State Machines:**<br>- `name`, `type`<br>  <br>- All AWS tags on the State Machine<br>  <br>**For Activities:**<br>- `name`<br>  <br>- All AWS tags on the Activity |
| Timestream | **For Databases:**<br>- `database_name`, `arn`<br>  <br>- All AWS tags on the Database<br>  <br>**For Tables:**<br>- `table_name`, `database_name`, `arn`<br>  <br>- All AWS tags on the Table |
| SES | **For Configuration Sets:**<br>- `name` (ConfigurationSetName)<br>  <br>- `tls_policy` (DeliveryOptions.TlsPolicy)<br>  <br>- `sending_enabled` (SendingOptions.SendingEnabled)<br>  <br>- All AWS tags on the Configuration Set |

## Reduce Cost of Metrics Ingestion

AWS CloudWatch metrics ingestion can be a high-cost operation. The driving factor here is the AWS `CW:MetricsStreamUsage` attribute, especially the `MetricsUpdate` statistical aggregate.

To reduce the cost of operating CloudWatch metrics ingestion, consider these factors:

1. Volume of Ingested Metrics

Control this by sending only the necessary Namespaces and metrics to the stream.

In other words, avoid selecting **_All Namespaces_** and **_All Metrics_** when configuring ingestion.

Some namespaces are very costly when deriving metrics. These include AWS NLB and AWS Lambda because they feature both a high volume of metrics, and multiple dimensions.

2. Data Retention

Our research indicates that you should modify the retention period of the CloudWatch metrics data by changing the retention setting for the log group of the firehose stream.

3. Sampling Frequency

The frequency of data sampling by CloudWatch is controlled internally by the AWS CloudWatch implementation.