Kubernetes Autoscaling (HPA) with Custom Metrics

You must have the following to scale components based on custom metrics:

• To collect metrics from applications and store them in the Kloudfuse database.

• To extend the Kubernetes Custom Metrics API with metrics supplied by a collector, the k8s-prometheus-adapter. The adapter is an implementation of the custom metrics API that supports arbitrary metrics.

Consider a case where Kloudfuse ingests an application-specific metric, some_metric. You plan to modify it, and use its value for autoscale the kubernetes pods. Follow these steps:

1. Install Prometheus adapter with proper configuration to emit a custom_metric based on a query for some_metric.

2. Create prometheus-values.yaml file for the Prometheus Adapter, configuring the proper endpoint, authentication, and custom metric rules.

   metricsRelistInterval: 5m

prometheus:
        # Replace with your endpoint for Kloudfuse
        url: <<https://your.hostname.kloudfuse>>
        port: 443
        path: ""

# Rules to generate `some_custom_metric` as an external metric
   rules:
        default: false
        external:
     - seriesQuery: '{__name__=~"some_metric"}'
       resources:
         template: <<.Resource>>
       name:
         as: "some_custom_metric"
       metricsQuery: avg (some_metric)

# Base64 Encoded Auth string in the format <<username:password>>
   extraArguments:
     - --prometheus-header=Authorization=Basic YWRtaW46cGFzc3dvcmQ=

3. Deploy Prometheus adapter on your cluster.

   helm install --name prometheus-adapter >prometheus-community/prometheus-adapter -f prometheus-values.yaml
   

4. Verify that the adapter emits custom-metric correctly, using the Kubernetes External Metrics API.

See Prometheus Adapter code and documentation for additional configuration information.

kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 | jq .

#Should show some output like this
{
     "kind": "APIResourceList",
     "apiVersion": "v1",
     "groupVersion": "external.metrics.k8s.io/v1beta1",
     "resources": [\
       {\
         "name": "some_custom_metric",\
         "singularName": "",\
         "namespaced": true,\
         "kind": "ExternalMetricValueList",\
         "verbs": [\
           "get"\
         ]\
       }\
     ]
}

5. After custom-metric becomes available to use within the Kubernetes Metrics API, configure HPA to use the metric.

   apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   metadata:
        name: <<some_name>>
   spec:
        scaleTargetRef:
          apiVersion: apps/v1
          kind: Deployment
          name: <<some_name>>
        minReplicas: 1
        maxReplicas: 10
        metrics:
     - type: External
       external:
         metric:
           name: some_custom_metric
         target:
           type: AverageValue
           averageValue: 100
   

6. Explore how to configure HPA with Custom metrics in Kubernetes documentation for Autoscaling on multiple metrics and custom metrics.