# Continuous Profiling with Kloudfuse Profiler Server

Continuous Profiling is a powerful addition to the Kloudfuse observability platform. While traditional monitoring methods — metrics, logs, and tracing — provide valuable insights, they leave gaps in understanding application performance at a granular level. Continuous Profiling addresses this lack of coverage by offering in-depth, line-level insights into the application’s code, exposing the precise details of resource utilization.

This low-overhead feature gathers profiles from production systems and stores them for subsequent analysis. It provides a comprehensive view of the application and its behavior in production, including CPU usage, memory allocation, and disk I/O, and ensures that every line of code operates efficiently.

## Benefits of Continuous Profiling

### Granular Insights

Continuous Profiling offers a detailed view of application performance that goes beyond traditional observability tools, providing line-level insights into resource utilization.

### In-Depth Code Analysis

With a comprehensive understanding of code performance and system interactions, developers can easily identify how specific code segments use resources, facilitating thorough analysis and optimization.

### Optimization Opportunities

By pinpointing inefficient lines of code, Continuous Profiling helps address performance bottlenecks and improve resource utilization across all applications.

### Effective Capacity Planning

The profiling data supports informed capacity planning and scaling efforts, ensuring that your application can meet growing demands while maintaining optimal performance.

### Cost Reduction

By identifying resource spikes in CPU and memory usage, Continuous Profiling aids in optimizing these areas to lower costs.

## Configuration

Enable `kfuse-profiling` in the `custom-values.yaml` file.

```yaml
global:
  kfuse-profiling:
    enabled: true
```

Kloudfuse default configuration saves the data in the PVC with size 50GB.

## Long-Term Retention

To retain profiling data for a longer time, change the configuration settings. Depending on the storage provider, configure one of the following options in the `custom-values.yaml` file: [AWS S3](https://docs.kloudfuse.com/platform/3.5.0/profiler-server-aws/) or [GCP Bucket](https://docs.kloudfuse.com/platform/3.5.0/profiler-server-gcp/). Remember that profiling data uses [parquet storage format](https://en.wikipedia.org/wiki/Apache_Parquet).

## Scrape the Profiling Data

Alloy queries the `pprof` endpoints of your Golang application, collects the profiles, and forwards them to the Kfuse Profiler server.

### Prerequisites

1. Ensure your Golang application exposes `pprof` endpoints.
2. In pull mode, the Alloy collector periodically retrieves profiles from Golang applications, specifically targeting the `/debug/pprof/*` endpoints.
3. Set up Go profiling in pull mode to generate profiles. See Grafana documentation on [Set up Go profiling in pull mode](https://grafana.com/docs/pyroscope/latest/configure-client/grafana-alloy/go_pull/#set-up-go-profiling-in-pull-mode).
4. Set up Java profiling to generate profiles. See Grafana documentation on [Java](https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/java/).

### Configure Scraping

1. Configure the alloy scraper in a new file, `alloy-values.yaml`. Download a copy of the default [alloy-values.yaml](https://raw.githubusercontent.com/grafana/alloy/main/operations/helm/charts/alloy/values.yaml) file from the Grafana repository, and customize the `alloy configMap` section.

2. Configure the `pyroscope.write` block in the [Alloy Configuration](https://docs.kloudfuse.com/platform/3.5.0/profiler-server/#ex-alloy-configuration) file to define the endpoint where to send profiling data.

Define Endpoint for Sending Profiling Data

```yaml
   pyroscope.write "write_job_name" { (1)
          endpoint {
              url = "https://<KFUSE ENDPOINT/DNS NAME>/profile" (2)
          }
   }
   ```

|     |     |
   | --- | --- |
   | **1** | Change `write_job_name` to appropriate name, like `kfuse_profiler_write`. |
   | **2** | Change url to `https://<KFUSE ENDPOINT/DNS NAME>/profile`. |

3. Configure `pyroscope.scrape` block in the [Alloy Configuration](https://docs.kloudfuse.com/platform/3.5.0/profiler-server/#ex-alloy-configuration) file to define the scraping configuration for profiling data.

Define the Scraping Configuration for Profiling Data

```yaml
   pyroscope.scrape "scrape_job_name" { (1)
              targets    = concat(discovery.relabel.kubernetes_pods.output) (2)
              forward_to = [pyroscope.write.write_job_name.receiver] (3)

profiling_config { (4)
                      profile.process_cpu { (5)
                              enabled = true
                      }

profile.godeltaprof_memory { (6)
                              enabled = true
                      }

profile.memory { // disable memory, use godeltaprof_memory instead
                              enabled = false
                      }

profile.godeltaprof_mutex  { (7)
                              enabled = true
                      }

profile.mutex { // disable mutex, use godeltaprof_mutex instead
                              enabled = false
                      }

profile.godeltaprof_block { (8)
                              enabled = true
                      }

profile.block { // disable block, use godeltaprof_block instead
                              enabled = false
                      }

profile.goroutine {
                              enabled = true (9)
                      }
              }
   }
   ```

|     |     |
   | --- | --- |
   | **1** | Change `scrape_job_name` to an appropriate name, like `kfuse_profiler_scrape`. |
   | **2** | Use `discovery.relabel.kubernetes_pods.output` as a target for `pyroscope.scrape` block to discover Kubernetes targets. Follow Grafana documentation to learn how to set up specific regex rules [Discover Kubernetes targets](https://grafana.com/docs/pyroscope/latest/configure-client/grafana-alloy/go_pull/#discover-kubernetes-targets). |
   | **3** | `forward_to`: Connects the scrape job to the write job. |
   | **4** | `profiling_config`: Enables or disables specific profiles. |
   | **5** | `profile.process_cpu`: Enables CPU profiling. |
   | **6** | `profile.godeltaprof_memory`: Enables delta memory profiling. |
   | **7** | `profile.godeltaprof_mutex`: Enables delta mutex profiling. |
   | **8** | `profile.godeltaprof_block`: Enables `delta block` profiling. |
   | **9** | `profile.goroutine`: Enables `goroutine` profiling. |

4. Configure the rest of the fields in [Alloy Configuration](https://docs.kloudfuse.com/platform/3.5.0/profiler-server/#ex-alloy-configuration) file.

Alloy Configuration

```yaml
   alloy:
        configMap:
          create: true (1)
          content: |- (2)
            logging { (3)
              level = "info"
              format = "logfmt"
            }
            discovery.kubernetes "pyroscope_kubernetes" {
                role = "pod"
            }

discovery.relabel "kubernetes_pods" {
                targets = concat(discovery.kubernetes.pyroscope_kubernetes.targets)

rule {
                    action        = "drop"
                    source_labels = ["__meta_kubernetes_pod_phase"]
                    regex         = "Pending|Succeeded|Failed|Completed"
                }

rule {
                    action = "labelmap"
                    regex  = "__meta_kubernetes_pod_label_(.+)"
                }

rule {
                    action        = "replace"
                    source_labels = ["__meta_kubernetes_namespace"]
                    target_label  = "kubernetes_namespace"
                }

rule {
                    action        = "replace"
                    source_labels = ["__meta_kubernetes_pod_name"]
                    target_label  = "kubernetes_pod_name"
                }

rule {
                    action        = "keep"
                    source_labels = ["__meta_kubernetes_pod_annotation_pyroscope_io_scrape"]
                    regex = "true"
                }

rule {
                    action        = "replace"
                    source_labels = ["__meta_kubernetes_pod_annotation_pyroscope_io_application_name"]
                    target_label = "service_name"
                }

rule {
                    action        = "replace"
                    source_labels = ["__meta_kubernetes_pod_annotation_pyroscope_io_spy_name"]
                    target_label = "__spy_name__"
                }

rule {
                    action        = "replace"
                    source_labels = ["__meta_kubernetes_pod_annotation_pyroscope_io_scheme"]
                    regex = "(https?)"
                    target_label = "__scheme__"
                }

rule {
                    action        = "replace"
                    source_labels = ["__address__", "__meta_kubernetes_pod_annotation_pyroscope_io_port"]
                    regex = "(.+?)(?::\d+)?;(\d+)"
                    replacement = "$1:$2"
                    target_label = "__address__"
                }

rule {
                    action = "labelmap"
                    regex  = "__meta_kubernetes_pod_annotation_pyroscope_io_profile_(.+)"
                    replacement = "__profile_$1"
                }
            }

pyroscope.scrape "pyroscope_scrape" { (4)
                clustering {
                    enabled = true
                }

targets    = concat(discovery.relabel.kubernetes_pods.output)
                forward_to = [pyroscope.write.pyroscope_write.receiver]

profiling_config {
                    profile.memory {
                        enabled = true
                    }

profile.process_cpu {
                        enabled = true
                    }

profile.goroutine {
                        enabled = true
                    }

profile.block {
                        enabled = false
                    }

profile.mutex {
                        enabled = false
                    }

profile.fgprof {
                        enabled = false
                    }
                }
            }

pyroscope.write "pyroscope_write" {
                endpoint {
                    url = "https://<KFUSE ENDPOINT/DNS NAME>/profile"
                }
            }

name: null (5)
        key: null (6)

clustering:
        enabled: false (7)
        name: "" (8)
        portName: http (9)

stabilityLevel: "generally-available" (10)
      storagePath: /tmp/alloy (11)
      listenAddr: 0.0.0.0 (12)
      listenPort: 12345 (13)
      listenScheme: HTTP (14)
      uiPathPrefix: / (15)
      enableReporting: true (16)
      extraEnv: [] (17)
      envFrom: [] (18)
      extraArgs: [] (19)
      extraPorts: [] (20)

mounts:
        varlog: false (21)
        dockercontainers: false
        extra: [] (22)

securityContext: {} (23)

resources: {} (24)
   ```

5. Refer to Grafana documentation on [Deploy Grafana Alloy on Kubernetes](https://grafana.com/docs/alloy/latest/set-up/install/kubernetes/) to correctly install `alloy` in the namespace where you want to scrape the data.

6. Update alloy using the `alloy-values.yaml` file you set up. Use the same `namespace` as in the previous step, where you installed alloy.

```console
   helm upgrade --namespace <namespace> alloy grafana/alloy -f <path/to/alloy-values.yaml>
   ```