Native and Exponential Histograms :: Kloudfuse Docs

Native and Exponential Histograms

Starting with Kloudfuse 3.4.3, Prometheus Native Histograms and OpenTelemetry Exponential Histograms are supported natively and will work out of the box when Kloudfuse receives them.

Prometheus Native Histograms

Prometheus Native Histograms provide a more efficient and accurate way to represent histogram data compared to traditional fixed-bucket histograms. They use a dynamic bucket schema that automatically adjusts to the distribution of observed values.

For more details, refer to the Prometheus Native Histograms specification.

Prometheus Server Configuration

To enable native histograms in Prometheus Server (requires Prometheus 2.40+):

Enable the native-histograms feature in Prometheus
Set send_native_histograms: true in the remote write configuration

For detailed Prometheus configuration instructions, see Prometheus Agent Configuration.

Client SDK Example (Go)

Here’s an example of using native histograms with the Prometheus Go client:

package main

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "net/http"
    "time"
)

func main() {
    // Create a native histogram
    requestDuration := prometheus.NewHistogram(prometheus.HistogramOpts{
        Name: "http_request_duration_seconds",
        Help: "HTTP request duration in seconds",
        // NativeHistogramBucketFactor determines the growth factor of buckets
        NativeHistogramBucketFactor: 1.1,
        // NativeHistogramMaxBucketNumber limits the number of buckets
        NativeHistogramMaxBucketNumber: 100,
        // Enable native histogram
        NativeHistogramMinResetDuration: 1 * time.Hour,
    })

prometheus.MustRegister(requestDuration)

// Record observations
    requestDuration.Observe(0.235)

// Expose metrics endpoint
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":8080", nil)
}

Other language SDKs with native histogram support:

OpenTelemetry Exponential Histograms

OpenTelemetry Exponential Histograms use an exponential formula to determine bucket boundaries, providing high resolution for small values and automatically adjusting resolution for larger values.

For more details, refer to the OpenTelemetry Exponential Histogram specification.

Client SDK Example (Go)

Here’s an example of using exponential histograms with the OpenTelemetry Go SDK:

package main

import (
    "context"
    "time"

"go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
    "go.opentelemetry.io/otel/metric"
    sdkmetric "go.opentelemetry.io/otel/sdk/metric"
    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
)

func main() {
    ctx := context.Background()

// Create OTLP exporter
    exporter, err := otlpmetricgrpc.New(ctx,
        otlpmetricgrpc.WithEndpoint("your-kloudfuse-instance:4317"),
        otlpmetricgrpc.WithInsecure(),
    )
    if err != nil {
        panic(err)
    }

// Create resource
    res, err := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceNameKey.String("example-service"),
        ),
    )
    if err != nil {
        panic(err)
    }

// Create meter provider with exponential histogram aggregation
    provider := sdkmetric.NewMeterProvider(
        sdkmetric.WithResource(res),
        sdkmetric.WithReader(
            sdkmetric.NewPeriodicReader(
                exporter,
                sdkmetric.WithInterval(30*time.Second),
            ),
        ),
        // Configure exponential histogram as default aggregation
        sdkmetric.WithView(
            sdkmetric.NewView(
                sdkmetric.Instrument{Kind: sdkmetric.InstrumentKindHistogram},
                sdkmetric.Stream{
                    Aggregation: sdkmetric.AggregationBase2ExponentialHistogram{
                        MaxSize: 160,
                        MaxScale: 20,
                    },
                },
            ),
        ),
    )

otel.SetMeterProvider(provider)
    meter := provider.Meter("example-meter")

// Create histogram instrument
    histogram, err := meter.Float64Histogram(
        "request.duration",
        metric.WithDescription("Request duration in milliseconds"),
        metric.WithUnit("ms"),
    )
    if err != nil {
        panic(err)
    }

// Record observations
    histogram.Record(ctx, 125.5)
    histogram.Record(ctx, 74.3)
    histogram.Record(ctx, 238.9)

// Ensure metrics are exported
    time.Sleep(35 * time.Second)
}

Other language SDKs with exponential histogram support:

Benefits

Both Native and Exponential Histograms offer several advantages:

Better accuracy: Dynamic bucket boundaries adapt to the actual data distribution
Lower memory footprint: More efficient storage compared to fixed-bucket histograms
Automatic resolution adjustment: No need to predefine bucket boundaries
Native support in Kloudfuse: Direct ingestion and querying without conversion overhead

Querying

Once ingested into Kloudfuse, these histograms can be queried using standard PromQL functions such as:

histogram_quantile() - Calculate quantiles
histogram_count() - Get the count of observations
histogram_sum() - Get the sum of all observations
histogram_avg() - Calculate the average of all observations
histogram_stddev() - Calculate the standard deviation of observations
histogram_stdvar() - Calculate the variance of observations
histogram_fraction() - Calculate the fraction of observations between two bounds

Example queries:

# 95th percentile of request duration
histogram_quantile(0.95, rate(http_request_duration_seconds[5m]))

# Average request duration
histogram_avg(rate(http_request_duration_seconds[5m]))

# Standard deviation of request durations
histogram_stddev(rate(http_request_duration_seconds[5m]))

# Variance of request durations
histogram_stdvar(rate(http_request_duration_seconds[5m]))

# Fraction of requests under 250ms
histogram_fraction(0, 0.25, rate(http_request_duration_seconds[5m]))