Aggregation operators

Aggregation operators combine input series into fewer series. Use by (label, …​) to keep the listed labels as grouping dimensions, or without (label, …​) to drop them.

avg

Averages the values of the input series per group. Every series weighs equally regardless of how it was produced.

Syntax

avg by (<labels>) (<expr>)    (also: avg without (<labels>) (...))

Parameters

Example

Apply avg to the goroutine gauges of the Kloudfuse query services, grouped by service name.

avg by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_name=~".*query-service"}
)

Expected output

bottomk

Returns the k input series with the smallest values, keeping their labels. The quiet outliers are often as interesting as the noisy ones.

Syntax

bottomk(<k>, <expr>)

Parameters

Example

Find the two Kloudfuse services running the fewest goroutines.

bottomk(2, sum by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_instance="kfuse"}
))

Expected output

count_values

Groups the input series by their sample value and counts how many series hold each value, writing the value itself into a new label. Use it for discrete-valued metrics — versions, capacities, states — to see the distribution at a glance.

Syntax

count_values("<label>", <expr>)

Parameters

Example

Count how many node-capacity series report each distinct CPU capacity value.

count_values("cores", kubernetes_state_node_cpu_capacity)

Expected output

count

Counts how many series the inner expression produced, per group. Useful for fleet-size questions: how many pods, how many targets.

Syntax

count by (<labels>) (<expr>)    (also: count without (<labels>) (...))

Example

Apply count to the goroutine gauges of the Kloudfuse query services, grouped by service name.

count by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_name=~".*query-service"}
)

Expected output

group

Collapses each group to a single series with value 1. Use it to deduplicate label combinations before counting or joining, when the values themselves do not matter.

Syntax

group by (<labels>) (<expr>)    (also: group without (<labels>) (...))

Example

Apply group to the goroutine gauges of the Kloudfuse query services, grouped by service name.

group by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_name=~".*query-service"}
)

Expected output

max

Returns the largest value among the input series per group — the worst-case detector.

Syntax

max by (<labels>) (<expr>)    (also: max without (<labels>) (...))

Example

Apply max to the goroutine gauges of the Kloudfuse query services, grouped by service name.

max by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_name=~".*query-service"}
)

Expected output

min

Returns the smallest value among the input series per group — the floor of a fleet.

Syntax

min by (<labels>) (<expr>)    (also: min without (<labels>) (...))

Example

Apply min to the goroutine gauges of the Kloudfuse query services, grouped by service name.

min by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_name=~".*query-service"}
)

Expected output

quantile

Computes the given quantile (0 to 1) across the values of the input series, per group. This is a cross-series statistic at one instant — for a quantile of one series over time, use quantile_over_time; for histogram data, use histogram_quantile.

Syntax

quantile(<q>, <expr>)    (also: quantile by (<labels>) (<q>, ...))

Parameters

Example

Find the 90th-percentile per-series goroutine count across all Kloudfuse service pods.

quantile(0.9, go_goroutines{app_kubernetes_io_instance="kfuse"})

Expected output

stddev

Computes the population standard deviation of the input values per group — a direct measure of imbalance across a fleet.

Syntax

stddev by (<labels>) (<expr>)    (also: stddev without (<labels>) (...))

Example

Apply stddev to the goroutine gauges of the Kloudfuse query services, grouped by service name.

stddev by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_name=~".*query-service"}
)

Expected output

stdvar

Computes the population variance of the input values per group — the square of stddev.

Syntax

stdvar by (<labels>) (<expr>)    (also: stdvar without (<labels>) (...))

Example

Apply stdvar to the goroutine gauges of the Kloudfuse query services, grouped by service name.

stdvar by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_name=~".*query-service"}
)

Expected output

sum

Adds up the values of all input series, per group when by (…​) is given. The most common aggregation: totals along the dimension you care about, per-stream detail collapsed.

Syntax

sum by (<labels>) (<expr>)    (also: sum without (<labels>) (...))

Example

Apply sum to the goroutine gauges of the Kloudfuse query services, grouped by service name.

sum by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_name=~".*query-service"}
)

Expected output

topk

Returns the k input series with the largest values, keeping their labels — ranking questions answered directly.

Syntax

topk(<k>, <expr>)

Parameters

Example

Rank the three Kloudfuse services running the most goroutines.

topk(3, sum by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_instance="kfuse"}
))

Expected output