APM Go Instrumentation :: Kloudfuse Docs

APM Go Instrumentation

Kloudfuse APM supports Go-based distributed systems via the OpenTelemetry Go SDK. This guide walks through setting up instrumentation and exporting spans to the Kloudfuse backend.

Prerequisites

OTLP Endpoints

There are three ways to route spans from a Go application to Kloudfuse, depending on how your cluster is deployed. Choose one and use it consistently across your OTEL_* environment variables.

The standard deployment runs kf-agent as a DaemonSet or sidecar in the cluster. Applications send spans to it over gRPC on port 4317; the agent batches and forwards them to the Kloudfuse backend.

OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>

Use the otlptracegrpc exporter package:

go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc

import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"

// All config is read from OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS.
exporter, err := otlptracegrpc.New(ctx)

If gRPC is blocked by a firewall or network policy, kf-agent also accepts OTLP/HTTP on port 4318. The SDK appends /v1/traces to the base endpoint automatically.

OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>

Use the otlptracehttp exporter package:

go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp

import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"

exporter, err := otlptracehttp.New(ctx)

When kf-agent is not deployed in the cluster, send spans directly to the Kloudfuse ingester through the nginx ingress over HTTPS. This is the path to use in environments where only port 443 is open.

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>
Variable used Resulting URL sent to the ingester
OTEL_EXPORTER_OTLP_ENDPOINT=https://host/ingester/otlp/traces https://host/ingester/otlp/traces/v1/traces ❌ (double suffix)
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://host/ingester/otlp/traces https://host/ingester/otlp/traces ✓ (used verbatim)

Use the otlptracehttp exporter — gRPC on port 4317 is not exposed by the nginx ingress:

import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"

// Endpoint, headers, and protocol are all read from OTEL_* env vars automatically.
exporter, err := otlptracehttp.New(ctx)

In Kubernetes, inject the API key from a Secret:

env:
  - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
    value: "https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value

Create the Secret with the full header expression (not just the raw token):

kubectl create secret generic kloudfuse-api-key \
  --from-literal=value="kf-api-key=<your-api-key>"

Authentication

Pass the Kloudfuse API key as a request header on the OTLP exporter.

Both the gRPC and HTTP exporters read OTEL_EXPORTER_OTLP_HEADERS automatically — no code changes are needed:

export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>"

The variable accepts a comma-separated list of name=value pairs, so multiple headers can be passed:

export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>,x-env=production"

In Kubernetes, pull the value from a Secret so it never appears in source:

env:
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value

If you cannot use environment variables, pass headers directly via WithHeaders. Read the key from the environment rather than hardcoding it:

import (
    "os"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
)

exporter, err := otlptracehttp.New(ctx,
    otlptracehttp.WithHeaders(map[string]string{
        "kf-api-key": os.Getenv("KLOUDFUSE_API_KEY"),
    }),
)

For the gRPC exporter, the option is identical — replace otlptracehttp with otlptracegrpc:

import (
    "os"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
)

exporter, err := otlptracegrpc.New(ctx,
    otlptracegrpc.WithHeaders(map[string]string{
        "kf-api-key": os.Getenv("KLOUDFUSE_API_KEY"),
    }),
)

HTTP Servers

For HTTP servers, use otelhttp.NewHandler from the contrib package. It automatically sets SpanKind = SERVER, extracts incoming W3C traceparent headers, and populates standard HTTP semantic convention attributes (http.method, http.status_code, http.target, etc.):

go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

http.Handle("/api/orders", otelhttp.NewHandler(ordersHandler, "GET /api/orders"))
http.ListenAndServe(":8080", nil)

For outbound HTTP calls, wrap the transport so traceparent headers are injected automatically:

client := &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
client.Do(req)

Manual Instrumentation

Step 1: Import OpenTelemetry Modules

Add the core OpenTelemetry packages to your Go module. This example uses the HTTP exporter (Option 2 or 3 above); swap otlptracehttp for otlptracegrpc if using gRPC.

go get go.opentelemetry.io/otel@v1.28.0 \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp@v1.28.0 \
  go.opentelemetry.io/otel/sdk@v1.28.0

Your go.mod should look like:

require (
  go.opentelemetry.io/otel v1.28.0
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.28.0
  go.opentelemetry.io/otel/sdk v1.28.0
)

Step 2: Configure the Tracer Provider

Set up the exporter and tracer provider in main(). All OTLP connection settings — endpoint, headers, protocol, compression — are read from OTEL_* environment variables automatically. Only the resource attributes (service name, environment, namespace) need to be set in code.

package main

import (
    "context"
    "log"

"go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func initTracer(ctx context.Context) (*sdktrace.TracerProvider, error) {
    // Endpoint, headers, and protocol are read from OTEL_EXPORTER_OTLP_* env vars.
    exporter, err := otlptracehttp.New(ctx)
    if err != nil {
        return nil, err
    }

res := resource.NewWithAttributes(
        "https://opentelemetry.io/schemas/1.21.0",
        attribute.String("service.name", "my-go-service"),
        attribute.String("deployment.environment.name", "production"),
        attribute.String("service.namespace", "my-team"),
    )

tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
    )

otel.SetTracerProvider(tp)
    return tp, nil
}

func main() {
    ctx := context.Background()
    tp, err := initTracer(ctx)
    if err != nil {
        log.Fatalf("failed to init tracer: %v", err)
    }
    defer tp.Shutdown(ctx)

// ... start your application
}

Call tp.Shutdown(ctx) on exit to flush any buffered spans before the process terminates.

Step 3: Create Spans

Use the global tracer to create spans. Always set SpanKind explicitly on the entry-point span of your service — SpanKind.Server for inbound HTTP/gRPC handlers, SpanKind.Client for outbound calls. Kloudfuse derives service-level throughput and latency metrics from SERVER spans.

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/trace"
)

tracer := otel.Tracer("my-go-service")

// Root SERVER span — represents an inbound request handled by this service.
ctx, serverSpan := tracer.Start(ctx, "handle-request",
    trace.WithSpanKind(trace.SpanKindServer),
)
defer serverSpan.End()

// Child CLIENT span — represents an outbound call (DB query, downstream service, etc.).
// Inherits the parent's trace ID automatically because ctx carries the parent.
_, clientSpan := tracer.Start(ctx, "db-query",
    trace.WithSpanKind(trace.SpanKindClient),
)
defer clientSpan.End()

To add attributes to a span:

import "go.opentelemetry.io/otel/attribute"

span.SetAttributes(
    attribute.String("db.system", "postgresql"),
    attribute.String("db.name", "orders"),
)

Verify Traces

  1. Open Kloudfuse UI → APM → Trace Explorer
  2. Filter by service.name = my-go-service
  3. Click a trace to inspect the span tree, attributes, and timing breakdown

References

Sample Manifest

An example Kubernetes manifest and instructions can be found at APM Demo - Go

Specification and Concepts

Go SDK and Packages

Exporter Configuration