# APM Go Instrumentation

Kloudfuse APM supports Go-based distributed systems via the OpenTelemetry Go SDK. This guide walks through setting up instrumentation and exporting spans to the Kloudfuse backend.

## Prerequisites

- Go 1.21 or later installed
- Internet access to fetch Go modules
- Your Kloudfuse API key — see [Ingestion Authentication with API Key](https://docs.kloudfuse.com/platform/4.0.2/administration/authentication/ingestion-api-key/)

## OTLP Endpoints

There are three ways to route spans from a Go application to Kloudfuse, depending on how your cluster is deployed. Choose one and use it consistently across your `OTEL_*` environment variables.

- kf-agent gRPC (port 4317)
- kf-agent HTTP (port 4318)
- Direct Ingester (HTTPS, port 443)

The standard deployment runs `kf-agent` as a DaemonSet or sidecar in the cluster. Applications send spans to it over gRPC on port 4317; the agent batches and forwards them to the Kloudfuse backend.

```bash
OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>
```

Use the `otlptracegrpc` exporter package:

```go
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
```

```go
import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"

// All config is read from OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS.
exporter, err := otlptracegrpc.New(ctx)
```

If gRPC is blocked by a firewall or network policy, `kf-agent` also accepts OTLP/HTTP on port 4318. The SDK appends `/v1/traces` to the base endpoint automatically.

```bash
OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>
```

Use the `otlptracehttp` exporter package:

```go
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp
```

```go
import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"

exporter, err := otlptracehttp.New(ctx)
```

When `kf-agent` is not deployed in the cluster, send spans directly to the Kloudfuse ingester through the nginx ingress over HTTPS. This is the path to use in environments where only port 443 is open.

```bash
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>
```

|     |     |
| --- | --- |
|  | Use `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` (the signal-specific variable), not<br>`OTEL_EXPORTER_OTLP_ENDPOINT` (the base variable). The base variable causes the SDK to append<br>`/v1/traces` to whatever path you provide, which would produce the wrong URL: |

| Variable used | Resulting URL sent to the ingester |
| --- | --- |
| `OTEL_EXPORTER_OTLP_ENDPOINT=https://host/ingester/otlp/traces` | `https://host/ingester/otlp/traces/v1/traces` ❌ (double suffix) |
| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://host/ingester/otlp/traces` | `https://host/ingester/otlp/traces` ✓ (used verbatim) |

Use the `otlptracehttp` exporter — gRPC on port 4317 is not exposed by the nginx ingress:

```go
import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"

// Endpoint, headers, and protocol are all read from OTEL_* env vars automatically.
exporter, err := otlptracehttp.New(ctx)
```

In Kubernetes, inject the API key from a Secret:

```yaml
env:
  - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
    value: "https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value
```

Create the Secret with the full header expression (not just the raw token):

```bash
kubectl create secret generic kloudfuse-api-key \
  --from-literal=value="kf-api-key=<your-api-key>"
```

## Authentication

Pass the Kloudfuse API key as a request header on the OTLP exporter.

- Environment Variable (Recommended)
- Programmatic (WithHeaders)

Both the gRPC and HTTP exporters read `OTEL_EXPORTER_OTLP_HEADERS` automatically — no code changes are needed:

```bash
export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>"
```

The variable accepts a comma-separated list of `name=value` pairs, so multiple headers can be passed:

```bash
export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>,x-env=production"
```

In Kubernetes, pull the value from a Secret so it never appears in source:

```yaml
env:
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value
```

If you cannot use environment variables, pass headers directly via `WithHeaders`. Read the key from the environment rather than hardcoding it:

```go
import (
	"os"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
)

exporter, err := otlptracehttp.New(ctx,
	otlptracehttp.WithHeaders(map[string]string{
		"kf-api-key": os.Getenv("KLOUDFUSE_API_KEY"),
	}),
)
```

For the gRPC exporter, the option is identical — replace `otlptracehttp` with `otlptracegrpc`:

```go
import (
	"os"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
)

exporter, err := otlptracegrpc.New(ctx,
	otlptracegrpc.WithHeaders(map[string]string{
		"kf-api-key": os.Getenv("KLOUDFUSE_API_KEY"),
	}),
)
```

|     |     |
| --- | --- |
|  | Do not hardcode the API key value in source code. Use an environment variable or a secrets<br>manager and inject it at runtime. |

## HTTP Servers

For HTTP servers, use `otelhttp.NewHandler` from the contrib package. It automatically sets
`SpanKind = SERVER`, extracts incoming W3C `traceparent` headers, and populates standard HTTP
semantic convention attributes (`http.method`, `http.status_code`, `http.target`, etc.):

```bash
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
```

```go
import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

http.Handle("/api/orders", otelhttp.NewHandler(ordersHandler, "GET /api/orders"))
http.ListenAndServe(" :8080", nil)
```

For outbound HTTP calls, wrap the transport so `traceparent` headers are injected automatically:

```go
client := &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
client.Do(req)
```

|     |     |
| --- | --- |
|  | `otelhttp` v0.53.x emits deprecated HTTP attribute names (`http.method`, `http.status_code`).<br>Upgrade to v0.56+ together with otel core v1.31+ to get the stable names (`http.request.method`,<br>`http.response.status_code`). See<br>[Go: Semantic Convention Limitations in otelhttp v0.53.x](https://docs.kloudfuse.com/platform/4.0.2/data-collection/apm/instrument-common-issues/#spankind-go-otelhttp-semconv)<br>for the full attribute mapping and upgrade steps. |

## Manual Instrumentation

### Step 1: Import OpenTelemetry Modules

Add the core OpenTelemetry packages to your Go module. This example uses the HTTP exporter
(Option 2 or 3 above); swap `otlptracehttp` for `otlptracegrpc` if using gRPC.

```bash
go get go.opentelemetry.io/otel@v1.28.0 \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp@v1.28.0 \
  go.opentelemetry.io/otel/sdk@v1.28.0
```

Your `go.mod` should look like:

```none
require (
  go.opentelemetry.io/otel v1.28.0
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.28.0
  go.opentelemetry.io/otel/sdk v1.28.0
)
```

### Step 2: Configure the Tracer Provider

Set up the exporter and tracer provider in `main()`. All OTLP connection settings — endpoint, headers,
protocol, compression — are read from `OTEL_*` environment variables automatically. Only the resource
attributes (service name, environment, namespace) need to be set in code.

```go
package main

import (
	"context"
	"log"

"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func initTracer(ctx context.Context) (*sdktrace.TracerProvider, error) {
	// Endpoint, headers, and protocol are read from OTEL_EXPORTER_OTLP_* env vars.
	exporter, err := otlptracehttp.New(ctx)
	if err != nil {
		return nil, err
	}

res := resource.NewWithAttributes(
		"https://opentelemetry.io/schemas/1.21.0",
		attribute.String("service.name", "my-go-service"),
		attribute.String("deployment.environment.name", "production"),
		attribute.String("service.namespace", "my-team"),
	)

tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(res),
	)

otel.SetTracerProvider(tp)
	return tp, nil
}

func main() {
	ctx := context.Background()
	tp, err := initTracer(ctx)
	if err != nil {
		log.Fatalf("failed to init tracer: %v", err)
	}
	defer tp.Shutdown(ctx)

// ... start your application
}
```

Call `tp.Shutdown(ctx)` on exit to flush any buffered spans before the process terminates.

### Step 3: Create Spans

Use the global tracer to create spans. Always set `SpanKind` explicitly on the entry-point span of
your service — `SpanKind.Server` for inbound HTTP/gRPC handlers, `SpanKind.Client` for outbound calls.
Kloudfuse derives service-level throughput and latency metrics from `SERVER` spans.

```go
import (
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/trace"
)

tracer := otel.Tracer("my-go-service")

// Root SERVER span — represents an inbound request handled by this service.
ctx, serverSpan := tracer.Start(ctx, "handle-request",
	trace.WithSpanKind(trace.SpanKindServer),
)
defert serverSpan.End()

// Child CLIENT span — represents an outbound call (DB query, downstream service, etc.).
// Inherits the parent's trace ID automatically because ctx carries the parent.
_, clientSpan := tracer.Start(ctx, "db-query",
	trace.WithSpanKind(trace.SpanKindClient),
)
defert clientSpan.End()
```

To add attributes to a span:

```go
import "go.opentelemetry.io/otel/attribute"

span.SetAttributes(
	attribute.String("db.system", "postgresql"),
	attribute.String("db.name", "orders"),
)
```

## Verify Traces

1. Open **Kloudfuse UI → APM → Trace Explorer**
2. Filter by `service.name = my-go-service`
3. Click a trace to inspect the span tree, attributes, and timing breakdown

## References

### Sample Manifest

An example Kubernetes manifest and instructions can be found at [APM Demo - Go](https://github.com/kloudfuse-ext/customer/tree/main/agents/apm/demo/go)

### Specification and Concepts

- [OpenTelemetry Concepts](https://opentelemetry.io/docs/concepts/) — traces, spans, context propagation, sampling
- [OpenTelemetry Trace SDK specification](https://opentelemetry.io/docs/specs/otel/trace/sdk/) — BatchSpanProcessor, sampler interface, resource
- [HTTP semantic conventions](https://opentelemetry.io/docs/specs/semconv/http/http-spans/) — stable attribute names (`http.request.method`, `http.response.status_code`, etc.)

### Go SDK and Packages

- [OpenTelemetry Go Instrumentation](https://opentelemetry.io/docs/languages/go/instrumentation/) — `TracerProvider` setup, `context.Context` propagation, contrib packages
- [OpenTelemetry Go — Using Instrumentation Libraries](https://opentelemetry.io/docs/languages/go/libraries/) — `otelhttp`, `otelgrpc`, `otelsql`
- [go.opentelemetry.io/otel — Go package reference](https://pkg.go.dev/go.opentelemetry.io/otel)
- [otlptracehttp — OTLP HTTP trace exporter](https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp)
- [otlptracegrpc — OTLP gRPC trace exporter](https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc)
- [otelhttp — HTTP client and server instrumentation](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp)

### Exporter Configuration

- [OTLP Exporter Configuration](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/) — `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, headers, protocol, compression