APM Instrumentation Best Practices :: Kloudfuse Docs

APM Instrumentation Best Practices

Effective APM instrumentation requires more than attaching an agent—it requires deliberate decisions about service naming, span design, sampling strategy, and attribute cardinality. This page consolidates production-grade guidance for instrumenting services with Kloudfuse APM across Java, Python, and Go.

Naming Conventions

Good naming is the foundation of useful trace data. Span names and service names are indexed by the platform and used as grouping keys—mistakes here are expensive to fix after data is collected.

Service Names

service.name is the single most important resource attribute. Every service must set it explicitly.

Use a stable, lowercase, kebab-case identifier: checkout-service, inventory-api, payments-worker
Names must be identical across all horizontally scaled instances of the same service
Never rely on the SDK default (unknown_service:<executable>)—it prevents all per-service filtering

Span Names

Span names must be low-cardinality. They are indexed by the backend and used to group spans for latency histograms and error rates.

Operation type	Correct span name	Incorrect span name
HTTP server	`GET /users/{userId}`	`GET /users/12345`
HTTP client	`POST /api/orders`	`POST https://api.example.com/api/orders?token=abc`
Database	`SELECT users`	`SELECT * FROM users WHERE id=99`
Message queue	`publish shop.orders`	`publish shop.orders msg-uuid-1234`
Background job	`process-invoice`	`process-invoice 2024-03-15T10:32:11Z`

Do not embed user IDs, request IDs, order IDs, timestamps, or full URLs in span names. Those values belong in span attributes where they are stored per-span without affecting aggregation keys.

Resource Attributes

Resource attributes describe the entity that produced telemetry. They are set once at SDK startup and attached to all spans, metrics, and logs.

Core Attributes

Attribute	Requirement	Description
`service.name`	Required	Logical name of the service. Must be set—no exceptions.
`service.namespace`	Recommended	Namespace grouping services by team or domain (e.g., `payments`, `platform`).
`service.version`	Recommended	Semantic version, git SHA, or build tag (e.g., `2.3.1`, `a01dbef8`).
`service.instance.id`	Recommended	Globally unique identifier for this instance. Use pod name or a UUID.
`deployment.environment.name`	Recommended	Deployment tier: `production`, `staging`, `development`.

Setting Attributes via Environment Variables

All OpenTelemetry SDKs support these standard environment variables:

OTEL_SERVICE_NAME=checkout-service
OTEL_RESOURCE_ATTRIBUTES=service.namespace=payments,service.version=1.4.2,deployment.environment.name=production,service.instance.id=pod-abc-123

Automatic Resource Detection

Resource detectors populate cloud and infrastructure metadata automatically at SDK startup—no manual configuration required for most environments.

Java (javaagent):

# Enable cloud provider detection (off by default)
-Dotel.resource.providers.aws.enabled=true
-Dotel.resource.providers.gcp.enabled=true
-Dotel.resource.providers.azure.enabled=true

Go:

res, err := resource.New(ctx,
    resource.WithFromEnv(),    // reads OTEL_RESOURCE_ATTRIBUTES
    resource.WithProcess(),    // PID, executable name
    resource.WithOS(),         // OS type
    resource.WithContainer(),  // container ID
    resource.WithHost(),       // hostname
)

For Kubernetes environments, use the k8sattributes processor to enrich all telemetry with pod name, namespace, node name, and deployment name without requiring any SDK changes.

Sampling Strategy

Sampling controls what fraction of traces are collected and exported. The right strategy balances visibility against cost and storage volume.

Head-Based Sampling

Sampling decisions are made at the root span before any child spans are created. The decision propagates to all downstream services via the traceparent header.

Sampler	When to use
`parentbased_always_on`	Low-traffic services or development—capture everything.
`parentbased_traceidratio`	Recommended for production. Samples a configured percentage of new traces while respecting sampling decisions from upstream callers.
`parentbased_always_off`	Silence a noisy service that adds no debugging value.

Tail-Based Sampling

Tail-based sampling defers the decision until after the full trace is collected, enabling criteria such as "always capture error traces" or "always capture the slowest 1%." This is implemented in the OpenTelemetry Collector, not in the SDK.

Context Propagation

Trace context must be propagated across service boundaries for distributed traces to assemble correctly.

W3C TraceContext (Default)

All OpenTelemetry SDKs default to W3C TraceContext propagation. Two HTTP headers carry the trace:

traceparent: contains the trace ID, span ID, and sampling flag
tracestate: carries vendor-specific metadata including sampling probability

Error Recording

Two operations are always required together when an error occurs. They do not imply each other—omitting either produces incomplete data.

RecordError / record_exception — records the exception as a span event with exception.type, exception.message, and exception.stacktrace
SetStatus(ERROR, description) — marks the span as failed for aggregation, alerting, and error rate calculations

Cardinality Management

High cardinality in span attributes and metric labels degrades backend performance and increases storage cost. It is one of the most common production problems with APM deployments.

Anti-Patterns to Avoid

Anti-pattern	Problem
User ID, session ID, or order ID in span name	Every unique ID creates a new grouping bucket. Aggregations become meaningless.
Full URL (`/users/12345`) in span name	Every URL is unique. No two requests share a span name.
Request ID as a metric label	Metric series count explodes. Backends run out of memory.
Timestamp embedded in any label or name	Always unique. Destroys aggregation.
Unbound enum values in metric labels	Product IDs, SKUs, transaction codes—each new value adds a new series.

What to Do Instead

Put user IDs, order IDs, and request IDs in span attributes, not span names or metric labels
Use route templates (/users/{id}) as span names, never resolved paths
Use the OpenTelemetry Collector attributes processor to drop or hash high-cardinality fields before they reach your metrics backend
Set OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT to prevent large exception stacktraces from bloating per-span storage

Performance Tuning

BatchSpanProcessor

Always use BatchSpanProcessor in production. The alternative (SimpleSpanProcessor) exports synchronously on every span end, adding latency to every instrumented operation.

Language-Specific Best Practices

Each language SDK has distinct setup patterns, lifecycle requirements, and common pitfalls. The sections below consolidate the most important production guidance for Java, Python, and Go.

Java

Agent Configuration File

otel.service.name=checkout-service
otel.traces.exporter=otlp
otel.exporter.otlp.endpoint=http://kf-agent:4317
otel.exporter.otlp.compression=gzip
otel.traces.sampler=parentbased_traceidratio
otel.traces.sampler.arg=0.05

Python

Auto-Instrumentation Setup

Install all detected library instrumentors in one step:

pip install opentelemetry-distro[otlp] opentelemetry-instrumentation
opentelemetry-bootstrap -a install    # detects and installs instrumentation packages

Go

SDK Initialization and Graceful Shutdown

Always call tp.Shutdown() on exit. Without it, the BatchSpanProcessor may not flush spans that are still queued when the process exits.

func initTracer(ctx context.Context) (func(context.Context) error, error) {
    exp, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("kf-agent:4317"),
        otlptracegrpc.WithInsecure(),
    )
    
    res, err := resource.Merge(
        resource.Default(),
        resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName("my-service"),
            semconv.ServiceVersion("1.0.0"),
            attribute.String("deployment.environment.name", "production"),
        ),
    )
    if err != nil {
        return nil, err
    }

tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp),
        sdktrace.WithResource(res),
        sdktrace.WithSampler(sdktrace.ParentBased(
            sdktrace.TraceIDRatioBased(0.05),
        )),
    )

otel.SetTracerProvider(tp)
    return tp.Shutdown, nil
}

Semantic Conventions Reference

OpenTelemetry defines standard attribute names for common operation types. Using these ensures Kloudfuse and other backends can correctly parse and display trace data.

HTTP

Attribute	Stability	Use
`http.request.method`	Stable	`GET`, `POST`, `PUT`, `DELETE`
`http.response.status_code`	Stable	Integer: `200`, `404`, `500`
`http.route`	Stable	Matched route template: `/users/{id}`
`url.full`	Stable	Absolute URL for client spans. Replaces deprecated `http.url`.
`url.scheme`	Stable	`http` or `https`
`server.address`	Stable	Server hostname or IP
`server.port`	Stable	Server port number
`error.type`	Stable	HTTP status code string or exception type for errors

Production Checklist

Set OTEL_SERVICE_NAME explicitly — never rely on the unknown_service default
Set service.namespace, service.version, and deployment.environment.name in resource attributes
Use parentbased_traceidratio sampler at 5–10% for most production services
Use BatchSpanProcessor (default) — never SimpleSpanProcessor in production
Enable gzip compression: `OTEL_EXPORTER_OTLP_COMPRESSION=gzip

References

The guidance on this page is drawn from the following OpenTelemetry specifications, SDK documentation, and language-specific guides.