Common Setup Issues :: Kloudfuse Docs

Common Setup Issues

This guide covers the most common problems encountered when instrumenting applications with OpenTelemetry and sending traces to Kloudfuse APM, with steps to diagnose and resolve each issue.

No Traces Appear in UI

Symptoms

Diagnose

  1. Confirm the application is actually emitting spans — add a test span at startup and check application logs for exporter errors.

  2. Verify kf-agent is running and reachable from the application pod:

    kubectl get pods -n <namespace>
kubectl exec -it <app-pod> -- nc -zv kf-agent 4317

  3. Check the OTLP endpoint format — gRPC uses http://kf-agent:4317 (no trailing slash, no path); HTTP/JSON uses http://kf-agent:4318.

  4. Confirm service.name is set — without it the SDK defaults to unknown_service:<process> and may not appear where you expect it.

  5. Confirm the sampler is not set to always_off or traceidratio with arg 0.

Fix

Enable debug logging for the SDK to see exporter errors in the application log output:

# Java agent
-Dotel.javaagent.debug=true

# Python
OTEL_LOG_LEVEL=debug opentelemetry-instrument python app.py

For Go applications, do not import OpenTelemetry internal packages. Instead, enable debug logging through your application’s normal logger and log any exporter or provider initialization errors using public OpenTelemetry APIs. Look for lines containing OTLP export failed, connection refused, or StatusCode. Fix the endpoint or firewall rules and retry.

Partial or Broken Traces

Symptoms

Causes

Fix

Ensure W3C TraceContext propagation is configured globally and that headers are forwarded on every outbound call.

Java — the agent propagates automatically for all supported HTTP clients. If using RestTemplate or WebClient manually, do not recreate the client inside a span — let the agent intercept it.

Python — inject headers on outbound requests using the propagator:

from opentelemetry.propagate import inject

headers = {}
inject(headers)   # adds traceparent / tracestate
response = requests.get("http://other-service/api", headers=headers)

Go — use otelhttp.NewTransport so the HTTP client injects headers automatically:

import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

client := &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
client.Do(req)

If services use different propagator formats, set a consistent list on all services:

OTEL_PROPAGATORS=tracecontext,baggage

Spans Are Dropped or Silently Lost

Symptoms

Causes

The BatchSpanProcessor has a finite in-memory queue. When the application produces spans faster than the exporter can flush them, excess spans are dropped without error.

Fix

Increase the queue and batch sizes using environment variables (applies to Java, Python, and Go):

OTEL_BSP_MAX_QUEUE_SIZE=8192        # default 2048
OTEL_BSP_MAX_EXPORT_BATCH_SIZE=1024 # default 512
OTEL_BSP_SCHEDULE_DELAY=2000        # flush interval in ms, default 5000
OTEL_EXPORTER_OTLP_COMPRESSION=gzip # reduce network overhead

If drops persist, consider reducing trace volume with sampling:

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1   # sample 10% of root traces

Sampling Issues

Symptoms

Causes

Fix

Use parentbased_traceidratio for consistent, end-to-end sampling and set the same rate on all services in a call chain:

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.05   # 5% of new traces; inherited by downstream

Set OTEL_TRACES_SAMPLER_ARG=1.0 temporarily to capture 100% of traces during debugging.

API Key or Authentication Errors

Symptoms

Fix

  1. Confirm the API key is valid and scoped for trace ingestion in the Kloudfuse UI under Administration → API Keys.
  2. Pass the key as an OTLP header, not as a query parameter:
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-key>
  1. In Kubernetes, store the key in a Secret and reference it as an environment variable — do not embed it in a ConfigMap or Helm values file committed to source control:
env:
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value

Java Agent Not Attaching

Symptoms

Causes

Fix

In containers, use JAVA_TOOL_OPTIONS — the JVM reads it before any command-line flags regardless of the entrypoint:

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-javaagent:/agent/opentelemetry-javaagent.jar"

Verify the agent is attaching by looking for this line in the startup log:

[otel.javaagent] ... opentelemetry-javaagent ... attached

If the line is absent, the flag is not reaching the JVM. Check that JAVA_TOOL_OPTIONS is set in the correct container and that the JAR exists at the specified path:

kubectl exec -it <pod> -- ls -l /agent/opentelemetry-javaagent.jar

For Gradle or Maven wrapper scripts that spawn a sub-process, the -javaagent: flag set in JAVA_TOOL_OPTIONS is automatically inherited by child JVMs.

Java Agent 2.x: ClassNotFoundException with Manual API Usage

Symptoms

Causes

The OTel Java agent 2.x bridges manual API usage (calls to GlobalOpenTelemetry, Tracer, etc.) through lazily-loaded instrumentation modules. These modules intercept and transform io.opentelemetry.api.* classes at load time. However, because the modules are loaded lazily — triggered by the first class reference in application code — there is a race: main() can reference GlobalOpenTelemetry before the agent has installed the transformer for it, causing a ClassNotFoundException.

Fix

For manual span creation without an auto-instrumented framework, replace the javaagent with the OTel Java SDK initialized directly in code. Use AutoConfiguredOpenTelemetrySdk to read all OTEL_* environment variables automatically — the same variables the javaagent would have consumed.

Add the following dependencies (Maven):

<dependencies>
  <!-- Reads OTEL_* env vars and sets up the SDK as global -->
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
    <version>1.40.0</version>
  </dependency>
  <!-- OTLP HTTP exporter -->
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
    <version>1.40.0</version>
  </dependency>
</dependencies>

Initialize the SDK at the start of main(), before creating any tracers:

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;

public class MyService {
    public static void main(String[] args) throws Exception {
        // Initialize from OTEL_* env vars and register as global.
        // Reads: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT,
        //        OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_PROTOCOL, etc.
        AutoConfiguredOpenTelemetrySdk.builder()
                .setResultAsGlobal()
                .build();

Tracer tracer = GlobalOpenTelemetry.getTracer("my-service");

Span span = tracer.spanBuilder("my-operation")
                .setSpanKind(SpanKind.SERVER)
                .startSpan();
        try (Scope scope = span.makeCurrent()) {
            // ... do work ...
        } finally {
            span.end();
        }
    }
}

The OTEL_* environment variables are identical to those used with the javaagent — no changes to deployment configuration are required.
The javaagent is still the recommended approach when an auto-instrumented framework (Spring Boot,
Jakarta Servlet, gRPC) is present, because the framework triggers the lazy module load before any
manual API calls are made. The direct SDK approach is appropriate for standalone batch jobs, background
loops, and any application where no auto-instrumented entry point precedes manual span creation.

Python Fork Safety (Gunicorn / uWSGI)

Symptoms

Causes

The BatchSpanProcessor creates background threads and opens a gRPC connection before the server forks worker processes. When the OS forks a process, open file descriptors and threading state are copied into each worker. The forked gRPC connection is no longer valid, and the background flush thread does not exist in the child process.

Fix

Initialize the TracerProvider after the fork by using the server’s post-fork hook, not at module import time.

Gunicorn — use post_fork in the config file:

# gunicorn.conf.py
def post_fork(server, worker):
    from opentelemetry import trace
    from opentelemetry.sdk.trace import TracerProvider
    from opentelemetry.sdk.trace.export import BatchSpanProcessor
    from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
    provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
    trace.set_tracer_provider(provider)

uWSGI — use the @postfork decorator:

from uwsgidecorators import postfork

@postfork
def init_tracing():
    # same provider setup as above
    ...

When using opentelemetry-instrument CLI, pass --no-auto-instrumentation and initialize manually inside post_fork instead.

Go: Spans Not Linked Across Goroutines

Symptoms

Causes

Go’s context.Context is not shared automatically between goroutines. If you launch a goroutine without passing the context, the goroutine’s spans have no parent.

Fix

Always pass ctx explicitly into every goroutine and function that creates spans:

// Wrong — goroutine has no trace context
go func() {
    _, span := tracer.Start(context.Background(), "background-job")
    defer span.End()
}()

// Correct — parent context is passed in
go func(ctx context.Context) {
    _, span := tracer.Start(ctx, "background-job")
    defer span.End()
}(ctx)

For work queues or goroutine pools, propagate the context through the work item struct rather than capturing it from the enclosing scope (which may have already finished by the time the goroutine runs).

Spans Not Visible in Trace Explorer and Missing from Service Map

Symptoms

Causes

Kloudfuse derives service-level metrics (throughput, latency, error rate) from spans with SpanKind = SERVER. If every span from a service has SpanKind = INTERNAL — the default when no kind is specified — the spans are stored and searchable but are not counted as inbound requests for that service.

SpanKind = INTERNAL is the correct kind for in-process business logic that sits inside a request, not for the entry point of the request itself. Using it on the outermost handler span means the platform cannot distinguish a service boundary from an internal function call.

SpanKind Reference

SpanKind Value When to use
SERVER 2 The span represents an inbound request handled by this service — HTTP, gRPC, message consumer. Use this on the outermost handler span. Required for service map and request metrics.
CLIENT 3 An outbound call to another service or external resource — HTTP client, database query, cache read.
PRODUCER 4 Publishing a message to a queue or topic (Kafka, SQS, RabbitMQ).
CONSUMER 5 Receiving and processing a message from a queue or topic.
INTERNAL 1 In-process business logic with no cross-service boundary. This is the default when no kind is set. Do not use it for the entry-point span of a service.

Fix

Always set SpanKind = SERVER on the outermost span of any service that handles inbound requests.

Java — the javaagent sets the correct kind automatically for all supported HTTP and gRPC servers. For manual spans representing an inbound call, set the kind explicitly:

Span span = tracer.spanBuilder("process-request")
    .setSpanKind(SpanKind.SERVER)
    .startSpan();

Python — opentelemetry-instrument with opentelemetry-instrumentation-flask (or Django, FastAPI) sets SpanKind.SERVER automatically. For manual spans:

with tracer.start_as_current_span("process-request", kind=trace.SpanKind.SERVER) as span:
    ...

Go — use otelhttp.NewHandler to wrap HTTP handlers. It sets SpanKind = SERVER, emits the correct HTTP semantic convention attributes, and extracts incoming W3C trace context automatically:

import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

// otelhttp sets SpanKind = SERVER and attributes http.method, http.status_code,
// http.target, net.host.name, net.protocol.version automatically.
http.Handle("/api/orders", otelhttp.NewHandler(ordersHandler, "GET /api/orders"))

For manual Go spans, pass trace.WithSpanKind:

ctx, span := tracer.Start(ctx, "handle-request",
    trace.WithSpanKind(trace.SpanKindServer),
)
defer span.End()

Go: Semantic Convention Limitations in otelhttp v0.53.x

otelhttp v0.53.x (compatible with go.opentelemetry.io/otel v1.28.x) emits the older, now-deprecated HTTP attribute names:

Emitted by otelhttp ≤ v0.55 Stable replacement (otelhttp ≥ v0.56)
http.method http.request.method
http.status_code http.response.status_code
http.target url.path
http.scheme url.scheme
net.host.name server.address
net.sock.peer.addr network.peer.address

Both the old and new attribute names are accepted by Kloudfuse. If you need the stable names — for example to match a filter or dashboard that uses http.request.method — upgrade otelhttp to v0.56.0 or later and align all otel core modules to v1.31.0 or later at the same time. The version alignment requirement is not limited to the three modules below — it applies to every go.opentelemetry.io/otel/ module in your go.mod. Common ones include otel/metric, otel/log, otel/bridge/, and any other otel/exporters/* you import. Running go mod tidy will catch any that are out of sync.

# Upgrade all otel core modules together, then the contrib package
go get go.opentelemetry.io/otel@v1.31.0
go get go.opentelemetry.io/otel/sdk@v1.31.0
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp@v1.31.0
# add any other go.opentelemetry.io/otel/* modules your project uses at the same version
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.56.0
go mod tidy

Wrong OTLP Protocol (gRPC vs HTTP)

Symptoms

Fix

kf-agent listens on two ports with different protocols:

Port Protocol Endpoint format
4317 gRPC (OTLP/gRPC) http://kf-agent:4317 — no path
4318 HTTP/JSON (OTLP/HTTP) http://kf-agent:4318 — no path; SDK appends /v1/traces automatically

Match the SDK exporter type to the port. Most SDK defaults use gRPC on port 4317. If you switch to HTTP, update both the endpoint and the exporter package:

# gRPC (default)
OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317

# HTTP
OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

SDK Version Conflicts

Symptoms

Fix

Java — always use the BOM to align versions. Do not mix opentelemetry-sdk and opentelemetry-api versions:

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-bom</artifactId>
      <version>1.40.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

Python — pin all opentelemetry-* packages to the same version in requirements.txt. Run opentelemetry-bootstrap -a install after updating to reinstall the correct instrumentation packages for your installed libraries.

Go — run go mod tidy after updating any go.opentelemetry.io/* module. Check that the otel core module version matches the contrib package versions — they must be aligned to the same minor version series.

TLS / Connection Errors to kf-agent

Symptoms

Fix

If kf-agent requires TLS, configure the exporter with the CA certificate. For Java:

-Dotel.exporter.otlp.certificate=/etc/ssl/certs/kf-agent-ca.pem

For Go:

import "google.golang.org/grpc/credentials"

creds, _ := credentials.NewClientTLSFromFile("/etc/ssl/certs/kf-agent-ca.pem", "")
exporter, _ := otlptracegrpc.New(ctx,
    otlptracegrpc.WithEndpoint("kf-agent:4317"),
    otlptracegrpc.WithTLSCredentials(creds),
)

For development or internal clusters where TLS is not required, use WithInsecure() (Go) or omit the certificate property (Java/Python). Never disable TLS verification in production — use the correct CA certificate instead.

High Cardinality Causing Ingestion Throttling

Symptoms

Fix

Avoid using high-cardinality values as span names or attribute keys. Span names should identify the operation type, not the specific value:

Avoid Use instead
GET /users/a3f9b1c2-…​ GET /users/{id}
process order 98312 process-order (put the order ID in an attribute)
SELECT * FROM orders WHERE id=12345 SELECT orders (use db.statement attribute for the full query)

Use attributes for variable data and keep the span name templated:

Span span = tracer.spanBuilder("process-order").startSpan();
span.setAttribute("order.id", orderId);   // attribute, not span name

See Cardinality Management for a full list of anti-patterns and mitigations.

Unsupported Environment or Language

Symptoms

Fix

Agent Deployment Failures

Symptoms

Fix

  1. Check pod status and events:
    kubectl describe pod -n <namespace> -l app=kf-agent
kubectl logs -n <namespace> -l app=kf-agent --previous
  2. Verify required Helm values are set: clusterName, apiKey, and imagePullPolicy.
  3. Confirm network policies allow inbound traffic on ports 4317 (gRPC) and 4318 (HTTP) from application pods.
  4. If pods are in Pending, check resource requests — the agent requires at least 256 MiB memory.

See Kubernetes Setup for full deployment prerequisites.

Multiple Services Show the Same Name

Symptoms

Fix

Set a unique service.name per application. The most reliable way is via environment variable so it can be configured per deployment without code changes:

OTEL_SERVICE_NAME=payments-api
OTEL_RESOURCE_ATTRIBUTES=service.namespace=payments,deployment.environment.name=production

For Java with the agent:

-Dotel.service.name=payments-api
-Dotel.resource.attributes=service.namespace=payments,deployment.environment.name=production

Avoid setting service.name to a pod name or hostname — use a stable logical name that identifies the service role, not the instance.

Validating Your Setup

After instrumentation, confirm data is flowing correctly:

  1. Open Kloudfuse UI → APM → Trace Explorer
  2. Filter by service.name = <your-service>
  3. Send a test request to your application — a trace should appear within a few seconds
  4. Click a trace and verify: span names, attributes, parent-child relationships, and timing
  5. Check APM → Service Map to confirm dependencies between services are visible

If traces appear but attributes are missing, re-check that resource attributes (service.namespace, deployment.environment.name) are set and that the SDK version supports the semantic conventions you expect.

Still Need Help?

Collect the following before contacting support:

References

The following OpenTelemetry documentation was used as the basis for the guidance on this page.

Specification and Concepts

Java

Python

Go