APM Java Instrumentation

How It Works

The OpenTelemetry Java agent is a standard Java agent JAR attached to the JVM at startup via the -javaagent: flag. It requires no source code changes and no build system integration.

At JVM startup the agent:

1. Registers a ClassFileTransformer using the java.lang.instrument API

2. Intercepts library classes as they are loaded and rewrites their bytecode

3. Inserts span creation, attribute collection, and context propagation calls at instrumentation points (HTTP entry, SQL execution, message dispatch, etc.)

4. Wires all generated spans through a BatchSpanProcessor to the configured OTLP exporter

Because instrumentation is injected at class-load time, your application code is unaware of it. The spans it produces are identical to what you would write manually with the OpenTelemetry SDK.

Prerequisites

• Java 8 or later (any JVM: HotSpot, OpenJ9, GraalVM)

• Access to set JVM startup flags (command line, environment variable, or Kubernetes pod spec)

• Your Kloudfuse API key — see Ingestion Authentication with API Key

OTLP Endpoints

There are three ways to route spans from a Java application to Kloudfuse, depending on how your cluster is deployed. Choose one and use it consistently across your OTEL_* environment variables or -Dotel.* system properties.

• kf-agent gRPC (port 4317)

• kf-agent HTTP (port 4318)

• Direct Ingester (HTTPS, port 443)

The standard deployment runs kf-agent as a DaemonSet or sidecar in the cluster. Applications send spans to it over gRPC on port 4317; the agent batches and forwards them to the Kloudfuse backend.

OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>

Or as JVM system properties:

-Dotel.exporter.otlp.endpoint=http://kf-agent:4317

This is the default protocol used by the Java Agent. No additional protocol flag is needed.

If gRPC is blocked by a firewall or network policy, kf-agent also accepts OTLP/HTTP on port 4318. The SDK appends /v1/traces to the base endpoint automatically.

OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>

When kf-agent is not deployed in the cluster, send spans directly to the Kloudfuse ingester through the nginx ingress over HTTPS. This is the path to use in environments where only port 443 is open.

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_COMPRESSION=gzip
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>

In Kubernetes, inject the API key from a Secret:

env:
  - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
    value: "https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"
  - name: OTEL_EXPORTER_OTLP_COMPRESSION
    value: "gzip"
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value

Create the Secret with the full header expression (not just the raw token):

kubectl create secret generic kloudfuse-api-key \
  --from-literal=value="kf-api-key=<your-api-key>"

Authentication

Pass the Kloudfuse API key as the kf-api-key request header on the OTLP exporter.

• Environment Variable (Recommended)

• JVM System Property

The Java agent reads OTEL_EXPORTER_OTLP_HEADERS automatically — no code or properties file changes are needed:

export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>"

The variable accepts a comma-separated list of name=value pairs:

export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>,x-env=production"

In Kubernetes, pull the value from a Secret so it never appears in source:

env:
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value

Create the Secret with the full name=value header expression:

kubectl create secret generic kloudfuse-api-key \
  --from-literal=value="kf-api-key=<your-api-key>"

Pass the header as a -D flag at startup. This is equivalent to the environment variable — use whichever your deployment process supports:

java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.exporter.otlp.headers=kf-api-key=<your-api-key> \
  -Dotel.service.name=my-java-service \
  -Dotel.exporter.otlp.endpoint=http://kf-agent:4317 \
  -jar myapp.jar

For deployments that use a properties file, add the header there instead of on the command line:

# /etc/otel/otel-config.properties
otel.service.name=my-java-service
otel.exporter.otlp.endpoint=http://kf-agent:4317
otel.exporter.otlp.headers=kf-api-key=<your-api-key>

java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.javaagent.configuration-file=/etc/otel/otel-config.properties \
  -jar myapp.jar

Supported Libraries

The agent instruments 1,000+ libraries across the most common Java ecosystem. Libraries are grouped below by category.

HTTP Servers

HTTP Clients and RPC

Databases and Caches

Messaging

Logging

The agent injects the active trace ID and span ID into log records automatically, enabling log-trace correlation in Kloudfuse.

After the agent injects trace context, each log line will contain trace_id and span_id fields that Kloudfuse uses to link logs directly to the corresponding trace.

Agent Configuration

Key Configuration Properties

All properties below can be set as JVM system properties (-Dotel.property.name=value) or as environment variables (OTEL_PROPERTY_NAME=value).

Selective Instrumentation

Disable Specific Libraries

To silence a noisy or irrelevant instrumentation, disable it by name without affecting everything else:

# Suppress Log4j appender instrumentation (MDC injection)
-Dotel.instrumentation.log4j-appender.enabled=false

# Suppress JDBC instrumentation
-Dotel.instrumentation.jdbc.enabled=false

# Suppress Spring Scheduling (cron jobs creating spans)
-Dotel.instrumentation.spring-scheduling.enabled=false

Start from Zero

To instrument only specific libraries and nothing else:

# Disable everything
-Dotel.instrumentation.common.default-enabled=false

# Re-enable only what you need
-Dotel.instrumentation.opentelemetry-api.enabled=true
-Dotel.instrumentation.spring-webmvc.enabled=true
-Dotel.instrumentation.jdbc.enabled=true
-Dotel.instrumentation.logback-appender.enabled=true

Automatic Instrumentation

Step 1: Download the Agent

Download the latest OpenTelemetry Java agent JAR:

curl -L -o opentelemetry-javaagent.jar \
  https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar

Place the JAR where your JVM can read it at startup — for container deployments, bake it into the image or mount it as a volume (see Kubernetes Integration).

Step 2: Attach the Agent

Add the -javaagent: flag when launching your application. Set the service name and OTLP endpoint at minimum. See OTLP Endpoints above to choose the right endpoint for your deployment.

Using kf-agent (gRPC, port 4317):

java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.service.name=my-java-service \
  -Dotel.exporter.otlp.endpoint=http://kf-agent:4317 \
  -Dotel.resource.attributes=service.namespace=payments,service.version=1.4.2,deployment.environment.name=production \
  -jar myapp.jar

Direct to Kloudfuse Ingester (HTTPS, no kf-agent):

java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.service.name=my-java-service \
  -Dotel.exporter.otlp.traces.endpoint=https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces \
  -Dotel.exporter.otlp.protocol=http/protobuf \
  -Dotel.exporter.otlp.compression=gzip \
  -Dotel.resource.attributes=service.namespace=payments,deployment.environment.name=production \
  -jar myapp.jar

For larger deployments, use a properties file instead of a long list of -D flags:

java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.javaagent.configuration-file=/etc/otel/otel-config.properties \
  -jar myapp.jar

# /etc/otel/otel-config.properties
otel.service.name=my-java-service
# Use otel.exporter.otlp.endpoint for kf-agent, or
# otel.exporter.otlp.traces.endpoint for direct ingester (see OTLP Endpoint Options above)
otel.exporter.otlp.endpoint=http://kf-agent:4317
otel.exporter.otlp.compression=gzip
otel.resource.attributes=service.namespace=payments,service.version=1.4.2,deployment.environment.name=production
otel.traces.sampler=parentbased_traceidratio
otel.traces.sampler.arg=0.05
otel.propagators=tracecontext,baggage
otel.javaagent.logging=application

All OTEL_* environment variables are equivalent to their -Dotel.* system property counterparts:

export OTEL_SERVICE_NAME=my-java-service
export OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317
export OTEL_EXPORTER_OTLP_COMPRESSION=gzip
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.05

java -javaagent:/path/to/opentelemetry-javaagent.jar -jar myapp.jar

Manual Instrumentation

The Java Agent covers most library boundaries automatically. Use manual instrumentation for business logic, background jobs, or code paths that auto-instrumentation does not reach.

Span Annotations (Recommended, Java Agent only)

When the Java Agent is attached, the @WithSpan annotation creates a span around a method automatically. No SDK imports are needed in your business logic:

import io.opentelemetry.instrumentation.annotations.WithSpan;
import io.opentelemetry.instrumentation.annotations.SpanAttribute;

@Service
public class PaymentService {

@WithSpan("process-payment")
    public PaymentResult processPayment(
        @SpanAttribute("payment.method") String method,
        @SpanAttribute("payment.amount")  double amount
    ) {
        // A span named "process-payment" is created automatically.
        // Method parameters become span attributes.
        return charge(method, amount);
    }
}

Add the annotations dependency to your build (no agent version pinning needed — the agent provides the implementation at runtime):

<dependency>
  <groupId>io.opentelemetry.instrumentation</groupId>
  <artifactId>opentelemetry-instrumentation-annotations</artifactId>
  <version>2.8.0</version>
</dependency>

Programmatic Spans with the Java Agent

For more control — custom attributes, conditional error recording, or nested operations — use the SDK Tracer API alongside the Java Agent. The agent registers the TracerProvider with GlobalOpenTelemetry during JVM startup, so GlobalOpenTelemetry.getTracer() is available in main() without any additional setup.

This approach works reliably when an auto-instrumented framework (Spring Boot, Servlet, gRPC) has already initialized before the first manual API call. For standalone batch jobs or background loops where no framework precedes your main() code, see SDK-Only Instrumentation (No Java Agent) below.

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;

Tracer tracer = GlobalOpenTelemetry.getTracer("com.example.payments");

public InvoiceResult processInvoice(String invoiceId) {
    Span span = tracer.spanBuilder("process-invoice")
        .setSpanKind(SpanKind.SERVER)   // required for service map + request metrics
        .setAttribute("invoice.id", invoiceId)
        .setAttribute("invoice.currency", "USD")
        .startSpan();

try (Scope scope = span.makeCurrent()) {
        InvoiceResult result = doProcessing(invoiceId);
        span.setAttribute("invoice.amount", result.getAmount());
        return result;
    } catch (Exception e) {
        span.recordException(e);
        span.setStatus(StatusCode.ERROR, e.getMessage());
        throw e;
    } finally {
        span.end();
    }
}

Always set SpanKind.SERVER on the outermost span of a service handling inbound requests. Kloudfuse derives throughput, latency, and error rate metrics from SERVER spans — spans with SpanKind.INTERNAL (the default) are stored but excluded from service-level aggregations.

SDK-Only Instrumentation (No Java Agent)

For standalone applications — batch jobs, background loops, Kubernetes pods with no HTTP server — the Java Agent is not required. Initialize the SDK directly using AutoConfiguredOpenTelemetrySdk, which reads all OTEL_* environment variables automatically (endpoint, headers, protocol, service name, etc.).

Add these two Maven dependencies (use the BOM or pin all io.opentelemetry artifacts to the same version):

<dependencies>
  <!-- Reads OTEL_* env vars and sets up the SDK as the global TracerProvider -->
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
    <version>1.40.0</version>
  </dependency>
  <!-- OTLP HTTP exporter — use with OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf -->
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
    <version>1.40.0</version>
  </dependency>
</dependencies>

Call AutoConfiguredOpenTelemetrySdk.builder().setResultAsGlobal().build() at the start of main() before creating any tracers:

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;

public class MyService {
    public static void main(String[] args) throws Exception {
        // Initialize SDK from OTEL_* env vars and register as global.
        // Reads: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT,
        //        OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_PROTOCOL, etc.
        AutoConfiguredOpenTelemetrySdk.builder()
                .setResultAsGlobal()
                .build();

Tracer tracer = GlobalOpenTelemetry.getTracer("my-service");

// Root SERVER span — visible in service map and request metrics
        Span serverSpan = tracer.spanBuilder("handle-request")
                .setSpanKind(SpanKind.SERVER)
                .startSpan();
        try (Scope serverScope = serverSpan.makeCurrent()) {

// Child CLIENT span — outbound call, shares the same trace ID
            Span clientSpan = tracer.spanBuilder("db-query")
                    .setSpanKind(SpanKind.CLIENT)
                    .startSpan();
            try (Scope clientScope = clientSpan.makeCurrent()) {
                // ... execute the query ...
            } finally {
                clientSpan.end();
            }

} finally {
            serverSpan.end();
        }
    }
}

The OTEL_* environment variables are identical to those consumed by the Java Agent — no changes to Kubernetes pod specs or Secrets are required when switching between the two approaches.

Kubernetes Integration

Option 1: Init Container with kf-agent (Recommended)

Use an init container to download the agent JAR into a shared volume. This keeps the agent version managed separately from your application image. JAVA_TOOL_OPTIONS is read by the JVM automatically before any other flags — it is the most reliable way to inject -javaagent: in containers where you do not control the entrypoint.

initContainers:
  - name: otel-agent-init
    image: busybox
    command:
      - sh
      - -c
      - |
        wget -q -O /agent/opentelemetry-javaagent.jar \
          https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
    volumeMounts:
      - name: otel-agent
        mountPath: /agent

containers:
  - name: java-app
    image: myapp:latest
    env:
      - name: JAVA_TOOL_OPTIONS
        value: "-javaagent:/agent/opentelemetry-javaagent.jar"
      - name: OTEL_SERVICE_NAME
        value: my-java-service
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://kf-agent:4317
      - name: OTEL_EXPORTER_OTLP_COMPRESSION
        value: gzip
      - name: OTEL_TRACES_SAMPLER
        value: parentbased_traceidratio
      - name: OTEL_TRACES_SAMPLER_ARG
        value: "0.05"
      - name: OTEL_RESOURCE_ATTRIBUTES
        value: service.namespace=payments,deployment.environment.name=production
      - name: OTEL_EXPORTER_OTLP_HEADERS
        valueFrom:
          secretKeyRef:
            name: kloudfuse-api-key
            key: value
    volumeMounts:
      - name: otel-agent
        mountPath: /agent

volumes:
  - name: otel-agent
    emptyDir: {}

Option 2: Direct to Kloudfuse Ingester (No kf-agent)

When kf-agent is not deployed, use the direct HTTPS ingester path with OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (signal-specific). The agent is still used here for auto-instrumentation — only the destination changes.

containers:

• name: java-app image: myapp:latest env:
  • name: JAVA_TOOL_OPTIONS value: "-javaagent:/agent/opentelemetry-javaagent.jar"
  • name: OTEL_SERVICE_NAME value: my-java-service

  Signal-specific endpoint — used verbatim, SDK does not append /v1/traces

  • name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT value: "https:///ingester/otlp/traces"
  • name: OTEL_EXPORTER_OTLP_PROTOCOL value: http/protobuf
  • name: OTEL_EXPORTER_OTLP_COMPRESSION value: gzip
  • name: OTEL_RESOURCE_ATTRIBUTES value: service.namespace=payments,deployment.environment.name=production
  • name: OTEL_EXPORTER_OTLP_HEADERS valueFrom: secretKeyRef: name: kloudfuse-api-key key: value

  volumeMounts:

  • name: otel-agent mountPath: /agent

volumes:

• name: otel-agent emptyDir: {}

### Option 3: Bake into the Image

Alternatively, include the JAR in your Docker image:

```dockerfile
FROM eclipse-temurin:21-jre

COPY opentelemetry-javaagent.jar /otel/opentelemetry-javaagent.jar
COPY myapp.jar /app/myapp.jar

ENV JAVA_TOOL_OPTIONS="-javaagent:/otel/opentelemetry-javaagent.jar"
ENV OTEL_SERVICE_NAME=my-java-service
ENV OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317

ENTRYPOINT ["java", "-jar", "/app/myapp.jar"]

Verify Traces

1. Open Kloudfuse UI → APM → Trace Explorer

2. In the search bar, filter by service.name = my-java-service

3. Click a trace to inspect the span tree, attributes, and timing breakdown

To confirm auto-instrumentation is active, enable agent debug logging temporarily:

-Dotel.javaagent.debug=true

Look for log lines like [opentelemetry-javaagent] Instrumented class: org.springframework.web.servlet.DispatcherServlet to confirm specific libraries are being instrumented.

For production-ready guidance on service naming, sampling strategy, cardinality, error handling, and Java-specific configuration, see APM Best Practices →.

References

Sample Manifest

An example Kubernetes manifest and instructions can be found at APM Demo - Java

Specification and Concepts

• OpenTelemetry Concepts — traces, spans, context propagation, sampling

• OpenTelemetry Trace SDK specification — BatchSpanProcessor, sampler interface, resource

• OTLP Exporter Configuration — OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, headers, protocol, compression

Java Agent

• OpenTelemetry Java Agent — Zero-Code Instrumentation — JAVA_TOOL_OPTIONS, agent configuration, supported libraries

• Java Agent Configuration Reference — all otel. system properties and OTEL_ env vars

• OpenTelemetry Java Agent releases — latest opentelemetry-javaagent.jar download

Java SDK (Manual and SDK-Only Instrumentation)

• OpenTelemetry Java Manual Instrumentation — @WithSpan, programmatic spans, GlobalOpenTelemetry

• OpenTelemetry Java SDK Configuration — AutoConfiguredOpenTelemetrySdk, SDK autoconfigure extension

• OpenTelemetry Java versioning and BOM — aligning SDK and API versions across Maven artifacts