APM Java Instrumentation

How It Works

The OpenTelemetry Java agent is a standard Java agent JAR attached to the JVM at startup via the -javaagent: flag. It requires no source code changes and no build system integration.

At JVM startup the agent:

1. Registers a ClassFileTransformer using the java.lang.instrument API
2. Intercepts library classes as they are loaded and rewrites their bytecode
3. Inserts span creation, attribute collection, and context propagation calls at instrumentation points (HTTP entry, SQL execution, message dispatch, etc.)
4. Wires all generated spans through a BatchSpanProcessor to the configured OTLP exporter

Because instrumentation is injected at class-load time, your application code is unaware of it. The spans it produces are identical to what you would write manually with the OpenTelemetry SDK.

Prerequisites

• Java 8 or later (any JVM: HotSpot, OpenJ9, GraalVM)
• Access to set JVM startup flags (command line, environment variable, or Kubernetes pod spec)
• Your Kloudfuse API key — see Ingestion Authentication with API Key

OTLP Endpoints

There are three ways to route spans from a Java application to Kloudfuse, depending on how your cluster is deployed. Choose one and use it consistently across your OTEL_* environment variables or -Dotel.* system properties.

• kf-agent gRPC (port 4317)
• kf-agent HTTP (port 4318)
• Direct Ingester (HTTPS, port 443)

The standard deployment runs kf-agent as a DaemonSet or sidecar in the cluster. Applications send spans to it over gRPC on port 4317; the agent batches and forwards them to the Kloudfuse backend.

OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>

Or as JVM system properties:

-Dotel.exporter.otlp.endpoint=http://kf-agent:4317

If gRPC is blocked by a firewall or network policy, kf-agent also accepts OTLP/HTTP on port 4318.

OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>

When kf-agent is not deployed in the cluster, send spans directly to the Kloudfuse ingester through the nginx ingress over HTTPS.

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_COMPRESSION=gzip
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>

In Kubernetes, inject the API key from a Secret:

env:
  - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
    value: "https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"
  - name: OTEL_EXPORTER_OTLP_COMPRESSION
    value: "gzip"
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value

Create the Secret with the full header expression:

kubectl create secret generic kloudfuse-api-key \
  --from-literal=value="kf-api-key=<your-api-key>"

Authentication

Pass the Kloudfuse API key as the kf-api-key request header on the OTLP exporter.

• Environment Variable (Recommended)
• JVM System Property

export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>"

The variable accepts a comma-separated list of name=value pairs:

export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>,x-env=production"

In Kubernetes, pull the value from a Secret so it never appears in source:

env:
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value

Create the Secret with the full name=value header expression:

kubectl create secret generic kloudfuse-api-key \
  --from-literal=value="kf-api-key=<your-api-key>"

Pass the header as a -D flag at startup:

java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.exporter.otlp.headers=kf-api-key=<your-api-key> \
  -Dotel.service.name=my-java-service \
  -Dotel.exporter.otlp.endpoint=http://kf-agent:4317 \
  -jar myapp.jar

For deployments that use a properties file, add the header there instead of on the command line:

# /etc/otel/otel-config.properties
otel.service.name=my-java-service
otel.exporter.otlp.endpoint=http://kf-agent:4317
otel.exporter.otlp.headers=kf-api-key=<your-api-key>

Supported Libraries

The agent instruments 1,000+ libraries across the most common Java ecosystem. Libraries are grouped below by category.

HTTP Servers

HTTP Clients and RPC

Databases and Caches

Messaging

Logging

The agent injects the active trace ID and span ID into log records automatically.

Agent Configuration

Key Configuration Properties

All properties below can be set as JVM system properties (-Dotel.property.name=value) or as environment variables (OTEL_PROPERTY_NAME=value).

Selective Instrumentation

Disable Specific Libraries

To suppress specific instrumentation:

# Suppress Log4j appender instrumentation
-Dotel.instrumentation.log4j-appender.enabled=false

# Suppress JDBC instrumentation
-Dotel.instrumentation.jdbc.enabled=false

Start from Zero

To instrument only specific libraries:

# Disable everything
-Dotel.instrumentation.common.default-enabled=false

# Re-enable only what you need
-Dotel.instrumentation.opentelemetry-api.enabled=true
-Dotel.instrumentation.spring-webmvc.enabled=true

Automatic Instrumentation

Step 1: Download the Agent

Download the latest OpenTelemetry Java agent JAR:

curl -L -o opentelemetry-javaagent.jar \
  https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar

Step 2: Attach the Agent

Add the -javaagent: flag when launching your application:

java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.service.name=my-java-service \
  -Dotel.exporter.otlp.endpoint=http://kf-agent:4317 \
  -jar myapp.jar

Manual Instrumentation

Span Annotations (Recommended)

When the Java Agent is attached, the @WithSpan annotation creates a span around a method automatically:

import io.opentelemetry.instrumentation.annotations.WithSpan;
import io.opentelemetry.instrumentation.annotations.SpanAttribute;

@Service
public class PaymentService {

@WithSpan("process-payment")
    public PaymentResult processPayment(
        @SpanAttribute("payment.method") String method,
        @SpanAttribute("payment.amount")  double amount
    ) {
        return charge(method, amount);
    }
}

Programmatic Spans with the Java Agent

For more control — custom attributes, conditional error recording, or nested operations — use the SDK Tracer API.

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;

Tracer tracer = GlobalOpenTelemetry.getTracer("com.example.payments");

public InvoiceResult processInvoice(String invoiceId) {
    Span span = tracer.spanBuilder("process-invoice").startSpan();
    try (Scope scope = span.makeCurrent()) {
        // ... processing ...
        return result;
    } finally {
        span.end();
    }
}

Kubernetes Integration

Option 1: Init Container with kf-agent

Use an init container to.download the agent JAR into a shared volume.

initContainers:
  - name: otel-agent-init
    image: busybox
    command:
      - sh
      - -c
      - |
        wget -q -O /agent/opentelemetry-javaagent.jar \
          https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
    volumeMounts:
      - name: otel-agent
        mountPath: /agent

Option 2: Direct to Kloudfuse Ingester

When kf-agent is not deployed, use the direct HTTPS ingester path:

containers:
  - name: java-app
    image: myapp:latest
    env:
      - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
        value: "https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces"

Option 3: Bake into the Image

Include the JAR in your Docker image:

FROM eclipse-temurin:21-jre

COPY opentelemetry-javaagent.jar /otel/opentelemetry-javaagent.jar

ENTRYPOINT ["java", "-jar", "/app/myapp.jar"]

Verify Traces

1. Open Kloudfuse UI → APM → Trace Explorer
2. In the search bar, filter by service.name = my-java-service