# APM Java Instrumentation

## How It Works

The OpenTelemetry Java agent is a standard Java agent JAR attached to the JVM at startup via the `-javaagent:` flag. It requires no source code changes and no build system integration.

At JVM startup the agent:

1. Registers a `ClassFileTransformer` using the `java.lang.instrument` API

2. Intercepts library classes as they are loaded and rewrites their bytecode

3. Inserts span creation, attribute collection, and context propagation calls at instrumentation points (HTTP entry, SQL execution, message dispatch, etc.)

4. Wires all generated spans through a `BatchSpanProcessor` to the configured OTLP exporter

Because instrumentation is injected at class-load time, your application code is unaware of it. The spans it produces are identical to what you would write manually with the OpenTelemetry SDK.

## Prerequisites

- Java 8 or later (any JVM: HotSpot, OpenJ9, GraalVM)

- Access to set JVM startup flags (command line, environment variable, or Kubernetes pod spec)

- Your Kloudfuse API key — see [Ingestion Authentication with API Key](https://docs.kloudfuse.com/platform/4.0.2/administration/authentication/ingestion-api-key/)

## OTLP Endpoints

There are three ways to route spans from a Java application to Kloudfuse, depending on how your cluster is deployed. Choose one and use it consistently across your `OTEL_*` environment variables or `-Dotel.*` system properties.

- kf-agent gRPC (port 4317)

- kf-agent HTTP (port 4318)

- Direct Ingester (HTTPS, port 443)

The standard deployment runs `kf-agent` as a DaemonSet or sidecar in the cluster. Applications send spans to it over gRPC on port 4317; the agent batches and forwards them to the Kloudfuse backend.

```bash
OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>
```

Or as JVM system properties:

```bash
-Dotel.exporter.otlp.endpoint=http://kf-agent:4317
```

This is the default protocol used by the Java Agent. No additional protocol flag is needed.

If gRPC is blocked by a firewall or network policy, `kf-agent` also accepts OTLP/HTTP on port 4318. The SDK appends `/v1/traces` to the base endpoint automatically.

```bash
OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>
```

When `kf-agent` is not deployed in the cluster, send spans directly to the Kloudfuse ingester through the nginx ingress over HTTPS. This is the path to use in environments where only port 443 is open.

```bash
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_COMPRESSION=gzip
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>
```

|     |     |
| --- | --- |
|  | Use `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` (the signal-specific variable), not<br>`OTEL_EXPORTER_OTLP_ENDPOINT` (the base variable). The base variable causes the SDK to append<br>`/v1/traces` to whatever path you provide, producing the wrong URL: |
| Variable used | Resulting URL sent to the ingester |
| --- | --- |
| `OTEL_EXPORTER_OTLP_ENDPOINT=https://host/ingester/otlp/traces` | `https://host/ingester/otlp/traces/v1/traces` ❌ (double suffix) |
| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://host/ingester/otlp/traces` | `https://host/ingester/otlp/traces` ✓ (used verbatim) |

In Kubernetes, inject the API key from a Secret:

```yaml
env:
  - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
    value: "https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"
  - name: OTEL_EXPORTER_OTLP_COMPRESSION
    value: "gzip"
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value
```

Create the Secret with the full header expression (not just the raw token):

```bash
kubectl create secret generic kloudfuse-api-key \
  --from-literal=value="kf-api-key=<your-api-key>"
```

## Authentication

Pass the Kloudfuse API key as the `kf-api-key` request header on the OTLP exporter.

- Environment Variable (Recommended)

- JVM System Property

The Java agent reads `OTEL_EXPORTER_OTLP_HEADERS` automatically — no code or properties file changes are needed:

```bash
export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>"
```

The variable accepts a comma-separated list of `name=value` pairs:

```bash
export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>,x-env=production"
```

In Kubernetes, pull the value from a Secret so it never appears in source:

```yaml
env:
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value
```

Create the Secret with the full `name=value` header expression:

```bash
kubectl create secret generic kloudfuse-api-key \
  --from-literal=value="kf-api-key=<your-api-key>"
```

Pass the header as a `-D` flag at startup. This is equivalent to the environment variable — use whichever your deployment process supports:

```bash
java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.exporter.otlp.headers=kf-api-key=<your-api-key> \
  -Dotel.service.name=my-java-service \
  -Dotel.exporter.otlp.endpoint=http://kf-agent:4317 \
  -jar myapp.jar
```

For deployments that use a properties file, add the header there instead of on the command line:

```properties
# /etc/otel/otel-config.properties
otel.service.name=my-java-service
otel.exporter.otlp.endpoint=http://kf-agent:4317
otel.exporter.otlp.headers=kf-api-key=<your-api-key>
```

```bash
java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.javaagent.configuration-file=/etc/otel/otel-config.properties \
  -jar myapp.jar
```

|     |     |
| --- | --- |
|  | Do not hardcode the API key value in a properties file committed to source control. Read it from an environment variable or a secrets manager and write the file at deploy time, or use `OTEL_EXPORTER_OTLP_HEADERS` instead. |

## Supported Libraries

The agent instruments 1,000+ libraries across the most common Java ecosystem. Libraries are grouped below by category.

### HTTP Servers

| Library | Versions |
| --- | --- |
| Spring MVC / Spring Boot | Spring 3.1+, Spring Boot 1.5 – 3.x |
| Servlet API (Tomcat, Jetty, Undertow, Glassfish) | Servlet 2.3+ |
| Netty HTTP server | 4.0+ |
| Quarkus (Vert.x-based) | 1.0+ |
| Micronaut HTTP server | 2.0+ |
| JAX-RS (Jersey, RESTEasy) | 1.0+ |
| Akka HTTP | 10.0+ |

### HTTP Clients and RPC

| Library | Versions |
| --- | --- |
| Apache HttpClient (4.x and 5.x) | 4.0+, 5.0+ |
| OkHttp | 3.0+ |
| Java `HttpClient` (JDK 11+) | JDK 11+ |
| gRPC client and server | 1.6+ |
| `HttpURLConnection` | all |
| Spring WebClient / WebFlux | 5.0+ |

### Databases and Caches

| Library | Versions |
| --- | --- |
| JDBC (all drivers: PostgreSQL, MySQL, Oracle, MSSQL, H2) | any JDBC driver |
| Hibernate ORM | 3.3+ |
| Spring Data JPA / Spring Data JDBC | 1.8+ |
| MongoDB driver | 3.1+ |
| Redis (Jedis and Lettuce) | 1.4+, 4.0+ |
| Elasticsearch client | 6.0+ |
| Cassandra driver | 3.0+ |
| Couchbase client | 2.0+ |

### Messaging

| Library | Versions |
| --- | --- |
| Apache Kafka (producer and consumer) | 0.11+ |
| RabbitMQ AMQP client | 2.7+ |
| AWS SDK v2 (SQS, SNS, S3, DynamoDB) | 2.2+ |
| AWS SDK v1 | 1.11+ |
| Spring Kafka | 2.7+ |
| Spring AMQP / Spring Rabbit | 1.0+ |
| JMS (ActiveMQ, IBM MQ) | 1.1+ |

### Logging

The agent injects the active trace ID and span ID into log records automatically, enabling log-trace correlation in Kloudfuse.

| Library | Versions |
| --- | --- |
| Log4j 2 (MDC injection) | 2.7+ |
| Logback (MDC injection) | 1.0+ |
| Log4j 1.x | 1.0+ |
| JUL (`java.util.logging`) | any |

After the agent injects trace context, each log line will contain `trace_id` and `span_id` fields that Kloudfuse uses to link logs directly to the corresponding trace.

## Agent Configuration

### Key Configuration Properties

All properties below can be set as JVM system properties (`-Dotel.property.name=value`) or as environment variables (`OTEL_PROPERTY_NAME=value`).

| Property | Description | Default |
| --- | --- | --- |
| `otel.service.name` | Service name — the most important attribute. Must be set. | `unknown_service:<executable>` |
| `otel.exporter.otlp.endpoint` | Base OTLP endpoint. The SDK appends `/v1/traces` for HTTP exporters. Use for kf-agent on port 4317 (gRPC) or 4318 (HTTP). | `http://localhost:4317` |
| `otel.exporter.otlp.traces.endpoint` | Signal-specific traces endpoint. Used verbatim — the SDK does NOT append `/v1/traces`. Required when sending directly to the Kloudfuse ingester at a non-standard path. | — |
| `otel.exporter.otlp.protocol` | Exporter wire protocol: `grpc` or `http/protobuf`. Default is gRPC. Set to `http/protobuf` when using port 4318 or the direct ingester path. | `grpc` |
| `otel.exporter.otlp.compression` | Set to `gzip` to reduce network bandwidth. | none |
| `otel.traces.sampler` | Sampling strategy. Use `parentbased_traceidratio` in production. | `parentbased_always_on` |
| `otel.traces.sampler.arg` | Sample rate for `traceidratio` and `parentbased_traceidratio` (0.0–1.0). | `1.0` |
| `otel.propagators` | Context propagation formats. Default covers W3C TraceContext and Baggage. | `tracecontext,baggage` |
| `otel.resource.attributes` | Comma-separated key=value resource attributes (namespace, version, env). | — |
| `otel.javaagent.configuration-file` | Path to a `.properties` file containing all OTel configuration. | — |
| `otel.javaagent.logging` | Agent log routing: `simple` (stderr), `none`, or `application` (routes to SLF4J). | `simple` |
| `otel.javaagent.debug` | Set to `true` to enable verbose agent debug logging. | `false` |
| `otel.bsp.schedule.delay` | How often the `BatchSpanProcessor` exports, in milliseconds. | `5000` |
| `otel.bsp.max.queue.size` | Maximum span queue size before spans are dropped. | `2048` |
| `otel.bsp.max.export.batch.size` | Maximum spans per export batch. | `512` |
| `otel.resource.providers.aws.enabled` | Enable AWS resource detection (EC2, ECS, EKS, Lambda). | `false` |
| `otel.resource.providers.gcp.enabled` | Enable GCP resource detection (GCE, GKE, Cloud Run). | `false` |
| `otel.resource.providers.azure.enabled` | Enable Azure resource detection (VMs, AKS). | `false` |

## Selective Instrumentation

### Disable Specific Libraries

To silence a noisy or irrelevant instrumentation, disable it by name without affecting everything else:

```bash
# Suppress Log4j appender instrumentation (MDC injection)
-Dotel.instrumentation.log4j-appender.enabled=false

# Suppress JDBC instrumentation
-Dotel.instrumentation.jdbc.enabled=false

# Suppress Spring Scheduling (cron jobs creating spans)
-Dotel.instrumentation.spring-scheduling.enabled=false
```

### Start from Zero

To instrument only specific libraries and nothing else:

```bash
# Disable everything
-Dotel.instrumentation.common.default-enabled=false

# Re-enable only what you need
-Dotel.instrumentation.opentelemetry-api.enabled=true
-Dotel.instrumentation.spring-webmvc.enabled=true
-Dotel.instrumentation.jdbc.enabled=true
-Dotel.instrumentation.logback-appender.enabled=true
```

## Automatic Instrumentation

### Step 1: Download the Agent

Download the latest OpenTelemetry Java agent JAR:

```bash
curl -L -o opentelemetry-javaagent.jar \
  https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
```

Place the JAR where your JVM can read it at startup — for container deployments, bake it into the image or mount it as a volume (see [Kubernetes Integration](https://docs.kloudfuse.com/platform/4.0.2/data-collection/apm/instrument-java/#kubernetes-integration)).

### Step 2: Attach the Agent

Add the `-javaagent:` flag when launching your application. Set the service name and OTLP endpoint at minimum. See [OTLP Endpoints](https://docs.kloudfuse.com/platform/4.0.2/data-collection/apm/instrument-java/#otlp-endpoints) above to choose the right endpoint for your deployment.

**Using kf-agent (gRPC, port 4317):**

```bash
java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.service.name=my-java-service \
  -Dotel.exporter.otlp.endpoint=http://kf-agent:4317 \
  -Dotel.resource.attributes=service.namespace=payments,service.version=1.4.2,deployment.environment.name=production \
  -jar myapp.jar
```

**Direct to Kloudfuse Ingester (HTTPS, no kf-agent):**

```bash
java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.service.name=my-java-service \
  -Dotel.exporter.otlp.traces.endpoint=https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces \
  -Dotel.exporter.otlp.protocol=http/protobuf \
  -Dotel.exporter.otlp.compression=gzip \
  -Dotel.resource.attributes=service.namespace=payments,deployment.environment.name=production \
  -jar myapp.jar
```

For larger deployments, use a properties file instead of a long list of `-D` flags:

```bash
java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.javaagent.configuration-file=/etc/otel/otel-config.properties \
  -jar myapp.jar
```

```properties
# /etc/otel/otel-config.properties
otel.service.name=my-java-service
# Use otel.exporter.otlp.endpoint for kf-agent, or
# otel.exporter.otlp.traces.endpoint for direct ingester (see OTLP Endpoint Options above)
otel.exporter.otlp.endpoint=http://kf-agent:4317
otel.exporter.otlp.compression=gzip
otel.resource.attributes=service.namespace=payments,service.version=1.4.2,deployment.environment.name=production
otel.traces.sampler=parentbased_traceidratio
otel.traces.sampler.arg=0.05
otel.propagators=tracecontext,baggage
otel.javaagent.logging=application
```

All `OTEL_*` environment variables are equivalent to their `-Dotel.*` system property counterparts:

```bash
export OTEL_SERVICE_NAME=my-java-service
export OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317
export OTEL_EXPORTER_OTLP_COMPRESSION=gzip
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.05

java -javaagent:/path/to/opentelemetry-javaagent.jar -jar myapp.jar
```

## Manual Instrumentation

The Java Agent covers most library boundaries automatically. Use manual instrumentation for business logic, background jobs, or code paths that auto-instrumentation does not reach.

### Span Annotations (Recommended, Java Agent only)

When the Java Agent is attached, the `@WithSpan` annotation creates a span around a method automatically. No SDK imports are needed in your business logic:

```java
import io.opentelemetry.instrumentation.annotations.WithSpan;
import io.opentelemetry.instrumentation.annotations.SpanAttribute;

@Service
public class PaymentService {

@WithSpan("process-payment")
    public PaymentResult processPayment(
        @SpanAttribute("payment.method") String method,
        @SpanAttribute("payment.amount")  double amount
    ) {
        // A span named "process-payment" is created automatically.
        // Method parameters become span attributes.
        return charge(method, amount);
    }
}
```

Add the annotations dependency to your build (no agent version pinning needed — the agent provides the implementation at runtime):

```xml
<dependency>
  <groupId>io.opentelemetry.instrumentation</groupId>
  <artifactId>opentelemetry-instrumentation-annotations</artifactId>
  <version>2.8.0</version>
</dependency>
```

### Programmatic Spans with the Java Agent

For more control — custom attributes, conditional error recording, or nested operations — use the SDK
`Tracer` API alongside the Java Agent. The agent registers the `TracerProvider` with `GlobalOpenTelemetry`
during JVM startup, so `GlobalOpenTelemetry.getTracer()` is available in `main()` without any additional setup.

This approach works reliably when an auto-instrumented framework (Spring Boot, Servlet, gRPC) has already
initialized before the first manual API call. For standalone batch jobs or background loops where no
framework precedes your `main()` code, see [SDK-Only Instrumentation (No Java Agent)](https://docs.kloudfuse.com/platform/4.0.2/data-collection/apm/instrument-java/#sdk-only) below.

```java
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;

Tracer tracer = GlobalOpenTelemetry.getTracer("com.example.payments");

public InvoiceResult processInvoice(String invoiceId) {
    Span span = tracer.spanBuilder("process-invoice")
        .setSpanKind(SpanKind.SERVER)   // required for service map + request metrics
        .setAttribute("invoice.id", invoiceId)
        .setAttribute("invoice.currency", "USD")
        .startSpan();

try (Scope scope = span.makeCurrent()) {
        InvoiceResult result = doProcessing(invoiceId);
        span.setAttribute("invoice.amount", result.getAmount());
        return result;
    } catch (Exception e) {
        span.recordException(e);
        span.setStatus(StatusCode.ERROR, e.getMessage());
        throw e;
    } finally {
        span.end();
    }
}
```

|     |     |
| --- | --- |
|  | Call both `span.recordException(e)` and `span.setStatus(StatusCode.ERROR, …​)` together. They do<br>not imply each other — omitting `setStatus` means the span is still marked OK and won’t appear in error<br>rate calculations. |

Always set `SpanKind.SERVER` on the outermost span of a service handling inbound requests. Kloudfuse
derives throughput, latency, and error rate metrics from `SERVER` spans — spans with `SpanKind.INTERNAL`
(the default) are stored but excluded from service-level aggregations.

### SDK-Only Instrumentation (No Java Agent)

For standalone applications — batch jobs, background loops, Kubernetes pods with no HTTP server — the
Java Agent is not required. Initialize the SDK directly using `AutoConfiguredOpenTelemetrySdk`, which
reads all `OTEL_*` environment variables automatically (endpoint, headers, protocol, service name, etc.).

Add these two Maven dependencies (use the BOM or pin all `io.opentelemetry` artifacts to the same version):

```xml
<dependencies>
  <!-- Reads OTEL_* env vars and sets up the SDK as the global TracerProvider -->
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
    <version>1.40.0</version>
  </dependency>
  <!-- OTLP HTTP exporter — use with OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf -->
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
    <version>1.40.0</version>
  </dependency>
</dependencies>
```

Call `AutoConfiguredOpenTelemetrySdk.builder().setResultAsGlobal().build()` at the start of `main()`
before creating any tracers:

```java
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;

public class MyService {
    public static void main(String[] args) throws Exception {
        // Initialize SDK from OTEL_* env vars and register as global.
        // Reads: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT,
        //        OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_PROTOCOL, etc.
        AutoConfiguredOpenTelemetrySdk.builder()
                .setResultAsGlobal()
                .build();

Tracer tracer = GlobalOpenTelemetry.getTracer("my-service");

// Root SERVER span — visible in service map and request metrics
        Span serverSpan = tracer.spanBuilder("handle-request")
                .setSpanKind(SpanKind.SERVER)
                .startSpan();
        try (Scope serverScope = serverSpan.makeCurrent()) {

// Child CLIENT span — outbound call, shares the same trace ID
            Span clientSpan = tracer.spanBuilder("db-query")
                    .setSpanKind(SpanKind.CLIENT)
                    .startSpan();
            try (Scope clientScope = clientSpan.makeCurrent()) {
                // ... execute the query ...
            } finally {
                clientSpan.end();
            }

} finally {
            serverSpan.end();
        }
    }
}
```

The `OTEL_*` environment variables are identical to those consumed by the Java Agent — no changes to
Kubernetes pod specs or Secrets are required when switching between the two approaches.

## Kubernetes Integration

### Option 1: Init Container with kf-agent (Recommended)

Use an init container to download the agent JAR into a shared volume. This keeps the agent version
managed separately from your application image. `JAVA_TOOL_OPTIONS` is read by the JVM automatically
before any other flags — it is the most reliable way to inject `-javaagent:` in containers where you
do not control the entrypoint.

```yaml
initContainers:
  - name: otel-agent-init
    image: busybox
    command:
      - sh
      - -c
      - |
        wget -q -O /agent/opentelemetry-javaagent.jar \
          https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
    volumeMounts:
      - name: otel-agent
        mountPath: /agent

containers:
  - name: java-app
    image: myapp:latest
    env:
      - name: JAVA_TOOL_OPTIONS
        value: "-javaagent:/agent/opentelemetry-javaagent.jar"
      - name: OTEL_SERVICE_NAME
        value: my-java-service
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://kf-agent:4317
      - name: OTEL_EXPORTER_OTLP_COMPRESSION
        value: gzip
      - name: OTEL_TRACES_SAMPLER
        value: parentbased_traceidratio
      - name: OTEL_TRACES_SAMPLER_ARG
        value: "0.05"
      - name: OTEL_RESOURCE_ATTRIBUTES
        value: service.namespace=payments,deployment.environment.name=production
      - name: OTEL_EXPORTER_OTLP_HEADERS
        valueFrom:
          secretKeyRef:
            name: kloudfuse-api-key
            key: value
    volumeMounts:
      - name: otel-agent
        mountPath: /agent

volumes:
  - name: otel-agent
    emptyDir: {}
```

### Option 2: Direct to Kloudfuse Ingester (No kf-agent)

When `kf-agent` is not deployed, use the direct HTTPS ingester path with `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`
(signal-specific). The agent is still used here for auto-instrumentation — only the destination changes.

containers:
  - name: java-app
    image: myapp:latest
    env:
      - name: JAVA_TOOL_OPTIONS
        value: "-javaagent:/agent/opentelemetry-javaagent.jar"
      - name: OTEL_SERVICE_NAME
        value: my-java-service
      # Signal-specific endpoint — used verbatim, SDK does not append /v1/traces
      - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
        value: "https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces"
      - name: OTEL_EXPORTER_OTLP_PROTOCOL
        value: http/protobuf
      - name: OTEL_EXPORTER_OTLP_COMPRESSION
        value: gzip
      - name: OTEL_RESOURCE_ATTRIBUTES
        value: service.namespace=payments,deployment.environment.name=production
      - name: OTEL_EXPORTER_OTLP_HEADERS
        valueFrom:
          secretKeyRef:
            name: kloudfuse-api-key
            key: value
    volumeMounts:
      - name: otel-agent
        mountPath: /agent

volumes:
  - name: otel-agent
    emptyDir: {}
```

### Option 3: Bake into the Image

Alternatively, include the JAR in your Docker image:

```dockerfile
FROM eclipse-temurin:21-jre

COPY opentelemetry-javaagent.jar /otel/opentelemetry-javaagent.jar
COPY myapp.jar /app/myapp.jar

ENV JAVA_TOOL_OPTIONS="-javaagent:/otel/opentelemetry-javaagent.jar"
ENV OTEL_SERVICE_NAME=my-java-service
ENV OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317

ENTRYPOINT ["java", "-jar", "/app/myapp.jar"]
```

## Verify Traces

1. Open **Kloudfuse UI → APM → Trace Explorer**

2. In the search bar, filter by `service.name = my-java-service`

3. Click a trace to inspect the span tree, attributes, and timing breakdown

To confirm auto-instrumentation is active, enable agent debug logging temporarily:

```bash
-Dotel.javaagent.debug=true
```

Look for log lines like `[opentelemetry-javaagent] Instrumented class: org.springframework.web.servlet.DispatcherServlet` to confirm specific libraries are being instrumented.

For production-ready guidance on service naming, sampling strategy, cardinality, error handling, and Java-specific configuration, see [APM Best Practices →](https://docs.kloudfuse.com/platform/4.0.2/data-collection/apm/best-practices/).

## References

### Sample Manifest

An example Kubernetes manifest and instructions can be found at [APM Demo - Java](https://github.com/kloudfuse-ext/customer/tree/main/agents/apm/demo/java)

### Specification and Concepts

- [OpenTelemetry Concepts](https://opentelemetry.io/docs/concepts/) — traces, spans, context propagation, sampling

- [OpenTelemetry Trace SDK specification](https://opentelemetry.io/docs/specs/otel/trace/sdk/) — BatchSpanProcessor, sampler interface, resource

- [OTLP Exporter Configuration](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/) — `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, headers, protocol, compression

### Java Agent

- [OpenTelemetry Java Agent — Zero-Code Instrumentation](https://opentelemetry.io/docs/zero-code/java/agent/) — `JAVA_TOOL_OPTIONS`, agent configuration, supported libraries

- [Java Agent Configuration Reference](https://opentelemetry.io/docs/zero-code/java/agent/configuration/) — all `otel.` **system properties and `OTEL_`** env vars

- [OpenTelemetry Java Agent releases](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest) — latest `opentelemetry-javaagent.jar` download

### Java SDK (Manual and SDK-Only Instrumentation)

- [OpenTelemetry Java Manual Instrumentation](https://opentelemetry.io/docs/languages/java/instrumentation/) — `@WithSpan`, programmatic spans, `GlobalOpenTelemetry`

- [OpenTelemetry Java SDK Configuration](https://opentelemetry.io/docs/languages/java/configuration/) — `AutoConfiguredOpenTelemetrySdk`, SDK autoconfigure extension

- [OpenTelemetry Java versioning and BOM](https://github.com/open-telemetry/opentelemetry-java/blob/main/VERSIONING.md) — aligning SDK and API versions across Maven artifacts