# Common Setup Issues

This guide covers the most common problems encountered when instrumenting applications with OpenTelemetry and sending traces to Kloudfuse APM, with steps to diagnose and resolve each issue.

## No Traces Appear in UI

### Symptoms
- APM Trace Explorer is empty after setup
- Service does not appear in the service list
- No spans appear even after sending requests

### Diagnose
1. Confirm the application is actually emitting spans — add a test span at startup and check application logs for exporter errors.
2. Verify `kf-agent` is running and reachable from the application pod:
   
   ```bash
   kubectl get pods -n <namespace>
   kubectl exec -it <app-pod> -- nc -zv kf-agent 4317
   ```
   
3. Check the OTLP endpoint format — gRPC uses `http://kf-agent:4317` (no trailing slash, no path); HTTP/JSON uses `http://kf-agent:4318`.
4. Confirm `service.name` is set — without it the SDK defaults to `unknown_service:<process>` and may not appear where you expect it.
5. Confirm the sampler is not set to `always_off` or `traceidratio` with arg `0`.

### Fix
Enable debug logging for the SDK to see exporter errors in the application log output:

```bash
# Java agent
-Dotel.javaagent.debug=true

# Python
OTEL_LOG_LEVEL=debug opentelemetry-instrument python app.py
```
For Go applications, do not import OpenTelemetry internal packages. Instead, enable debug logging through your application’s normal logger and log any exporter or provider initialization errors using public OpenTelemetry APIs. Look for lines containing `OTLP export failed`, `connection refused`, or `StatusCode`. Fix the endpoint or firewall rules and retry.

## Partial or Broken Traces

### Symptoms
- Trace timeline shows orphan spans with no parent
- `trace_id` is the same but spans appear under separate traces
- Parent-child relationships are missing between services

### Causes
- Trace context headers (`traceparent`, `tracestate`) are not forwarded on outbound HTTP or gRPC calls
- An intermediate service strips or rewrites headers
- Services use different propagator formats (e.g., B3 vs W3C TraceContext)

### Fix
Ensure W3C TraceContext propagation is configured globally and that headers are forwarded on every outbound call.

**Java** — the agent propagates automatically for all supported HTTP clients. If using `RestTemplate` or `WebClient` manually, do not recreate the client inside a span — let the agent intercept it.

**Python** — inject headers on outbound requests using the propagator:

```python
from opentelemetry.propagate import inject

headers = {}
inject(headers)   # adds traceparent / tracestate
response = requests.get("http://other-service/api", headers=headers)
```

**Go** — use `otelhttp.NewTransport` so the HTTP client injects headers automatically:

```go
import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

client := &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
client.Do(req)
```

If services use different propagator formats, set a consistent list on all services:

```bash
OTEL_PROPAGATORS=tracecontext,baggage
```

## Spans Are Dropped or Silently Lost

### Symptoms
- Traces appear incomplete under high load but are fine at low throughput
- Application logs show `BatchSpanProcessor queue is full` or `dropping span`

### Causes
The `BatchSpanProcessor` has a finite in-memory queue. When the application produces spans faster than the exporter can flush them, excess spans are dropped without error.

### Fix
Increase the queue and batch sizes using environment variables (applies to Java, Python, and Go):

```bash
OTEL_BSP_MAX_QUEUE_SIZE=8192        # default 2048
OTEL_BSP_MAX_EXPORT_BATCH_SIZE=1024 # default 512
OTEL_BSP_SCHEDULE_DELAY=2000        # flush interval in ms, default 5000
OTEL_EXPORTER_OTLP_COMPRESSION=gzip # reduce network overhead
```

If drops persist, consider reducing trace volume with sampling:

```bash
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1   # sample 10% of root traces
```

## Sampling Issues

### Symptoms
- Only a small fraction of traces appear, even at low throughput
- Trace Explorer shows gaps in timeline
- `parentbased_traceidratio` produces inconsistent coverage

### Causes
- Sample rate set too low — `OTEL_TRACES_SAMPLER_ARG=0.01` keeps only 1% of traces
- `parentbased_traceidratio` inherits the parent’s decision: if an upstream service samples out a trace, all downstream spans for that trace are also dropped
- `traceidratio` (without `parentbased_`) ignores the parent decision, which can produce inconsistent traces

### Fix
Use `parentbased_traceidratio` for consistent, end-to-end sampling and set the same rate on all services in a call chain:

```bash
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.05   # 5% of new traces; inherited by downstream
```

Set `OTEL_TRACES_SAMPLER_ARG=1.0` temporarily to capture 100% of traces during debugging.

## API Key or Authentication Errors

### Symptoms
- `kf-agent` logs show `401 Unauthorized` or `403 Forbidden`
- No data appears in the UI despite correct endpoint and instrumentation

### Fix
1. Confirm the API key is valid and scoped for trace ingestion in the Kloudfuse UI under **Administration → API Keys**.
2. Pass the key as an OTLP header, not as a query parameter:

```bash
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-key>
```

3. In Kubernetes, store the key in a Secret and reference it as an environment variable — do not embed it in a ConfigMap or Helm values file committed to source control:

```yaml
env:
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value
```

## Java Agent Not Attaching

### Symptoms
- No spans appear from a Java application despite having the agent JAR
- Application starts without any `opentelemetry-javaagent` log lines
- `Caused by: java.lang.ClassNotFoundException` for OTel classes

### Causes
- `-javaagent:` flag is not reaching the JVM — the flag must be passed to the JVM, not the application
- In containers, the entrypoint uses `exec java` directly but `JAVA_TOOL_OPTIONS` is not set
- Multiple `-javaagent:` flags conflict (e.g., a profiler and the OTel agent)
- The JAR path contains spaces or is not accessible at the path specified

### Fix
In containers, use `JAVA_TOOL_OPTIONS` — the JVM reads it before any command-line flags regardless of the entrypoint:

```yaml
env:
  - name: JAVA_TOOL_OPTIONS
    value: "-javaagent:/agent/opentelemetry-javaagent.jar"
```

Verify the agent is attaching by looking for this line in the startup log:

```bash
[otel.javaagent] ... opentelemetry-javaagent ... attached
```

If the line is absent, the flag is not reaching the JVM. Check that `JAVA_TOOL_OPTIONS` is set in the correct container and that the JAR exists at the specified path:

```bash
kubectl exec -it <pod> -- ls -l /agent/opentelemetry-javaagent.jar
```

For Gradle or Maven wrapper scripts that spawn a sub-process, the `-javaagent:` flag set in `JAVA_TOOL_OPTIONS` is automatically inherited by child JVMs.

## Java Agent 2.x: ClassNotFoundException with Manual API Usage

### Symptoms
- `java.lang.NoClassDefFoundError: io/opentelemetry/api/GlobalOpenTelemetry` at startup
- The javaagent IS attached (its startup log lines appear) but `main()` crashes immediately
- The javaagent log shows `Loading instrumentation opentelemetry-api` entries appearing **after** the exception
- The error stack trace points to application code that calls `GlobalOpenTelemetry.getTracer()`

### Causes
The OTel Java agent 2.x bridges manual API usage (calls to `GlobalOpenTelemetry`, `Tracer`, etc.) through lazily-loaded instrumentation modules. These modules intercept and transform `io.opentelemetry.api.*` classes at load time. However, because the modules are loaded lazily — triggered by the first class reference in application code — there is a race: `main()` can reference `GlobalOpenTelemetry` before the agent has installed the transformer for it, causing a `ClassNotFoundException`.
### Fix
For manual span creation without an auto-instrumented framework, replace the javaagent with the OTel Java SDK initialized directly in code. Use `AutoConfiguredOpenTelemetrySdk` to read all `OTEL_*` environment variables automatically — the same variables the javaagent would have consumed.

Add the following dependencies (Maven):

```xml
<dependencies>
  <!-- Reads OTEL_* env vars and sets up the SDK as global -->
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
    <version>1.40.0</version>
  </dependency>
  <!-- OTLP HTTP exporter -->
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
    <version>1.40.0</version>
  </dependency>
</dependencies>
```

Initialize the SDK at the start of `main()`, before creating any tracers:

```java
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;

public class MyService {
    public static void main(String[] args) throws Exception {
        // Initialize from OTEL_* env vars and register as global.
        // Reads: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT,
        //        OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_PROTOCOL, etc.
        AutoConfiguredOpenTelemetrySdk.builder()
                .setResultAsGlobal()
                .build();

Tracer tracer = GlobalOpenTelemetry.getTracer("my-service");

Span span = tracer.spanBuilder("my-operation")
                .setSpanKind(SpanKind.SERVER)
                .startSpan();
        try (Scope scope = span.makeCurrent()) {
            // ... do work ...
        } finally {
            span.end();
        }
    }
}
```

The `OTEL_*` environment variables are identical to those used with the javaagent — no changes to deployment configuration are required.

|     |     |
| --- | --- |
|  | The javaagent is still the recommended approach when an auto-instrumented framework (Spring Boot,<br>Jakarta Servlet, gRPC) is present, because the framework triggers the lazy module load before any<br>manual API calls are made. The direct SDK approach is appropriate for standalone batch jobs, background<br>loops, and any application where no auto-instrumented entry point precedes manual span creation. |

## Python Fork Safety (Gunicorn / uWSGI)

### Symptoms
- Traces appear for the first few requests then stop completely
- Application hangs or deadlocks after startup under Gunicorn or uWSGI
- `BrokenPipeError` or `ResourceWarning: unclosed socket` in logs

### Causes
The `BatchSpanProcessor` creates background threads and opens a gRPC connection before the server forks worker processes. When the OS forks a process, open file descriptors and threading state are copied into each worker. The forked gRPC connection is no longer valid, and the background flush thread does not exist in the child process.

### Fix
Initialize the `TracerProvider` **after** the fork by using the server’s post-fork hook, not at module import time.

**Gunicorn** — use `post_fork` in the config file:

```python
# gunicorn.conf.py
def post_fork(server, worker):
    from opentelemetry import trace
    from opentelemetry.sdk.trace import TracerProvider
    from opentelemetry.sdk.trace.export import BatchSpanProcessor
    from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
    provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
    trace.set_tracer_provider(provider)
```

**uWSGI** — use the `@postfork` decorator:

```python
from uwsgidecorators import postfork

@postfork
def init_tracing():
    # same provider setup as above
    ...
```

When using `opentelemetry-instrument` CLI, pass `--no-auto-instrumentation` and initialize manually inside `post_fork` instead.

## Go: Spans Not Linked Across Goroutines

### Symptoms
- Spans created in goroutines appear as root spans instead of children
- Service map shows the same service calling itself
- `trace_id` differs between the parent handler and work done in goroutines

### Causes
Go’s `context.Context` is not shared automatically between goroutines. If you launch a goroutine without passing the context, the goroutine’s spans have no parent.

### Fix
Always pass `ctx` explicitly into every goroutine and function that creates spans:

```go
// Wrong — goroutine has no trace context
go func() {
    _, span := tracer.Start(context.Background(), "background-job")
    defer span.End()
}()

// Correct — parent context is passed in
go func(ctx context.Context) {
    _, span := tracer.Start(ctx, "background-job")
    defer span.End()
}(ctx)
```

For work queues or goroutine pools, propagate the context through the work item struct rather than capturing it from the enclosing scope (which may have already finished by the time the goroutine runs).

## Spans Not Visible in Trace Explorer and Missing from Service Map

### Symptoms
- Spans do not appear in the Trace Explorer but the service does appear under Services
- Request throughput and latency metrics are absent for the service
- Error rate is always zero even when requests fail
- Service-level aggregations (p50/p95/p99) show no data

### Causes
Kloudfuse derives service-level metrics (throughput, latency, error rate) from spans with `SpanKind = SERVER`. If every span from a service has `SpanKind = INTERNAL` — the default when no kind is specified — the spans are stored and searchable but are not counted as inbound requests for that service.

`SpanKind = INTERNAL` is the correct kind for in-process business logic that sits _inside_ a request, not for the entry point of the request itself. Using it on the outermost handler span means the platform cannot distinguish a service boundary from an internal function call.

### SpanKind Reference
| SpanKind | Value | When to use |
| --- | --- | --- |
| `SERVER` | 2 | The span represents an inbound request handled by this service — HTTP, gRPC, message consumer. Use this on the outermost handler span. Required for service map and request metrics. |
| `CLIENT` | 3 | An outbound call to another service or external resource — HTTP client, database query, cache read. |
| `PRODUCER` | 4 | Publishing a message to a queue or topic (Kafka, SQS, RabbitMQ). |
| `CONSUMER` | 5 | Receiving and processing a message from a queue or topic. |
| `INTERNAL` | 1 | In-process business logic with no cross-service boundary. This is the default when no kind is set. Do not use it for the entry-point span of a service. |

### Fix
Always set `SpanKind = SERVER` on the outermost span of any service that handles inbound requests.

**Java** — the javaagent sets the correct kind automatically for all supported HTTP and gRPC servers. For manual spans representing an inbound call, set the kind explicitly:

```java
Span span = tracer.spanBuilder("process-request")
    .setSpanKind(SpanKind.SERVER)
    .startSpan();
```

**Python** — `opentelemetry-instrument` with `opentelemetry-instrumentation-flask` (or Django, FastAPI) sets `SpanKind.SERVER` automatically. For manual spans:

```python
with tracer.start_as_current_span("process-request", kind=trace.SpanKind.SERVER) as span:
    ...
```

**Go** — use `otelhttp.NewHandler` to wrap HTTP handlers. It sets `SpanKind = SERVER`, emits the correct HTTP semantic convention attributes, and extracts incoming W3C trace context automatically:

```go
import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

// otelhttp sets SpanKind = SERVER and attributes http.method, http.status_code,
// http.target, net.host.name, net.protocol.version automatically.
http.Handle("/api/orders", otelhttp.NewHandler(ordersHandler, "GET /api/orders"))
```

For manual Go spans, pass `trace.WithSpanKind`:

```go
ctx, span := tracer.Start(ctx, "handle-request",
    trace.WithSpanKind(trace.SpanKindServer),
)
defer span.End()
```

### Go: Semantic Convention Limitations in otelhttp v0.53.x

`otelhttp` v0.53.x (compatible with `go.opentelemetry.io/otel` v1.28.x) emits the
[older, now-deprecated](https://opentelemetry.io/docs/specs/semconv/http/http-spans/) HTTP attribute names:
| Emitted by otelhttp ≤ v0.55 | Stable replacement (otelhttp ≥ v0.56) |
| --- | --- |
| `http.method` | `http.request.method` |
| `http.status_code` | `http.response.status_code` |
| `http.target` | `url.path` |
| `http.scheme` | `url.scheme` |
| `net.host.name` | `server.address` |
| `net.sock.peer.addr` | `network.peer.address` |

Both the old and new attribute names are accepted by Kloudfuse. If you need the stable names — for example to match a filter or dashboard that uses `http.request.method` — upgrade `otelhttp` to v0.56.0 or later and align **all** otel core modules to v1.31.0 or later at the same time.
The version alignment requirement is not limited to the three modules below — it applies to every `go.opentelemetry.io/otel/` **module in your `go.mod`. Common ones include `otel/metric`, `otel/log`,** **`otel/bridge/`**, and any other `otel/exporters/*` you import. Running `go mod tidy` will catch any that are out of sync.

```none
# Upgrade all otel core modules together, then the contrib package
go get go.opentelemetry.io/otel@v1.31.0
go get go.opentelemetry.io/otel/sdk@v1.31.0
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp@v1.31.0
# add any other go.opentelemetry.io/otel/* modules your project uses at the same version
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.56.0
go mod tidy
```

## Wrong OTLP Protocol (gRPC vs HTTP)

### Symptoms
- Export fails with `HTTP 415 Unsupported Media Type` or `HTTP 405 Method Not Allowed`
- gRPC exporter logs `failed to connect` or `transport: error while dialing`
- Data arrives at agent but is malformed

### Fix
`kf-agent` listens on two ports with different protocols:
| Port | Protocol | Endpoint format |
| --- | --- | --- |
| `4317` | gRPC (OTLP/gRPC) | `http://kf-agent:4317` — no path |
| `4318` | HTTP/JSON (OTLP/HTTP) | `http://kf-agent:4318` — no path; SDK appends `/v1/traces` automatically |

Match the SDK exporter type to the port. Most SDK defaults use gRPC on port `4317`. If you switch to HTTP, update both the endpoint and the exporter package:

```bash
# gRPC (default)
OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317

# HTTP
OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```

## SDK Version Conflicts

### Symptoms
- `NoSuchMethodError` or `ClassCastException` in Java at startup
- Python shows `ImportError: cannot import name 'TracerProvider' from 'opentelemetry.sdk.trace'`
- Go build fails with incompatible module versions

### Fix
**Java** — always use the BOM to align versions. Do not mix `opentelemetry-sdk` and `opentelemetry-api` versions:

```xml
<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-bom</artifactId>
      <version>1.40.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>
```

**Python** — pin all `opentelemetry-*` packages to the same version in `requirements.txt`. Run `opentelemetry-bootstrap -a install` after updating to reinstall the correct instrumentation packages for your installed libraries.

**Go** — run `go mod tidy` after updating any `go.opentelemetry.io/*` module. Check that the `otel` core module version matches the `contrib` package versions — they must be aligned to the same minor version series.

## TLS / Connection Errors to kf-agent

### Symptoms
- Exporter logs show `x509: certificate signed by unknown authority`
- `failed to obtain credentials` or `transport: authentication handshake failed`
- Traces arrive in development but not in production (where TLS is enforced)

### Fix
If `kf-agent` requires TLS, configure the exporter with the CA certificate. For Java:

```bash
-Dotel.exporter.otlp.certificate=/etc/ssl/certs/kf-agent-ca.pem
```

For Go:

```go
import "google.golang.org/grpc/credentials"

creds, _ := credentials.NewClientTLSFromFile("/etc/ssl/certs/kf-agent-ca.pem", "")
exporter, _ := otlptracegrpc.New(ctx,
    otlptracegrpc.WithEndpoint("kf-agent:4317"),
    otlptracegrpc.WithTLSCredentials(creds),
)
```

For development or internal clusters where TLS is not required, use `WithInsecure()` (Go) or omit the certificate property (Java/Python). Never disable TLS verification in production — use the correct CA certificate instead.

## High Cardinality Causing Ingestion Throttling

### Symptoms
- Trace ingestion rate drops or data is throttled in the Kloudfuse Usage Dashboard
- Span attributes contain user IDs, UUIDs, or request payloads
- Service map is slow to render with hundreds of unique operation names

### Fix
Avoid using high-cardinality values as span names or attribute keys. Span names should identify the operation type, not the specific value:
| Avoid | Use instead |
| --- | --- |
| `GET /users/a3f9b1c2-…​` | `GET /users/{id}` |
| `process order 98312` | `process-order` (put the order ID in an attribute) |
| `SELECT * FROM orders WHERE id=12345` | `SELECT orders` (use `db.statement` attribute for the full query) |

Use attributes for variable data and keep the span name templated:

```java
Span span = tracer.spanBuilder("process-order").startSpan();
span.setAttribute("order.id", orderId);   // attribute, not span name
```

See [Cardinality Management](https://docs.kloudfuse.com/platform/4.1.0/data-collection/apm/best-practices/#cardinality-management) for a full list of anti-patterns and mitigations.

## Unsupported Environment or Language

### Symptoms
- No OpenTelemetry SDK available for the runtime or language version
- Manual span creation fails at import or compile time

### Fix
- Consult the [OpenTelemetry Registry](https://opentelemetry.io/ecosystem/registry/) for community-maintained SDKs and instrumentation libraries.
- For runtimes without a native SDK, use a sidecar or proxy approach: route traffic through Envoy with the OpenTelemetry extension, or emit spans using the OTLP HTTP/JSON wire format from any HTTP client.
- For legacy JVM languages (Groovy, Scala, Kotlin), the OpenTelemetry Java agent instruments all JVM bytecode regardless of the source language.

## Agent Deployment Failures

### Symptoms
- Helm chart fails to install or upgrade
- `kf-agent` pods are in `CrashLoopBackOff` or `Pending`
- Agent starts but immediately exits

### Fix
1. Check pod status and events:
   ```bash
   kubectl describe pod -n <namespace> -l app=kf-agent
   kubectl logs -n <namespace> -l app=kf-agent --previous
   ```
2. Verify required Helm values are set: `clusterName`, `apiKey`, and `imagePullPolicy`.
3. Confirm network policies allow inbound traffic on ports `4317` (gRPC) and `4318` (HTTP) from application pods.
4. If pods are in `Pending`, check resource requests — the agent requires at least 256 MiB memory.

See [Kubernetes Setup](https://docs.kloudfuse.com/platform/4.1.0/data-collection/apm/kubernetes-agent/) for full deployment prerequisites.

## Multiple Services Show the Same Name

### Symptoms
- All traces appear under `unknown_service` or a single service name
- Different applications are merged in the service map

### Fix
Set a unique `service.name` per application. The most reliable way is via environment variable so it can be configured per deployment without code changes:

```bash
OTEL_SERVICE_NAME=payments-api
OTEL_RESOURCE_ATTRIBUTES=service.namespace=payments,deployment.environment.name=production
```

For Java with the agent:

```bash
-Dotel.service.name=payments-api
-Dotel.resource.attributes=service.namespace=payments,deployment.environment.name=production
```

Avoid setting `service.name` to a pod name or hostname — use a stable logical name that identifies the service role, not the instance.

## Validating Your Setup

After instrumentation, confirm data is flowing correctly:
1. Open **Kloudfuse UI → APM → Trace Explorer**
2. Filter by `service.name = <your-service>`
3. Send a test request to your application — a trace should appear within a few seconds
4. Click a trace and verify: span names, attributes, parent-child relationships, and timing
5. Check **APM → Service Map** to confirm dependencies between services are visible

If traces appear but attributes are missing, re-check that resource attributes (`service.namespace`, `deployment.environment.name`) are set and that the SDK version supports the semantic conventions you expect.

## Still Need Help?

Collect the following before contacting support:
- SDK language and version (`opentelemetry-javaagent-X.Y.Z`, `opentelemetry-sdk==X.Y.Z`, `go.opentelemetry.io/otel vX.Y.Z`)
- Exporter debug log output (a few lines showing the failed export)
- A sample `trace_id` from Kloudfuse UI that illustrates the problem
- Your `OTEL_*` environment variables (redact `OTEL_EXPORTER_OTLP_HEADERS`)
- `kubectl describe` output for the `kf-agent` pod if the agent is not reachable

## References
The following OpenTelemetry documentation was used as the basis for the guidance on this page.

### Specification and Concepts
- [OpenTelemetry Concepts](https://opentelemetry.io/docs/concepts/) — traces, spans, context propagation, sampling
- [OpenTelemetry Trace SDK specification](https://opentelemetry.io/docs/specs/otel/trace/sdk/) — BatchSpanProcessor, sampler interface, resource
- [Context and Propagators specification](https://opentelemetry.io/docs/specs/otel/context/)
- [W3C TraceContext specification](https://www.w3.org/TR/trace-context/) — `traceparent` and `tracestate` header format

### Java
- [OpenTelemetry Java Agent — Zero-Code Instrumentation](https://opentelemetry.io/docs/zero-code/java/agent/) — `JAVA_TOOL_OPTIONS`, agent configuration, supported libraries
- [Java Agent Configuration Reference](https://opentelemetry.io/docs/zero-code/java/agent/configuration/) — all `otel.` **system properties and `OTEL_`** env vars
- [OpenTelemetry Java Manual Instrumentation](https://opentelemetry.io/docs/languages/java/instrumentation/) — `@WithSpan`, programmatic spans, `GlobalOpenTelemetry`
- [OpenTelemetry Java versioning and BOM](https://github.com/open-telemetry/opentelemetry-java/blob/main/VERSIONING.md) — aligning SDK and API versions

### Python
- [OpenTelemetry Python Auto-Instrumentation](https://opentelemetry.io/docs/zero-code/python/) — `opentelemetry-instrument` CLI, `opentelemetry-bootstrap`
- [OpenTelemetry Python Manual Instrumentation](https://opentelemetry.io/docs/languages/python/instrumentation/) — `TracerProvider`, `BatchSpanProcessor`, context injection
- [OpenTelemetry Python SDK API Reference](https://opentelemetry-python.readthedocs.io/en/latest/)
- [Gunicorn server hooks](https://docs.gunicorn.org/en/stable/settings.html#server-hooks) — `post_fork` hook for fork-safe initialization

### Go
- [OpenTelemetry Go Instrumentation](https://opentelemetry.io/docs/languages/go/instrumentation/) — `TracerProvider` setup, `context.Context` propagation, contrib packages
- [OpenTelemetry Go — Using Instrumentation Libraries](https://opentelemetry.io/docs/languages/go/libraries/) — `otelhttp`, `otelgrpc`, `otelsql`
- [go.opentelemetry.io/otel — Go package reference](https://pkg.go.dev/go.opentelemetry.io/otel)
- [otelhttp — HTTP client and server instrumentation](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp)