# APM Python Instrumentation

Kloudfuse APM supports Python applications via the OpenTelemetry Python SDK. This guide covers both automatic instrumentation (zero-code, via the `opentelemetry-instrument` CLI wrapper) and manual SDK initialization for full control over span creation and export.

## Prerequisites

- Python 3.7 or later
- `pip` available in your environment
- Your Kloudfuse API key — see [Ingestion Authentication with API Key](https://docs.kloudfuse.com/platform/4.1.0/administration/authentication/ingestion-api-key/)

## OTLP Endpoints

There are three ways to route spans from a Python application to Kloudfuse, depending on how your cluster is deployed. Choose one and use it consistently across your `OTEL_*` environment variables.

- kf-agent gRPC (port 4317)
- kf-agent HTTP (port 4318)
- Direct Ingester (HTTPS, port 443)

The standard deployment runs `kf-agent` as a DaemonSet or sidecar in the cluster. Applications send spans to it over gRPC on port 4317; the agent batches and forwards them to the Kloudfuse backend.

```bash
OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>
```

Use the gRPC exporter package:

```bash
pip install opentelemetry-exporter-otlp-proto-grpc
```

```python
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Endpoint and headers are read from OTEL_EXPORTER_OTLP_ENDPOINT
# and OTEL_EXPORTER_OTLP_HEADERS automatically.
exporter = OTLPSpanExporter()
```

If gRPC is blocked by a firewall or network policy, `kf-agent` also accepts OTLP/HTTP on port 4318. The SDK appends `/v1/traces` to the base endpoint automatically.

```bash
OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>
```

Use the HTTP exporter package:

```bash
pip install opentelemetry-exporter-otlp-proto-http
```

```python
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter()
```

When `kf-agent` is not deployed in the cluster, send spans directly to the Kloudfuse ingester through the nginx ingress over HTTPS. This is the path to use in environments where only port 443 is open.

```bash
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>
```

|     |     |
| --- | --- |
|   | Use `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` (the signal-specific variable), not<br>`OTEL_EXPORTER_OTLP_ENDPOINT` (the base variable). The base variable causes the SDK to append<br>`/v1/traces` to whatever path you provide, producing the wrong URL: |

| Variable used | Resulting URL sent to the ingester |
| --- | --- |
| `OTEL_EXPORTER_OTLP_ENDPOINT=https://host/ingester/otlp/traces` | `https://host/ingester/otlp/traces/v1/traces` ❌ (double suffix) |
| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://host/ingester/otlp/traces` | `https://host/ingester/otlp/traces` ✓ (used verbatim) |

In Kubernetes, inject the API key from a Secret:

```yaml
env:
  - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
    value: "https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value
```

Create the Secret with the full header expression (not just the raw token):

```bash
kubectl create secret generic kloudfuse-api-key \
  --from-literal=value="kf-api-key=<your-api-key>"
```

## Authentication

Pass the Kloudfuse API key as the `kf-api-key` request header on the OTLP exporter.

- Environment Variable (Recommended)
- Programmatic (headers parameter)

Both the gRPC and HTTP exporters read `OTEL_EXPORTER_OTLP_HEADERS` automatically — no code changes are needed:

```bash
export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>"
```

The variable accepts a comma-separated list of `name=value` pairs:

```bash
export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>,x-env=production"
```

This also works with the `opentelemetry-instrument` CLI — just set the variable before running it:

```bash
OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>" \
opentelemetry-instrument --traces_exporter otlp python app.py
```

In Kubernetes, pull the value from a Secret so it never appears in source:

```yaml
env:
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value
```

Create the Secret with the full header expression:

```bash
kubectl create secret generic kloudfuse-api-key \
  --from-literal=value="kf-api-key=<your-api-key>"
```

If you cannot use environment variables, pass the header directly to `OTLPSpanExporter`. Read the key from the environment rather than hardcoding it:

```python
import os
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor

exporter = OTLPSpanExporter(
    headers={"kf-api-key": os.environ["KLOUDFUSE_API_KEY"]},
)
provider.add_span_processor(BatchSpanProcessor(exporter))
```

For the gRPC exporter, the parameter is identical — replace the import:

```python
import os
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    headers={"kf-api-key": os.environ["KLOUDFUSE_API_KEY"]},
)
```

## Fork Safety (Gunicorn / uWSGI)

If you run your application under Gunicorn or uWSGI, the SDK must be initialized **after** the server forks worker processes. Initializing at module import time (before the fork) causes the `BatchSpanProcessor` background thread and exporter connection to be copied into each worker in a broken state.

See [Python Fork Safety](https://docs.kloudfuse.com/platform/4.1.0/data-collection/apm/instrument-common-issues/#python-fork-safety) for the `post_fork` hook pattern and full explanation.

## Automatic Instrumentation

For web frameworks (Flask, Django, FastAPI), the `opentelemetry-instrument` CLI wrapper monkey-patches the framework at import time, automatically creating `SpanKind = SERVER` spans for every HTTP request without any code changes.

### Flask

```bash
pip install \
  opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-http \
  opentelemetry-instrumentation-flask
```

```bash
OTEL_SERVICE_NAME=my-flask-service \
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>" \
opentelemetry-instrument \
  --traces_exporter otlp \
  --metrics_exporter none \
  --logs_exporter none \
  python app.py
```

The Flask instrumentation sets `SpanKind = SERVER` and populates standard HTTP attributes (`http.method`, `http.target`, `http.status_code`, `http.scheme`, `net.host.name`) for every request. No changes to `app.py` are required.

### Django

```bash
pip install \
  opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-http \
  opentelemetry-instrumentation-django
```

```bash
OTEL_SERVICE_NAME=my-django-service \
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>" \
opentelemetry-instrument python manage.py runserver
```

### Setting Resource Attributes

Pass resource attributes via `OTEL_RESOURCE_ATTRIBUTES` — they are merged into every span exported by `opentelemetry-instrument`:

```bash
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production,service.namespace=my-team
```

## Manual Instrumentation

### Step 1: Install the SDK

Install the core SDK and the HTTP exporter. The HTTP exporter works for Options 2 and 3 above and is recommended for Kubernetes deployments where only port 443 is guaranteed to be open.

```bash
pip install \
  opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-http
```

For auto-instrumentation of web frameworks (Flask, Django, FastAPI), also install:

```bash
pip install opentelemetry-instrumentation-flask   # or -django, -fastapi, etc.
```

To auto-detect and install instrumentation for all libraries already installed in your environment:

```bash
pip install opentelemetry-distro
opentelemetry-bootstrap --action=install
```

### Step 2: Configure the SDK

Initialize the `TracerProvider` with a `BatchSpanProcessor` and an `OTLPSpanExporter`. All connection settings — endpoint, headers, protocol — are read from `OTEL_*` environment variables automatically. Only the resource attributes (service name, environment, namespace) need to be set in code.

```python
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

# Resource attributes identify this service in Kloudfuse APM.
resource = Resource.create({
    "service.name": "my-python-service",
    "deployment.environment.name": "production",
    "service.namespace": "my-team",
})

# OTLPSpanExporter() with no arguments reads OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (or
# OTEL_EXPORTER_OTLP_ENDPOINT), OTEL_EXPORTER_OTLP_HEADERS, and
# OTEL_EXPORTER_OTLP_PROTOCOL from the environment automatically.
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
```

Call this initialization once at application startup, before creating any tracers or spans.

### Step 3: Create Spans

Use `start_as_current_span` as a context manager to create spans. Always set `kind` explicitly on the entry-point span of your service — `SpanKind.SERVER` for inbound request handlers, `SpanKind.CLIENT` for outbound calls. Kloudfuse derives service-level throughput and latency metrics from `SERVER` spans.

```python
from opentelemetry import trace
from opentelemetry.trace import SpanKind

tracer = trace.get_tracer("my-python-service")

# Root SERVER span — represents an inbound request handled by this service.
# Required for the service to appear in the Service Map with request metrics.
with tracer.start_as_current_span("handle-request", kind=SpanKind.SERVER) as server_span:
    server_span.set_attribute("http.method", "GET")
    server_span.set_attribute("http.route", "/orders")

# Child CLIENT span — outbound call (DB, downstream service, cache, etc.).
    # Inherits the parent trace ID automatically via context manager nesting.
    with tracer.start_as_current_span("db-query", kind=SpanKind.CLIENT) as client_span:
        client_span.set_attribute("db.system", "postgresql")
        client_span.set_attribute("db.name", "orders")
        # ... execute the query ...
```

To record an exception and mark a span as failed:

```python
from opentelemetry.trace import StatusCode

with tracer.start_as_current_span("process-order", kind=SpanKind.SERVER) as span:
    try:
        result = process(order_id)
    except Exception as e:
        span.record_exception(e)
        span.set_status(StatusCode.ERROR, str(e))
        raise
```

|     |     |
| --- | --- |
|   | Call both `record_exception()` and `set_status(StatusCode.ERROR, …​)` together. They do not<br>imply each other — omitting `set_status` leaves the span marked OK and it will not appear in error<br>rate calculations. |

## Kubernetes Integration

The following pod spec shows a complete setup for direct ingester export (Option 3). The application code is written inline and run directly with `python` — no `opentelemetry-instrument` wrapper is needed when the SDK is initialized in code.

```yaml
env:
  - name: OTEL_SERVICE_NAME
    value: "my-python-service"
  - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
    value: "https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "deployment.environment.name=production,service.namespace=my-team"
```

Create the Secret with the full header expression:

```bash
kubectl create secret generic kloudfuse-api-key \
  --from-literal=value="kf-api-key=<your-api-key>"
```

For kf-agent deployments (Option 1), replace the endpoint env vars with:

```yaml
env:
  - name: OTEL_SERVICE_NAME
    value: "my-python-service"
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://kf-agent:4317"
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value
```

## Verify Traces

1. Open **Kloudfuse UI → APM → Trace Explorer**
2. Filter by `service.name = my-python-service`
3. Click a trace to inspect the span tree, attributes, and timing breakdown

## References

### Sample Manifest

An example Kubernetes manifest and instructions can be found at [APM Demo - Python](https://github.com/kloudfuse-ext/customer/tree/main/agents/apm/demo/python)

### Specification and Concepts

- [OpenTelemetry Concepts](https://opentelemetry.io/docs/concepts/) — traces, spans, context propagation, sampling
- [OpenTelemetry Trace SDK specification](https://opentelemetry.io/docs/specs/otel/trace/sdk/) — BatchSpanProcessor, sampler interface, resource
- [OTLP Exporter Configuration](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/) — `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, headers, protocol, compression

### Python SDK and Packages

- [OpenTelemetry Python Auto-Instrumentation](https://opentelemetry.io/docs/zero-code/python/) — `opentelemetry-instrument` CLI, `opentelemetry-bootstrap`, supported frameworks
- [OpenTelemetry Python Manual Instrumentation](https://opentelemetry.io/docs/languages/python/instrumentation/) — `TracerProvider`, `BatchSpanProcessor`, context injection, `SpanKind`
- [OpenTelemetry Python SDK API Reference](https://opentelemetry-python.readthedocs.io/en/latest/) — full API docs for `opentelemetry-sdk`, exporters, and instrumentation packages

### Framework and Server Hooks

- [Gunicorn server hooks](https://docs.gunicorn.org/en/stable/settings.html#server-hooks) — `post_fork` hook for fork-safe SDK initialization
- [uWSGI Python decorators](https://uwsgi-docs.readthedocs.io/en/latest/PythonDecorators.html) — `@postfork` decorator for fork-safe initialization