APM Python Instrumentation :: Kloudfuse Docs

APM Python Instrumentation

Kloudfuse APM supports Python applications via the OpenTelemetry Python SDK. This guide covers both automatic instrumentation (zero-code, via the opentelemetry-instrument CLI wrapper) and manual SDK initialization for full control over span creation and export.

Prerequisites

OTLP Endpoints

There are three ways to route spans from a Python application to Kloudfuse, depending on how your cluster is deployed. Choose one and use it consistently across your OTEL_* environment variables.

The standard deployment runs kf-agent as a DaemonSet or sidecar in the cluster. Applications send spans to it over gRPC on port 4317; the agent batches and forwards them to the Kloudfuse backend.

OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>

Use the gRPC exporter package:

pip install opentelemetry-exporter-otlp-proto-grpc

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Endpoint and headers are read from OTEL_EXPORTER_OTLP_ENDPOINT
# and OTEL_EXPORTER_OTLP_HEADERS automatically.
exporter = OTLPSpanExporter()

If gRPC is blocked by a firewall or network policy, kf-agent also accepts OTLP/HTTP on port 4318. The SDK appends /v1/traces to the base endpoint automatically.

OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>

Use the HTTP exporter package:

pip install opentelemetry-exporter-otlp-proto-http

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter()

When kf-agent is not deployed in the cluster, send spans directly to the Kloudfuse ingester through the nginx ingress over HTTPS. This is the path to use in environments where only port 443 is open.

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>
Variable used Resulting URL sent to the ingester
OTEL_EXPORTER_OTLP_ENDPOINT=https://host/ingester/otlp/traces https://host/ingester/otlp/traces/v1/traces ❌
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://host/ingester/otlp/traces https://host/ingester/otlp/traces ✓

In Kubernetes, inject the API key from a Secret:

env:
  - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
    value: "https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value

Create the Secret with the full header expression:

kubectl create secret generic kloudfuse-api-key \
  --from-literal=value="kf-api-key=<your-api-key>"

Authentication

Pass the Kloudfuse API key as the kf-api-key request header on the OTLP exporter.

Both the gRPC and HTTP exporters read OTEL_EXPORTER_OTLP_HEADERS automatically — no code changes are needed:

export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>"

The variable accepts a comma-separated list of name=value pairs:

export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>,x-env=production"

In Kubernetes, pull the value from a Secret so it never appears in source:

env:
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value

If you cannot use environment variables, pass the header directly to OTLPSpanExporter. Read the key from the environment rather than hardcoding it:

import os
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor

exporter = OTLPSpanExporter(
    headers={"kf-api-key": os.environ["KLOUDFUSE_API_KEY"]},
)
provider.add_span_processor(BatchSpanProcessor(exporter))

Fork Safety (Gunicorn / uWSGI)

If you run your application under Gunicorn or uWSGI, the SDK must be initialized after the server forks worker processes. Initializing at module import time (before the fork) causes the BatchSpanProcessor background thread and exporter connection to be copied into each worker in a broken state.

Automatic Instrumentation

For web frameworks (Flask, Django, FastAPI), the opentelemetry-instrument CLI wrapper monkey-patches the framework at import time, automatically creating SpanKind = SERVER spans for every HTTP request without any code changes.

Flask

pip install \
  opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-http \
  opentelemetry-instrumentation-flask

OTEL_SERVICE_NAME=my-flask-service \
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>" \
opentelemetry-instrument \
  --traces_exporter otlp \
  --metrics_exporter none \
  --logs_exporter none \
  python app.py

Django

pip install \
  opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-http \
  opentelemetry-instrumentation-django

OTEL_SERVICE_NAME=my-django-service \
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>" \
opentelemetry-instrument python manage.py runserver

Setting Resource Attributes

Pass resource attributes via OTEL_RESOURCE_ATTRIBUTES — they are merged into every span exported by opentelemetry-instrument:

OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production,service.namespace=my-team

Manual Instrumentation

Step 1: Install the SDK

Install the core SDK and the HTTP exporter. The HTTP exporter works for Options 2 and 3 above and is recommended for Kubernetes deployments where only port 443 is guaranteed to be open.

pip install \
  opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-http

For auto-instrumentation of web frameworks (Flask, Django, FastAPI), also install:

pip install opentelemetry-instrumentation-flask   # or -django, -fastapi, etc.

Step 2: Configure the SDK

Initialize the TracerProvider with a BatchSpanProcessor and an OTLPSpanExporter. All connection settings — endpoint, headers, protocol — are read from OTEL_* environment variables automatically. Only the resource attributes (service name, environment, namespace) need to be set in code.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

# Resource attributes identify this service in Kloudfuse APM.
resource = Resource.create({
    "service.name": "my-python-service",
    "deployment.environment.name": "production",
    "service.namespace": "my-team",
})

# OTLPSpanExporter() with no arguments reads OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (or
# OTEL_EXPORTER_OTLP_ENDPOINT), OTEL_EXPORTER_OTLP_HEADERS, and
# OTEL_EXPORTER_OTLP_PROTOCOL from the environment automatically.
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

Step 3: Create Spans

Use start_as_current_span as a context manager to create spans. Always set kind explicitly on the entry-point span of your service — SpanKind.SERVER for inbound request handlers, SpanKind.CLIENT for outbound calls. Kloudfuse derives service-level throughput and latency metrics from SERVER spans.

from opentelemetry import trace
from opentelemetry.trace import SpanKind

tracer = trace.get_tracer("my-python-service")

# Root SERVER span — represents an inbound request handled by this service.
# Required for the service to appear in the Service Map with request metrics.
with tracer.start_as_current_span("handle-request", kind=SpanKind.SERVER) as server_span:
    server_span.set_attribute("http.method", "GET")
    server_span.set_attribute("http.route", "/orders")

# Child CLIENT span — outbound call (DB, downstream service, cache, etc.).
    # Inherits the parent trace ID automatically via context manager nesting.
    with tracer.start_as_current_span("db-query", kind=SpanKind.CLIENT) as client_span:
        client_span.set_attribute("db.system", "postgresql")
        client_span.set_attribute("db.name", "orders")
        # ... execute the query ...

To record an exception and mark a span as failed:

from opentelemetry.trace import StatusCode

with tracer.start_as_current_span("process-order", kind=SpanKind.SERVER) as span:
    try:
        result = process(order_id)
    except Exception as e:
        span.record_exception(e)
        span.set_status(StatusCode.ERROR, str(e))
        raise

Kubernetes Integration

The following pod spec shows a complete setup for direct ingester export (Option 3). The application code is written inline and run directly with python — no opentelemetry-instrument wrapper is needed when the SDK is initialized in code.

env:
  - name: OTEL_SERVICE_NAME
    value: "my-python-service"
  - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
    value: "https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "deployment.environment.name=production,service.namespace=my-team"

For kf-agent deployments (Option 1), replace the endpoint env vars with:

env:
  - name: OTEL_SERVICE_NAME
    value: "my-python-service"
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://kf-agent:4317"
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value

Verify Traces

  1. Open Kloudfuse UI → APM → Trace Explorer
  2. Filter by service.name = my-python-service
  3. Click a trace to inspect the span tree, attributes, and timing breakdown

References

Sample Manifest

An example Kubernetes manifest and instructions can be found at APM Demo - Python

Specification and Concepts

Python SDK and Packages

Framework and Server Hooks