OpenTelemetry Observability v0.0.16

Klio provides built-in support for OpenTelemetry, enabling comprehensive observability through distributed tracing and metrics collection. This allows you to monitor backup operations, performance characteristics, and system health across your Klio deployment.

Available Telemetry

Klio automatically collects the following:

• Traces
  • Distributed WAL streaming and processing
  • Backup lifecycle (backup, backup run, verification, maintenance)
• Metrics
  • Server
    • Server uptime
    • Backup metrics
      • Number of snapshots
      • Number of files in the latest snapshot
      • Number of directories in the latest snapshot
      • Size of the latest snapshot
      • Age of the latest snapshot
      • Age of the oldest snapshot
      • Total number of backup verifications (split by outcome and tier)
    • WAL processing metrics
      • Number of WAL files written
      • Bytes written
      • Timestamp of the most recently written WAL file
      • LSN progress of WAL ingestion (Tier 1) and archival (Tier 2)
      • Timeline of the latest WAL on Tier 1 and Tier 2
    • Queue metrics
      • Number of messages in the queue
      • Number of bytes in the queue
    • GRPC metrics
    • Go runtime statistics
    • Host metrics
    • Controller runtime metrics
  • Sidecar
    • Backup metrics
      • Number of backups currently in progress
      • Timestamp of the most recent backup start
      • Timestamp of the most recent successful completion
      • Timestamp of the most recent failure
      • Duration of the most recent backup
      • Total number of backup runs (split by outcome)
      • Total number of backup verifications (split by outcome)
    • GRPC metrics
    • Go runtime statistics
    • Host metrics
    • Controller runtime metrics

Traces Reference

Backup lifecycle spans

When a backup is triggered through CNPG-I, Klio creates the following spans under the klio.plugin.backup tracer:

The backup span includes the following attributes:

On failure, the span records the error and sets its status to ERROR.

Metrics Reference

Klio metric names follow the klio.<component>.<domain>.<measurement> taxonomy. The component segment identifies which process emits the metric:

• plugin — the CNPG plugin sidecar running in each PostgreSQL pod.
• server — the Klio server StatefulSet (hosts the Kopia server, the WAL gRPC ingest, the embedded NATS JetStream queue, and the tier-2 WAL consumer).
• operator — the Klio operator deployment. Bridges controller-runtime Prometheus metrics to OTLP and adds Go runtime and host instrumentation.
• client — reserved for future instrumentation; no metrics are emitted today.

Attributes

Klio metrics carry the following attributes. Each per-metric table below repeats the applicable attributes in its descriptions; this section is the central reference for the value space of each attribute key.

Backup lifecycle metrics (plugin sidecar)

These metrics are emitted by the plugin sidecar and track backup operations on each PostgreSQL instance:

WAL ingest metrics (server)

The WAL ingest series is unified across tiers: WAL bytes and files written to local disk by the WAL gRPC server (tier 1) and uploaded to remote storage by the consumer (tier 2) share a single instrument family and are distinguished by the tier attribute ("tier1" or "tier2").

Every recording carries a cluster_name attribute identifying the PostgreSQL cluster, alongside the tier discriminator.

Backup verification metrics (server)

The server verifies each backup that the plugin records via klio.plugin.backup.*. Verification happens at two points: once against the tier-1 local copy immediately after the backup completes, and again against the tier-2 remote copy after migration. The tier attribute ("tier1" or "tier2") identifies which check the recording refers to.

Alerting on stalled WAL processing

The same klio.server.wal.latest_written_time instrument is emitted from two stages of the WAL pipeline, distinguished by the tier attribute. A stale value signals a different failure depending on the tier:

• tier="tier1" reflects when the Klio server last received a WAL file from PostgreSQL streaming replication and persisted it to local disk. A stale value means PostgreSQL is no longer shipping WALs to Klio, which may indicate a replication problem, or that writing on disk is failing.

• tier="tier2" reflects when the consumer last uploaded a WAL file to tier-2 object storage. A stale value means the remote backend is no longer receiving WALs, even though PostgreSQL replication may still be working, or that uploading the WAL to the object store is failing.

Server metrics

The klio.server.wal.latest_written_lsn instrument provides a complementary view of the same two pipeline stages, expressed as a byte offset rather than a wall-clock timestamp:

• tier="tier1" is updated on every flushed WAL block received by the WAL server (tracks pg_current_wal_flush_lsn() semantics).

• tier="tier2" is updated once per completed WAL file by the consumer. Its value is the LSN of the last byte of the WAL segment just archived.

The companion klio.server.wal.latest_written_timeline gauge exposes the timeline ID of the WAL file each tier is currently handling.

Use these gauges alongside the timestamp gauges to distinguish a slow pipeline (timestamps advancing, LSN gap growing) from a stalled one (timestamps and LSN both frozen).

Base backup metrics (server)

These metrics are emitted by the Klio server base backup component and track Kopia snapshot statistics:

Every recording carries a tier attribute (tier1 for the local disk repository, tier2 for the remote object store) and a snapshot_source attribute identifying the source descriptor (userName@hostName:path) the snapshot belongs to.

Queue metrics (server)

These metrics are emitted by the Klio server and track the state of the embedded NATS JetStream streams used for asynchronous Tier 2 offloading of WAL files and backups. Each sample carries a stream attribute identifying the source stream — typically klio-wal-stream (WAL work queue), klio-backup-stream (backup work queue), and klio-latest-uploaded-wal-per-cluster-stream (retention safeguard, capped to one message per cluster):

Migration from the previous metric names

Klio is in alpha and previously emitted metrics under a flat namespace. The component-based taxonomy above replaces those names in a single hard rename — there is no dual emission. Update dashboards and alerts according to the following table:

Configuration

Klio automatically detects OpenTelemetry configuration through standard environment variables. If no OpenTelemetry environment variables are present, Klio will use no-op providers that don't collect any telemetry data.

Traces and metrics exporters can be configured independently through the autoexport package.

General Settings

The following environment variables are used to configure OpenTelemetry:

• OTEL_SERVICE_NAME: (required) Name of the service, e.g., klio-server
• OTEL_RESOURCE_ATTRIBUTES: Comma-separated list of resource attributes (e.g., deployment.environment=production,service.namespace=klio-system)
• OTEL_RESOURCE_DETECTORS: Comma-separated list of resource detectors from the autodetect package, used to automatically populate resource attributes

Traces exporter

To enable the traces exporter, set the OTEL_TRACES_EXPORTER environment variable to one of the supported exporters:

• otlp: OpenTelemetry Protocol (OTLP) exporter
• console: Console exporter (useful for debugging)
• none: No-op exporter (disables tracing)

You can define the OTLP protocol using the OTEL_EXPORTER_OTLP_TRACES_PROTOCOL variable, or the general OTEL_EXPORTER_OTLP_PROTOCOL. Supported protocols include:

• http/protobuf (default)
• grpc

Additional configuration options for trace exporters can be found in the documentation of the respective exporters:

• OTLP Trace gRPC Exporter
• OTLP Trace HTTP Exporter

Metrics Exporter

To enable the metrics exporter, set the OTEL_METRICS_EXPORTER environment variable to one of the supported exporters:

• otlp: OpenTelemetry Protocol (OTLP) exporter
• prometheus: Prometheus exporter + HTTP server
• console: Console exporter (useful for debugging)
• none: No-op exporter (disables metrics)

You can define the OTLP protocol using the OTEL_EXPORTER_OTLP_METRICS_PROTOCOL variable, or the general OTEL_EXPORTER_OTLP_PROTOCOL. Supported protocols include:

• http/protobuf (default)
• grpc

Additional configuration options for metrics exporters can be found in the documentation of the respective exporters:

• OTLP Metric gRPC Exporter
• OTLP Metric HTTP Exporter

For the Prometheus exporter, you can configure the host and port of the HTTP server using the following environment variables:

• OTEL_EXPORTER_PROMETHEUS_HOST (default: localhost)
• OTEL_EXPORTER_PROMETHEUS_PORT (default: 9464)

Exporters and receivers

The OTLP exporter pushes telemetry to any OTLP-compatible receiver. Common options include:

• An OpenTelemetry Collector, which can receive OTLP data and fan it out to multiple backends (Prometheus, Jaeger, Grafana, etc.). In Kubernetes, the OpenTelemetry Operator manages collectors via the OpenTelemetryCollector CRD and can expose a stable in-cluster OTLP endpoint for Klio to target.
• Any backend with native OTLP support.

The Prometheus exporter starts a local HTTP server that Prometheus scrapes directly, with no intermediate collector required.

Configuring Klio with OpenTelemetry in Kubernetes

When running in a Kubernetes environment, Klio will automatically define CONTAINER_NAME, POD_NAME and NAMESPACE_NAME environment variables. When any of these environment variables are set, Klio will automatically add the corresponding resource attributes (k8s.container.name, k8s.pod.name, k8s.namespace.name) to all telemetry data. Each attribute is added independently - you don't need all three environment variables to be present.

Klio server with OpenTelemetry

When deploying a Klio Server, you can configure OpenTelemetry by specifying the necessary settings in the template section of the Server spec:

1. Set the required environment variables for OpenTelemetry configuration in the server container.
2. Mount any necessary TLS certificates for secure communication with the OpenTelemetry Collector.

For simpler management, use a ConfigMap to store the OpenTelemetry configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: klio-otel-config
data:
  OTEL_SERVICE_NAME: "klio-server"
  OTEL_RESOURCE_DETECTORS: "telemetry.sdk,host,os.type,process.executable.name"
  OTEL_TRACES_EXPORTER: "otlp"
  OTEL_EXPORTER_OTLP_TRACES_PROTOCOL: "grpc"
  OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: "https://otel-collector:4317"
  OTEL_EXPORTER_OTLP_TRACES_COMPRESSION: "gzip"
  OTEL_EXPORTER_OTLP_TRACES_TIMEOUT: "10000"
  OTEL_EXPORTER_OTLP_TRACES_INSECURE: "false"
  OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE: "/otel/ca.crt"
  OTEL_EXPORTER_OTLP_TRACES_CLIENT_CERTIFICATE: "/otel/tls.crt"
  OTEL_EXPORTER_OTLP_TRACES_CLIENT_KEY: "/otel/tls.key"
  OTEL_METRICS_EXPORTER: "otlp"
  OTEL_METRIC_EXPORT_INTERVAL: "60000"
  OTEL_EXPORTER_OTLP_METRICS_PROTOCOL: "grpc"
  OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: "https://otel-collector:4317"
  OTEL_EXPORTER_OTLP_METRICS_TIMEOUT: "60000"
  OTEL_EXPORTER_OTLP_METRICS_INSECURE: "false"
  OTEL_EXPORTER_OTLP_METRICS_CERTIFICATE: "/otel/ca.crt"
  OTEL_EXPORTER_OTLP_METRICS_CLIENT_CERTIFICATE: "/otel/tls.crt"
  OTEL_EXPORTER_OTLP_METRICS_CLIENT_KEY: "/otel/tls.key"
---
apiVersion: klio.enterprisedb.io/v1alpha1
kind: Server
metadata:
  name: my-klio-server
spec:
  # ... other configuration ...
  template:
    spec:
      containers:
        - name: server
          envFrom:
            - configMapRef:
                name: klio-otel-config
          volumeMounts:
            - mountPath: /otel
              name: otel
      volumes:
        - name: otel
          projected:
            sources:
              - secret:
                  name: otel-collector-tls
                  items:
                    - key: ca.crt
                      path: ca.crt
              - secret:
                  name: otel-client-cert
                  items:
                    - key: tls.crt
                      path: tls.crt
                    - key: tls.key
                      path: tls.key

Klio plugins with OpenTelemetry

When deploying Klio as a CNPG Cluster plugin, configure OpenTelemetry by specifying the necessary environment variables in the containers section of the PluginConfiguration spec. The available container names are:

• klio-plugin: Main plugin sidecar for backup management
• klio-restore: Restore operations sidecar

Create a ConfigMap for the shared OpenTelemetry configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-klio-otel-config
data:
  OTEL_RESOURCE_DETECTORS: "telemetry.sdk,host,os.type,process.executable.name"
  OTEL_TRACES_EXPORTER: "otlp"
  OTEL_METRICS_EXPORTER: "otlp"
  OTEL_EXPORTER_OTLP_PROTOCOL: "grpc"
  OTEL_EXPORTER_OTLP_ENDPOINT: "https://otel-collector:4317"
  OTEL_EXPORTER_OTLP_COMPRESSION: "gzip"
  OTEL_EXPORTER_OTLP_TIMEOUT: "10000"
  OTEL_EXPORTER_OTLP_INSECURE: "false"
  OTEL_EXPORTER_OTLP_CERTIFICATE: "/projected/ca.crt"
  OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE: "/projected/tls.crt"
  OTEL_EXPORTER_OTLP_CLIENT_KEY: "/projected/tls.key"

Configure the PluginConfiguration to inject the environment variables into each sidecar container:

apiVersion: klio.enterprisedb.io/v1alpha1
kind: PluginConfiguration
metadata:
  name: client-config-cluster-example
spec:
  serverAddress: klio.default
  clientSecretName: cluster-example-klio-user
  serverSecretName: klio-server-tls
  clusterName: cluster-example
  containers:
    - name: klio-plugin
      env:
        - name: OTEL_SERVICE_NAME
          value: "klio-plugin"
      envFrom:
        - configMapRef:
            name: cluster-klio-otel-config
    - name: klio-restore
      env:
        - name: OTEL_SERVICE_NAME
          value: "klio-restore"
      envFrom:
        - configMapRef:
            name: cluster-klio-otel-config

Mount the OpenTelemetry certificates using the Cluster's projectedVolumeTemplate. The projected volume is mounted at /projected/ and is accessible to all sidecar containers:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cluster-example
spec:
  instances: 3

  projectedVolumeTemplate:
    sources:
      - secret:
          name: otel-collector-tls
          items:
            - key: ca.crt
              path: ca.crt
      - secret:
          name: otel-client-cert
          items:
            - key: tls.crt
              path: tls.crt
            - key: tls.key
              path: tls.key

  plugins:
    - name: klio.enterprisedb.io
      enabled: true
      parameters:
        pluginConfigurationRef: client-config-cluster-example

  storage:
    size: 10Gi

Klio operator with OpenTelemetry

The operator bridges the controller-runtime Prometheus metrics registry to OTLP and adds Go runtime and host instrumentation. When no OTEL_* environment variables are present, a no-op meter provider is installed and the operator runs without telemetry overhead.

The existing Prometheus /metrics endpoint remains available for pull-based scraping regardless of whether OTLP export is enabled.

To enable OTLP export, set OTEL_* variables through the Helm chart's controllerManager.manager.env value:

controllerManager:
  manager:
    env:
      OTEL_SERVICE_NAME: "klio-operator"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4318"
      OTEL_EXPORTER_OTLP_PROTOCOL: "http/protobuf"  # or "grpc"

The Helm chart automatically injects POD_NAME, NAMESPACE_NAME, and CONTAINER_NAME via the Kubernetes downward API, so the corresponding k8s.* resource attributes are populated without additional configuration.