IOS-XE Telemetry

Beta

IOSXETelemetry is Beta (v1alpha1). The MDT pipeline architecture and gNMI subscription mechanics are stable; the subscription schema and OpenTelemetry output shapes may change between releases. Evaluate in non-production environments before broader rollout.

The IOSXETelemetry CRD declares MDT-over-gNMI subscriptions for one CiscoDevice and converts them into OpenTelemetry signals.

---

Deploying MDT with Cisco Virtual Kubelet

Prerequisites

1. The target device must have gNMI enabled. For IOS-XE:

   gnxi
    server
    secure-allow-self-signed-trustpoint
   !
   

2. CVK must be configured with an OTLP endpoint — set on the CiscoDevice CR or as an environment variable on the VK pod:

   spec:
     otelConfig:
       endpoint: otelcol.observability:4317
       insecure: true
       interval: 60s
   

   Or via Helm values:

   otel:
     endpoint: "otelcol.observability:4317"
     insecure: true
   

3. The IOSXETelemetry CRD must be installed. Verify with:

   kubectl get crd iosxetelemetries.config.cisco.vk
   

Step 1 — Create an IOSXETelemetry CR

Each IOSXETelemetry CR attaches one or more gNMI subscriptions to a single device. CVK handles stream multiplexing, reconnect, and notification routing.

Interface counters via OpenConfig:

apiVersion: config.cisco.vk/v1alpha1
kind: IOSXETelemetry
metadata:
  name: cat9k-interfaces
  namespace: default
spec:
  deviceRef:
    name: cat9k-smoke
  subscriptions:
    - paths:
        - /interfaces/interface/state/counters
      sampleInterval: 30000000000

IOS-XE native YANG (app-hosting oper-data):

For IOS-XE native paths, set preservePathPrefix: true — IOS-XE rejects module names in Path.Origin and expects them inline on the first path element instead:

apiVersion: config.cisco.vk/v1alpha1
kind: IOSXETelemetry
metadata:
  name: cat9k-apps
  namespace: default
spec:
  deviceRef:
    name: cat9k-smoke
  subscriptions:
    - paths:
        - /Cisco-IOS-XE-app-hosting-oper:app-hosting-oper-data/apps
      sampleInterval: 10000000000
      preservePathPrefix: true

On-change subscription for link state:

spec:
  subscriptions:
    - paths:
        - /interfaces/interface/state/oper-status
      streamMode: ON_CHANGE

Step 2 — Watch subscription health

$ kubectl get iosxetelemetry -w
NAME              DEVICE        PHASE    AGE
cat9k-interfaces  cat9k-smoke   Active   8s
cat9k-apps        cat9k-smoke   Active   3s

Check detailed status:

$ kubectl describe iosxetelemetry cat9k-interfaces
...
Status:
  Phase:  Active
  Subscription Stats:
    Active Subscriptions:   1
    Notifications Total:    142
    Buffer Overflow Total:  0
Events:
  Normal  Subscribed  8s  gNMI Subscribe stream established

If a stream drops, CVK reconnects with exponential backoff and emits a Warning StreamError event. The phase transitions to Degraded until the stream recovers.

Step 3 — Verify data in your collector

CVK emits each MDT notification as one of:

• OTel log records — for string/state-change leaves (e.g. oper-status transitions)
• OTel metrics — for numeric leaves (e.g. counter, gauge readings)

All records carry cisco.device.name and cisco.subscription.path as resource attributes.

Example Prometheus remote-write output from interface counters:

interfaces_interface_state_counters_in_octets{
  cisco_device_name="cat9k-smoke",
  interface_name="GigabitEthernet1/0/1"
} 1234567890
interfaces_interface_state_counters_out_octets{
  cisco_device_name="cat9k-smoke",
  interface_name="GigabitEthernet1/0/1"
} 987654321

Path quick reference

What to monitor	YANG path	Style
Interface counters	/interfaces/interface/state/counters	OpenConfig
Interface oper-status	/interfaces/interface/state/oper-status	OpenConfig
BGP neighbors	/network-instances/network-instance/protocols/protocol/bgp/neighbors/neighbor/state	OpenConfig
OSPF neighbors	/network-instances/network-instance/protocols/protocol/ospf/areas/area/interfaces/interface/neighbors/neighbor/state	OpenConfig
CPU utilization	/Cisco-IOS-XE-process-cpu-oper:cpu-usage/cpu-utilization	IOS-XE native (preservePathPrefix)
Memory	/Cisco-IOS-XE-memory-oper:memory-statistics/memory-statistic	IOS-XE native (preservePathPrefix)
App-hosting apps	/Cisco-IOS-XE-app-hosting-oper:app-hosting-oper-data/apps	IOS-XE native (preservePathPrefix)
Environment sensors	/Cisco-IOS-XE-environment-oper:environment-sensors	IOS-XE native (preservePathPrefix)

For ready-to-apply example CRs see the Worked examples section below.

---

Subscription Mechanics

• IOSXETelemetry CRD: create, update, delete, status reporting
• Per-device subscriber in the cvk-<device> pod
• Dedicated gNMI Subscribe client connection (separate from the config driver's gNMI conn)
• STREAM subscriptions with SAMPLE, ON_CHANGE, or TARGET_DEFINED stream modes
• Multiplexing compatible subscriptions into one gNMI Subscribe RPC per (encoding, sampleInterval) bucket
• Bounded notification buffering with buffer_overflow drops counted in status
• Reconnect with exponential backoff (spec.reconnect)
• Per-stream Path.Origin populated from path syntax (first element's YANG-module prefix) or the explicit subscriptions[].origin field
• subscriptions[].preservePathPrefix keeps the module prefix on the first PathElem.Name instead of lifting it into Path.Origin. Required for IOS-XE native YANG paths because IOS-XE gnxi rejects Cisco-IOS-XE-* values as Path.Origin and instead expects RFC 7951-style module-qualified element names. Leave unset for OpenConfig paths.

Path style	spec.subscriptions[]
OpenConfig	paths: ["/interfaces/interface/state"] and origin: openconfig
IOS-XE native	paths: ["/Cisco-IOS-XE-app-hosting-oper:app-hosting-oper-data/apps"] and preservePathPrefix: true

OpenTelemetry Output

Each gNMI notification is converted into an OpenTelemetry record by the mapper pipeline. CVK can emit logs, metrics, and traces — controlled by output.signal in the CR spec.

Mapper

internal/telemetry/mapper is a pure library. It converts raw *gpb.Notification values into MappedEvents ready for emission, with zero side effects:

• FlattenPath joins prefix + path and preserves the wire order of PathElem keys (no re-sorting)
• Filter runs in two stages: wirePath allow/deny on raw flattened paths (load-shed before alias); metricName allow/deny on the user-facing emitted name (semantic filtering after alias)
• AliasResolver matches longest-prefix-wins
• ResourceAttrExtractor reads pinned leaves into resource attributes
• SeriesKeyCache enforces cardinalityLimits.maxSeriesPerSubscription with dropNewSeries semantics (existing series keep flowing; new series after the cap are suppressed and counted into droppedEvents.cardinality_limit)
• Severity inference for log bodies: UP/ESTABLISHED → INFO, DOWN → WARN, critical/error → ERROR
• Timestamp policy honors timestamps.useCollectorTimestamp (default true): collector wall clock on the OTel record, device timestamp on cisco.device.timestamp

The mapper has no implicit driver dependency and is importable as a standalone package — a future OpenTelemetry Collector receiver can consume it directly.

Logs emitter

internal/telemetry/emit writes OTel LogRecords for:

• string/ASCII leaves (the body becomes the leaf value)
• Delete notifications (body: deleted: <path>, severity Info)

Per-record attributes carry the canonicalized PathElem keys.

Three-provider OTel stack

internal/otelproviders constructs TracerProvider, MeterProvider, and LoggerProvider over a single OTLP gRPC connection. Endpoint and TLS are read from the standard SDK environment:

• OTEL_EXPORTER_OTLP_ENDPOINT (e.g. otelcol.observability:4317)
• OTEL_EXPORTER_OTLP_INSECURE=true to disable TLS

Providers.Logger is wired to the IOSXETelemetryReconciler so each subscriber emits log records via the shared exporter.

When OTEL_EXPORTER_OTLP_ENDPOINT is unset the providers are nil and the emitters fall back to noop providers (no records leave the process).

Status surface

status.observedSubscriptionState[].logRecordsEmitted reports the running count of OTel log records emitted for that subscription.

Metrics

Numeric MDT leaves are emitted as OpenTelemetry metrics. The mapper creates MappedEvent{Signal: metrics} for IntVal, UintVal, FloatVal, DoubleVal, DecimalVal, and JSON/JSON-IETF values whose root value is numeric.

Metric classifier

The reconciler builds the classifier chain per IOSXETelemetry CR:

OverrideClassifier(spec.mapping.metricTypeOverrides, CuratedClassifier())

metricTypeOverrides use longest-prefix-wins matching, so operators can pin a subtree to gauge or sum without waiting for curated defaults. The curated classifier marks well-known monotonic counters as sum, including OpenConfig interface counters, IOS-XE interface statistics, BGP message counters, TCP/UDP packet counters, and app-hosting network counters. CPU, memory, temperature, power, and PoE readings are gauges. Unknown numeric paths default to gauge.

Metrics emitter

MetricsEmitter records:

• gauge events with OTel Float64Gauge.Record
• sum events with OTel Float64Counter.Add

gNMI counters arrive as cumulative values. The emitter stores the last value per series key and emits only the positive delta. If the current value is lower than the previous value, the emitter treats it as a counter reset, records cisco_vk_telemetry_counter_resets_total, stores the new baseline, and skips the negative point.

The mapper also tracks a StartTimestamp per (streamEpoch, seriesKey). Each new Subscribe RPC gets a fresh stream epoch, so downstream consumers can distinguish counter segments after stream restarts.

Self metrics

The metrics path registers these counters on Providers.Meter:

• cisco_vk_telemetry_metric_points_emitted_total (device, subscription, kind)
• cisco_vk_telemetry_classifier_decisions_total (device, subscription, kind)
• cisco_vk_telemetry_counter_resets_total (device, subscription, metric)

status.observedSubscriptionState[].metricPointsEmitted and GET /telemetry/health report metric points emitted per subscription.

State-Transition Traces and YANG Classification

Optional YANG-driven metric classification, recovery spans for watched state transitions, and a shared trace provider for topology spans.

YANG-driven classification

When YANG_MODELS_DIR points at a directory of .yang files, the cvk-<device> process loads those modules once and compiles resolved leaf types into the metric classifier chain:

OverrideClassifier(
  spec.mapping.metricTypeOverrides,
  YangClassifier(yangRegistry, fallback: CuratedClassifier()),
)

The YANG loader resolves typedefs, imports, groupings, uses, leaves, leaf-lists, containers, and lists. Types named counter32 or counter64 classify as sum; integer, unsigned integer, decimal64, string, and enumeration leaves classify as gauge. Unknown or unresolved paths fall through to the curated classifier, so users without YANG_MODELS_DIR keep the Phase 3 behavior.

YANG lookups are memoized in a bounded registry cache. The default cap is 4096 compiled (module, leafPath) lookups; on cap hit, the cache resets all entries rather than doing per-entry LRU eviction.

State-transition spans

spec.mapping.transitions declares state leaves to watch. The mapper emits trace candidate events for those paths when output.signal includes traces. The TracesEmitter tracks each (path, key-set) independently:

• healthy → unhealthy records the observation timestamp
• unhealthy → healthy emits a span from the latest unhealthy observation to the healthy observation
• healthy-only observations do not emit spans

Recovery span names use state.transition.<aliasedName>. Span attributes include from-state, to-state, duration, cisco.gnmi.path, and the path keys such as interface name.

The emitter also records cisco_vk_telemetry_state_transitions_total on Providers.Meter with device, subscription, path, from, and to attributes.

Topology trace consolidation

The topology exporter now accepts a shared trace.TracerProvider. When the MDT telemetry provider stack exists, topology spans use otelproviders.Providers.Tracer and keep their own instrumentation scope, cisco-virtual-kubelet/topology. When no shared provider is available, the exporter preserves the older behavior and builds its own per-device OTLP trace provider from device.otel.

When topology uses the shared provider, endpoint, TLS, headers, and shutdown are owned by internal/otelproviders; topology-specific data remains on span attributes and scope name rather than a separate topology-only resource.

Example: Logs only

apiVersion: config.cisco.vk/v1alpha1
kind: IOSXETelemetry
metadata:
  name: c9300x-state
  namespace: network
spec:
  deviceRef:
    name: c9300x-01
  subscriptions:
    - name: interface-state
      enabled: true
      origin: openconfig
      paths:
        - /interfaces/interface/state
      mode: STREAM
      streamMode: SAMPLE
      sampleInterval: 10s
      encoding: PROTO
  reconnect:
    initialBackoff: 1s
    maxBackoff: 30s
  cardinalityLimits:
    maxSeriesPerSubscription: 10000
    onExceeded: dropNewSeries
  timestamps:
    useCollectorTimestamp: true
  mapping:
    pathAliases:
      - prefix: /interfaces/interface/state
        rename: oc.interface.state
    resourceAttributes:
      - path: /interfaces/interface/state/name
        key: cisco.interface.name
    filter:
      wirePath:
        deny:
          - "**/last-change-time"
      metricName:
        allow:
          - "oc.interface.state.*"
  output:
    signal:
      - logs

Example: Metrics

apiVersion: config.cisco.vk/v1alpha1
kind: IOSXETelemetry
metadata:
  name: c9300x-interface-metrics
  namespace: network
spec:
  deviceRef:
    name: c9300x-01
  subscriptions:
    - name: interface-counters
      enabled: true
      origin: openconfig
      paths:
        - /interfaces/interface/state/counters
      mode: STREAM
      streamMode: SAMPLE
      sampleInterval: 30s
      encoding: PROTO
    - name: cpu-memory
      enabled: true
      paths:
        - /Cisco-IOS-XE-process-cpu-oper:cpu-usage/cpu-utilization
        - /Cisco-IOS-XE-memory-oper:memory-statistics
      mode: STREAM
      streamMode: SAMPLE
      sampleInterval: 30s
      encoding: PROTO
  mapping:
    pathAliases:
      - prefix: /interfaces/interface/state/counters
        rename: cvk.interface.counters
    metricTypeOverrides:
      - prefix: /Cisco-IOS-XE-process-cpu-oper:cpu-usage/cpu-utilization
        type: gauge
      - prefix: /interfaces/interface/state/counters
        type: sum
  output:
    signal:
      - metrics

Example: Full OpenTelemetry — logs, metrics, and traces

apiVersion: config.cisco.vk/v1alpha1
kind: IOSXETelemetry
metadata:
  name: c9300x-full-otel
  namespace: network
spec:
  deviceRef:
    name: c9300x-01
  subscriptions:
    - name: interface-counters
      enabled: true
      origin: openconfig
      paths:
        - /interfaces/interface/state
        - /interfaces/interface/state/counters
      mode: STREAM
      streamMode: SAMPLE
      sampleInterval: 30s
      encoding: PROTO
    - name: bgp-state
      enabled: true
      paths:
        - /network-instances/network-instance/protocols/protocol/bgp/neighbors/neighbor/state/session-state
      mode: STREAM
      streamMode: ON_CHANGE
      encoding: PROTO
  mapping:
    pathAliases:
      - prefix: /interfaces/interface/state
        rename: oc.interface.state
      - prefix: /network-instances/network-instance/protocols/protocol/bgp/neighbors/neighbor/state
        rename: oc.bgp.neighbor.state
    transitions:
      - path: /interfaces/interface[name=*]/state/oper-status
        healthyValues:
          - UP
        unhealthyValues:
          - DOWN
          - LOWER_LAYER_DOWN
      - path: /network-instances/network-instance/protocols/protocol/bgp/neighbors/neighbor[neighbor-address=*]/state/session-state
        healthyValues:
          - ESTABLISHED
        unhealthyValues:
          - IDLE
          - ACTIVE
          - CONNECT
    metricTypeOverrides:
      - prefix: /interfaces/interface/state/counters
        type: sum
  output:
    signal:
      - logs
      - metrics
      - traces

Worked examples

Ready-to-apply IOSXETelemetry CRs for common C9300X MDT-over-gNMI use cases:

• Environmental sensors
• Interface counters and oper-status transitions
• BGP and OSPF counters plus adjacency transitions

Status output examples

After applying an IOSXETelemetry CR, watch the subscription come up:

$ kubectl get iosxetelemetry -w
NAME               DEVICE        PHASE     AGE
cat9k-interfaces   cat9k-smoke   Active    8s

Full status showing per-subscription health and buffer stats:

$ kubectl describe iosxetelemetry cat9k-interfaces
Name:         cat9k-interfaces
Namespace:    default
API Version:  config.cisco.vk/v1alpha1
Kind:         IOSXETelemetry
Spec:
  Device Ref:
    Name:  cat9k-smoke
  Subscriptions:
    Paths:
      /interfaces/interface/state/counters
    Sample Interval:  30000000000
Status:
  Conditions:
    Last Transition Time:  2026-05-30T10:00:08Z
    Message:               1 subscription active
    Reason:                Active
    Status:                True
    Type:                  Ready
  Phase:  Active
  Subscription Stats:
    Active Subscriptions:  1
    Buffer Overflow Total: 0
    Notifications Total:   142
Events:
  Type    Reason     Age   Message
  ----    ------     ----  -------
  Normal  Subscribed  8s   gNMI Subscribe stream established for /interfaces/interface/state/counters

When a subscription error occurs the phase transitions to Degraded and the condition message carries the gRPC status code:

$ kubectl describe iosxetelemetry cat9k-interfaces
...
Status:
  Conditions:
    Last Transition Time:  2026-05-30T10:05:32Z
    Message:               rpc error: code = Unavailable desc = transport is closing
    Reason:                SubscriptionError
    Status:                False
    Type:                  Ready
  Phase:  Degraded
Events:
  Type     Reason       Age   Message
  ----     ------       ----  -------
  Warning  StreamError  12s   gNMI stream lost, reconnecting (attempt 1/5, backoff 2s)
  Normal   Subscribed   8s    gNMI Subscribe stream re-established

Metrics emitted to your OpenTelemetry collector carry the device name and subscription path as attributes, for example:

# Prometheus remote-write preview (interface counters)
interfaces_interface_state_counters_in_octets{
  device="cat9k-smoke",
  interface_name="GigabitEthernet1/0/1"
} 1234567890 1748599200000
interfaces_interface_state_counters_out_octets{
  device="cat9k-smoke",
  interface_name="GigabitEthernet1/0/1"
} 987654321 1748599200000

Future work

• Add curated transition presets for common IOS-XE and OpenConfig operational state paths.