Architecture

This reference is for contributors and operators investigating implementation behavior. For a short introduction and the application deployment path, start with How CVK works. Task guides for installation, placement and operations are linked from the documentation home.

Overview

Cisco Virtual Kubelet implements the Virtual Kubelet provider interface. In the default per-device worker model, a supported device appears as a virtual Node; assigned Pods are translated into device app-hosting operations. IOS-XE uses RESTCONF for this path, while the Beta NX-OS provider uses NX-API CLI.

The runtime split

Everything the project does comes out of a single cisco-vk CLI with isolated runtime subcommands:

cisco-vk manager — the Kubernetes controller. Watches infrastructure CRs and reconciles them into Kubernetes worker resources. In the default worker model, it delegates device and external-controller API calls to those workers. The optional configuration-aggregator mode instead runs supported configuration reconciliation inside the manager process.
cisco-vk run — the Virtual Kubelet provider. One process per device. Reads its config, registers a virtual node in the cluster, and drives the device via RESTCONF when pods come and go.
cisco-vk controller-worker — the network-controller runtime. One process per NetworkController whose type has a registered adapter; it starts that adapter and watches only its Kubernetes namespace. Two adapters are registered today, sdwan and nexus-dashboard; other types create no worker.

The default split keeps device/controller API calls and device-side signing workflows inside their workers. The opt-in configuration aggregator belongs to the manager's trust boundary and is incompatible with managed topology. The manager writes credential Secret references and locally validates dedicated gNOI trust and provisioning material, including signer keys when present. Its cluster-wide Secret watch also places typed Secret objects in its cache and memory. Treat the manager as part of the credential trust boundary described in Security. Each controller worker receives only its target's mounted credentials; its Kubernetes API RBAC and cache are namespace-scoped.

Network controller extension boundary

External network controllers use the same isolation principle without changing the per-device provider or its transports. The manager resolves a registered controller type and creates one namespace-scoped worker per endpoint. A product adapter inside that worker consumes resolved, versioned controller-centric Network as Code intent and speaks the controller's native API. Endpoint processes, credentials, failure domains, and rate limits are separate, but the namespace remains the API trust/RBAC/cache boundary. Use a dedicated namespace when endpoints or config authors are not mutually trusted. Duplicate fencing compares the exact stored endpoint string within that namespace only. The sdwan adapter applies intent to SD-WAN Manager only in apply mode and implements no remote prune or delete; the nexus-dashboard adapter does not change ND configuration and, when opted in, writes only CiscoDevice objects in the cluster. Each adapter's worker RBAC is a static, least-privilege ClusterRole bound per namespace. See SD-WAN and Nexus Dashboard. See the Network Controller Extension Guide for the registry, worker, model, ownership, and security contracts.

Opt-in managed topology: placement and software campaigns

Managed topology adds an opt-in ownership and campaign-admission path while retaining the native Kubernetes scheduler. The manager projects protected declared topology and owns Node identity; the app-hosting worker reports its permitted status and handles eligible Pods. The network-management worker handles configuration, diagnostics and explicitly enabled software operations. Two shared functional ServiceAccounts have RO/RW role options; exact worker binding and native admission remain essential because RBAC alone is not per-device authority.

Decision	Responsible component	Input / result
Where a Pod runs	Native Kubernetes scheduler	Affinity/spread over projected Node labels; app worker reconciles the assigned Pod
Which devices may enter a campaign	CVK rollout manager	Immutable approved `IOSXESoftwareRollout` plan, canaries and declared domain budgets
Which source supplies each device	CVK source selection	Topology-scoped endpoint with frozen URL, digest and Secret identity
How software is changed	Network worker / IOS-XE lifecycle driver	Gated secure gNOI install/activate/verify and recovery; `PrepareOnly` retains an install receipt and requires separate activation approval
Whether an eligible Pod can be drained	Preview manager/worker drain protocol	Native Eviction/PDB plus exact Pod/device-clean and recovery evidence

The existing app-hosting diagram below remains the simple path; managed software changes are CRD operations, not Pods scheduled onto a device. Topology labels do not establish live forwarding health, link headroom or critical-service availability. Native 1.37 TAS is a separate experimental scheduler lane and is not required for the Kubernetes 1.35+ managed baseline. See Managed Topology and Rollouts for opt-in gates, ownership migration, workload eligibility and remaining qualification limits.

Component architecture

---
config:
  layout: dagre
  theme: base
  look: classic
  themeVariables:
    background: "#ffffff"
    mainBkg: "#f8fafc"
    clusterBkg: "#f1f5f9"
    clusterBorder: "#94a3b8"
    lineColor: "#64748b"
    edgeLabelBackground: "#f8fafc"
    titleColor: "#1e293b"
---
flowchart TB
    subgraph Cluster["Kubernetes cluster"]
        API["API server"]
        CRD["CiscoDevice CR"]
        Controller["CiscoDevice controller / cisco-vk manager"]
        CM["ConfigMap / device config"]
        Dep["Deployment / device VK"]
        VK["VK pod / cisco-vk run"]
        Node["Virtual node"]
        Pod["User pod"]
    end

    subgraph IOSXE["Cisco IOS-XE device"]
        IOx["IOx platform"]
        Container["App-hosting container"]
    end

    API --> CRD
    CRD --> Controller
    Controller --> CM
    Controller --> Dep
    Dep --> VK
    VK --> Node
    Pod -->|"scheduled to"| Node
    VK -->|"RESTCONF over HTTPS"| IOx
    IOx --> Container

    style Controller fill:#ede9fe,stroke:#7c3aed,color:#3b0764
    style VK fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    style IOx fill:#dcfce7,stroke:#16a34a,color:#14532d
    style Container fill:#d1fae5,stroke:#059669,color:#064e3b
    style API fill:#f1f5f9,stroke:#64748b,color:#1e293b
    style CRD fill:#f1f5f9,stroke:#64748b,color:#1e293b
    style CM fill:#f1f5f9,stroke:#64748b,color:#1e293b
    style Dep fill:#f1f5f9,stroke:#64748b,color:#1e293b
    style Node fill:#f1f5f9,stroke:#64748b,color:#1e293b
    style Pod fill:#e2e8f0,stroke:#94a3b8,color:#334155

Core components

AppHostingProvider

Implements the Virtual Kubelet nodeutil.Provider interface — the main entry point for pod operations.

Method	Purpose
`CreatePod(ctx, pod)`	Delegates to `driver.DeployPod`, then forces a node status update
`UpdatePod(ctx, pod)`	Oper-state-aware redeploy: skips apps already RUNNING; reinstalls apps that are stuck in unexpected states
`DeletePod(ctx, pod)`	Delegates to `driver.DeletePod` then forces a node status update
`GetPod(ctx, ns, name)`	Fast path: informer cache. Fallback: device query for cleanup
`GetPodStatus(ctx, ns, name)`	Queries live pod status from the device; returns `PullingImage` waiting state while a copy-then-install fallback is in progress
`GetPods(ctx)`	Lists all managed pods discovered on the device
`GetStatsSummary(ctx)`	Returns Kubernetes stats/summary data (see Observability)
`GetMetricsResource(ctx)`	Returns Prometheus metrics (see Observability)

Operations that don't make sense for app-hosting — RunInContainer, AttachToContainer, GetContainerLogs, PortForward — return HTTP 501.

AppHostingNode

Implements node.NodeProvider for node registration, heartbeat, and status.

Ping() — throttled to a 30-second minimum; triggers syncNodeStatus asynchronously
NotifyNodeStatus(cb) — fires the callback whenever node status changes (labels, annotations, capacity, conditions)
ForceStatusUpdate() — called after every pod lifecycle event so resource accounting stays fresh

The node's labels include the CVK identity and type=virtual-kubelet. Standalone compatibility mode preserves the historical platform-family fallback for otherwise-unknown region/zone values. Opt-in managed topology does not invent placement facts: the manager projects only administrator allowlisted CiscoDevice.metadata.labels, while the per-device worker writes Node status only. The default scheduler consumes those labels through native affinity and topology-spread constraints. See Managed topology and topology-aware IOS-XE rollouts.

Node annotations populated from driver observations remain advisory telemetry; CDP/OSPF observations never rewrite declared scheduler topology. See Observability → Node annotations.

Conditions published:

Ready — true when IOx is enabled on the device
DiskPressure — true when device storage < 5% available

Driver factory

func NewDriver(ctx, spec) (CiscoKubernetesDeviceDriver, error)

Selects a driver based on spec.Driver:

XE → IOS-XE driver (production)
NXOS → NX-OS driver (NX-API CLI app-hosting and the initial NX-API REST/DME Network as Code config slice: system, vlan, interface_ethernet)
FAKE → mock driver for testing
XR, OPENCONFIG → placeholders, currently unsupported

Driver interfaces

Every driver must implement CiscoKubernetesDeviceDriver:

type CiscoKubernetesDeviceDriver interface {
    GetDeviceResources(ctx) (*v1.ResourceList, error)
    GetDeviceInfo(ctx) (*common.DeviceInfo, error)
    DeployPod(ctx, pod)
    UpdatePod(ctx, pod)
    DeletePod(ctx, pod)
    GetPodStatus(ctx, pod) (*v1.Pod, error)
    ListPods(ctx) ([]*v1.Pod, error)
    GetGlobalOperationalData(ctx) (*common.AppHostingOperData, error)
}

Drivers may additionally implement the optional TopologyProvider interface — this is what enables OTEL topology export, the cisco_device_cdp_*/ospf_*/interface_* metrics, and the cisco.io/* node annotations.

type TopologyProvider interface {
    GetCDPNeighbors(ctx) ([]common.CDPNeighbor, error)
    GetOSPFNeighbors(ctx) ([]common.OSPFNeighbor, error)
    GetInterfaceStats(ctx) ([]common.InterfaceStats, error)
    GetInterfaceIPs(ctx) ([]common.InterfaceIP, error)
    GetHostedApps(ctx) ([]common.HostedApp, error)
}

The IOS-XE driver implements it. Drivers without topology support still work — their VKs just skip the optional metrics/traces/annotations.

IOS-XE driver — internal layering

internal/drivers/iosxe/ is organised by responsibility. Each layer is independently testable.

File	Responsibility
`driver.go`	Driver construction, marshallers, config hooks, recovery state helpers
`device.go`	Device-level queries (connectivity, resources, device info)
`client.go`	App-hosting RPCs, image delivery (device-native HTTP pull + copy-then-install fallback), Kubernetes event emission
`reconciler.go`	App lifecycle state machine (see below)
`pod_lifecycle.go`	`DeployPod` / `DeletePod` / `GetPodStatus` / `ListPods`
`pod_transforms.go`	Pod.Spec → IOS-XE `AppHostingConfig`
`status_transforms.go`	Device oper-data → Pod.Status
`ip_discovery.go`	Pod IP discovery (oper-data first, ARP table fallback)
`topology.go`	`TopologyProvider` implementation — CDP, OSPF, interfaces
`models.go`	YANG structs, auto-generated via ygot

IOS-XE Config Driver and NetAsCode

The IOS-XE configuration plane is separate from App Hosting. IOSXEConfig resources carry NetAsCode-shaped intent, and the configdriver resolves the same kind of hierarchy NetAsCode uses: defaults, device groups, interface groups, templates, and per-device configuration. The resolved intent remains plain YAML/JSON data until a family writer owns it.

Family writers are the translation boundary. They take canonical NetAsCode fields, apply release-aware overrides, and emit transport.Op values addressed to IOS-XE YANG paths. The engine then validates and applies those operations over RESTCONF, NETCONF, or gNMI.

IOSXEConfig source
  -> intent resolver
  -> family writer
  -> version override table
  -> YANG validation boundary
  -> RESTCONF, NETCONF, or gNMI transport
  -> IOS-XE device
  -> status, revision, and apply log

The validation boundary is deliberately device-facing. CONFIG_YANG_VALIDATION controls it:

Value	Behaviour
`disabled`	no validation gate; default for backward compatibility
`warn`	log validation failures and continue
`strict`	fail before mutation

This preserves NetAsCode as the stable public model while giving CVK a place to use release-specific ygot/ytypes validation as those generated model packages are added.

NX-OS Config Driver

NX-OS uses the same common config runtime and status contract, but the device-facing boundary is NX-API REST/DME rather than YANG. The NX-OS adapter preserves the full NetAsCode nxos: envelope long enough to resolve global, device_groups, devices, model templates, variables, and interface_groups, then normalizes supported families into the writer contract.

NXOSConfig source
  -> NetAsCode envelope resolver
  -> family writer
  -> DME object payload
  -> neutral REST helper
  -> NX-API REST/DME transport
  -> NX-OS device
  -> common status and apply result

The same transport contract also exposes a read-only diagnostic execution path. For NX-OS that path intentionally uses NX-API CLI, while declarative configuration writes use NX-API REST/DME.

Data flow

Controller reconciliation

---
config:
  layout: dagre
  theme: default
  look: classic
---
sequenceDiagram
    actor User
    participant API as K8s API
    participant Ctrl as CiscoDevice Controller
    participant CM as ConfigMap
    participant Dep as Deployment
    participant VK as VK Pod

    User->>API: kubectl apply ciscodevice.yaml
    API->>Ctrl: Reconcile event
    Ctrl->>Ctrl: Render device config (strip password + secretRef)
    Ctrl->>CM: Create/Update {name}-config
    Ctrl->>Dep: Create or update {name}-vk with VK_DEVICE_PASSWORD from Secret
    Dep->>VK: Start pod
    VK->>API: Register virtual node

The recommended credentialSecretRef path passes the password through a valueFrom.secretKeyRef environment variable. Both password and credentialSecretRef are stripped from the rendered device ConfigMap. The legacy inline spec.password fallback still persists the plaintext value in the CiscoDevice and the Deployment environment; use a Secret reference instead. See credential injection and the legacy fallback.

Pod creation flow

---
config:
  layout: dagre
  theme: default
  look: classic
---
sequenceDiagram
    actor User
    participant API as Kubernetes API
    participant VK as AppHostingProvider
    participant Drv as IOS-XE driver
    participant Dev as IOS-XE device

    User->>API: Create Pod targeting the virtual node
    API->>VK: CreatePod
    VK->>Drv: DeployPod
    Drv->>Dev: POST app-hosting config with RESTCONF

    alt Flash package already on the device
        Drv->>Dev: Install from flash path
        Dev-->>Drv: DEPLOYED or RUNNING
    else HTTP URL supported by the device
        Drv->>Dev: Install from HTTP URL
        Dev-->>Drv: DEPLOYED
    else HTTP URL requires copy fallback
        Drv->>Dev: Copy image to flash
        Note over VK,Drv: Pod status reports PullingImage while the copy runs
        Drv->>Dev: Install from copied flash path
        Dev-->>Drv: DEPLOYED
    end

    opt App is not already running
        Drv->>Dev: RPC activate
        Dev-->>Drv: ACTIVATED
        Drv->>Dev: RPC start
        Dev-->>Drv: RUNNING
    end

    Drv->>Dev: Query oper-data for pod IP
    alt IP in oper-data
        Dev-->>Drv: IPv4
    else fallback
        Drv->>Dev: Query ARP for container MAC
        Dev-->>Drv: IPv4
    end
    Drv-->>VK: Pod ready

Pod deletion flow

Reverse of creation: RUNNING → stop → ACTIVATED → deactivate → DEPLOYED → uninstall → config DELETE. If a state is skipped (e.g. app already stopped), the reconciler picks up from wherever the observed state actually is.

App lifecycle state machine

The IOS-XE driver's reconciler (reconciler.go) drives each app toward its desired state by observing live state and issuing a single RPC per pass.

---
config:
  layout: dagre
  theme: default
  look: classic
---
stateDiagram-v2
    [*] --> NoConfig
    NoConfig --> INSTALLING: POST config + install RPC
    INSTALLING --> DEPLOYED: validation passes
    INSTALLING --> Error: pkg-policy invalid + notification
    DEPLOYED --> ACTIVATED: activate RPC
    ACTIVATED --> RUNNING: start RPC
    STOPPED --> RUNNING: start RPC
    RUNNING --> [*]
    Error --> [*]

    RUNNING --> STOPPED: stop RPC (reverse)
    STOPPED --> DEPLOYED: deactivate RPC
    DEPLOYED --> NoConfig: uninstall + DELETE config

Key behaviours:

INSTALLING — normally transient. The reconciler waits unless pkg-policy = iox-pkg-policy-invalid and a confirming install notification has been received from the device. During the first few seconds of every install the device reports pkg-policy = invalid as a YANG default; waiting for the notification prevents this transient value from being treated as a fatal error. If spec.allowUnsignedApps = true, the check is skipped entirely — use this when you're running unsigned packages (e.g. your own custom application builds, or devices not enforcing signed-verification).
STOPPED — restartable directly via start (no re-activate needed).
No oper data with config present — reconciler re-issues the install RPC.
Error — surfaces as Pod Failed with reason PackagePolicyInvalid and a message from the device's notification.
RPC acceptance is not lifecycle completion — IOS-XE can return HTTP 2xx while placing a command error in Cisco-IOS-XE-rpc:output/result. The driver decodes that bounded response and accepts only a complete, operation-specific success result bound to the expected app ID, package, and target state. Explicit rejection fails closed; an unrecognized or empty 2xx result is mutation-ambiguous and is reconciled from oper-data before any replay. Device text is never copied into errors or logs. Even a matched result means only that the request was accepted; lifecycle completion is determined exclusively from oper-data.

Reverse path (desired = Deleted)

Observed state	Action
`RUNNING`	`stop`
`ACTIVATED` or `STOPPED`	`deactivate`
`DEPLOYED`	`uninstall`
No oper data	`DELETE` config

Pod status and discovery

Pod-to-app naming

App IDs use cvk%04d_<podUID>: the container index is zero-padded to four digits and the Pod UID has hyphens stripped. Example: cvk0000_a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4. This identity format is not a statement of a device's supported container count; platform capabilities and application packaging determine that limit.

This lets the driver rebuild pod identity when listing apps from the device — the UID is embedded in the app ID, and pod metadata (name, namespace, container name) is stored as key-value labels in the app's --run-opts:

io.kubernetes.pod.name=<name>
io.kubernetes.pod.namespace=<namespace>
io.kubernetes.pod.uid=<uid>
io.kubernetes.container.name=<container>

Container state mapping

IOS-XE state	Container state	Ready
`RUNNING`	Running	`true`
`DEPLOYED`, `ACTIVATED`, `INSTALLING`	Pending	`false`
`STOPPED`	Terminated (exit 0)	`false`
Error / `pkg-policy-invalid` confirmed	Terminated with exit code + reason	`false`

IP discovery

Query app-hosting oper-data for ipv4-addr
Fallback: ARP table lookup by the container's MAC address
Default: 0.0.0.0 (no IP yet)

Pod recovery loop

A background goroutine in cisco-vk run periodically lists all pods on this node in Failed phase and resets them to Pending if their status.Reason is one of:

NotFound
ProviderFailed
PackagePolicyInvalid

This lets the VK controller reprocess the pod without manual kubectl delete/apply. The recovery loop uses exponential backoff: 15 s → 30 s → 60 s → 120 s → 240 s → 300 s (5 min cap). On any successful recovery the interval resets to 15 s.

Pods with a DeletionTimestamp are skipped — the normal deletion path owns those.

Networking

Three interface modes are supported (see Configuration for field reference, and the platform-specific pages Catalyst 8000V / Catalyst 9000 for device-side setup):

Mode	IP source	Use case
VirtualPortGroup	DHCP pool on VPG or static from `podCIDR`	Catalyst 8000V; containers share a private L3 to the device
AppGigabitEthernet	DHCP (access) or DHCP in VLAN (trunk)	Catalyst 9000; containers on a dedicated front-panel interface with optional VLAN tagging
Management	DHCP or static on the management interface	Containers sharing the device management network

DHCP flow (VirtualPortGroup example)

┌────────────┐    ┌──────────────────┐    ┌──────────────────┐
│ Container  │───▶│ VirtualPortGroup0│───▶│   DHCP Pool      │
│ (eth0)     │    │  gateway .254    │    │   /24 subnet     │
│            │◀───│                  │◀───│   assigns IP     │
└────────────┘    └──────────────────┘    └──────────────────┘

Provider-side IP discovery always happens after deploy completes — it polls oper-data first, then falls back to ARP, then retries on the next reconcile.

Observability

See Observability for the full reference. At a high level:

Prometheus metrics are served on the standard kubelet /metrics/resource endpoint and cover device CPU, memory, storage, per-interface rates/state, and CDP/OSPF neighbor counts.
Kubernetes stats/summary is served on /stats/summary, enabling kubectl top node.
OpenTelemetry topology traces are emitted on a configurable interval (default 60 s) and include a device root span with child link spans (one per CDP/OSPF neighbor) and app spans (one per hosted container).
Node annotations (cisco.io/router-id, cisco.io/hostname, cisco.io/cdp-neighbor-count, cisco.io/ospf-neighbor-count, cisco.io/protocols) make topology context queryable via kubectl describe node.

gNOI and software lifecycle

gNOI operations and IOS-XE software lifecycle management have their own operator-facing section: gNOI and Software Lifecycle. At the architecture level, the important boundary is that read-only operations, write-class actions, and software upgrades use separate CRDs, separate runtime gates, and separate RBAC grants. Read-only gNOI access does not implicitly enable reboot, file write, factory reset, or OS activation.

The API keeps protocol-level transport and optional gNOI-specific trust under spec.gnoi; system roots are the implicit default, while reusable custom trust remains under spec.tls. Platform behavior lives under the driver section, currently spec.xe.gnoi.certificateProvisioning. A single worker-side resolver selects the effective port, authentication adapter, and one trust source. The bundled secure-password metadata adapter is IOS-XE-specific: explicit username/password authentication fails closed for other drivers until they provide their own adapter, while certificate-only generic TLS remains portable. Generic gNOI TLS and IOS-XE provisioning are intentionally mutually exclusive: provisioning derives its bootstrap and steady-state trust from its own Secret. This leaves a stable protocol boundary for future drivers without generalizing IOS-XE CSR or certificate-install semantics prematurely.

Generic Kubernetes TLS Secret resolution stays manager-owned. The controller validates a same-namespace gNOI TLS Secret, projects only ca.crt and the optional tls.crt/tls.key pair read-only, and renders fixed internal paths for the worker. A valid Secret resource-version change rolls the worker when gNOI is enabled in per-device topology, rebuilding its immutable TLS client state; Secret bytes and the generic TLS Secret name are not copied into its ConfigMap. With global gNOI disablement, the per-device worker remains but the trust projection and Secret-driven rollout are omitted. Aggregated config-only topology does not create a per-device worker.

The manager also validates provisioning Secret contents locally, inside the Secret trust boundary. This validation does not contact the device. Known-invalid optional gNOI material degrades only that worker's gNOI configuration, allowing signer removal and unrelated configuration/credential updates to proceed. Transient Kubernetes read failures retry without discarding a working projection. Resource sizing belongs here too: spec.worker controls the real worker's requests/limits and /tmp cap, is removed from rendered device config, and never changes a virtual Node's advertised application capacity.

The per-device gNOI client uses a workload-classed gRPC connection pool so small control RPCs and bulk OS/file transfers use separate connections and do not block each other. gNMI configuration and telemetry currently maintain independent connections. The client also caches per-service capability probes: when a device returns codes.Unimplemented, later calls to that service fail fast with ErrServiceUnsupported instead of repeatedly attempting an unsupported RPC.

The worker-scoped IOS-XE runtime lives in internal/drivers/iosxe/gnoiruntime, not in command wiring. Its Provider owns the per-device pool leases and client reset lifecycle. Certificate installation is a separate Provisioner, injected into the operational-action reconciler only when certificate provisioning is configured and write-class gNOI is explicitly enabled. The base client provider therefore does not implicitly grant certificate-install authority.

Certificate policy and protocol validation remain in the gnoi package behind the CertificateSigner interface. The included implementation is only a transitional, PEM-backed local signer for ca.key. It remains reusable for a new immutable action after a definitive pre-Install failure; network Install replay protection belongs to the action reconciler. The parsed key remains in the worker until process exit or the controller completes a key-free rollout. This repository does not implement or configure an external CA, KMS, or HSM signer backend.

The runtime serializes client resets against the target CSR/Load exchange and submits Install at most once. After IOS-XE restarts gNXI, it uses fresh connections for bounded, read-only certificate and OS.Verify checks; only an exact installed-certificate/TLS-peer match followed by OS.Verify on that same peer is reported as a successful mutation. A pre-existing successful OS.Verify is a non-mutating service-ready result and does not attest the requested certificate ID or digest.

Software lifecycle separates portable gNOI operations from platform-native inventory handling. URL and ConfigMap sources are content-addressed byte streams consumed by gNOI OS.Install. A small driver-neutral lifecycle interface exposes exact inventory inspection and optional device-file registration; the IOS-XE implementation uses only its RESTCONF install RPC. It has no CLI fallback, and another driver must explicitly register its own backend before accepting a device-file source. Preinstalled sources never call registration.

The reconciler persists source identity and request markers before side effects. Staging, activation, and rollback therefore use at-most-once replay policy: after an ambiguous response or restart, CVK observes durable device state and does not repeat a recorded mutation. This favors preventing duplicate reloads and install operations over claiming exactly-once delivery. Inventory matching is exact and unique before activation; unsupported or ambiguous platform state fails closed. ISSU remains disabled until a driver can verify the selected activation path, so NoReboot must not be interpreted as a non-disruptive-upgrade guarantee.

The driver-neutral internal/devicecoordination package supplies one device-disruptive-mutation Lease family shared by operational actions and software upgrades. Namespaced device identity is hashed into the Lease key, while workflow and immutable object UID form the holder identity. The orchestration layer owns acquisition, durable intent, release, and uncertain- outcome quarantine; platform drivers only implement their native observation and mutation boundary. Replicas reconciling the same CR intentionally share that holder identity, so the Lease cannot distinguish an overlapping old worker from its replacement; per-device Deployments therefore use Recreate while mutation controllers are enabled. Because the key uses CiscoDevice namespace and name rather than network address, aliases for one physical device do not contend and must be prohibited operationally. A future NX-OS or IOS XR lifecycle driver should reuse this coordination family and the generic gNOI byte-source primitives, add any native inventory/device-file backend, and provide its own platform API, controller registration, and runtime wiring.

internal/provider/maintenance joins ordinary IOS-XE configuration and app-hosting writes to this boundary and owns the maintenance Node taint. Injected callbacks cover entire transactions and asynchronous app recovery, not individual RESTCONF calls; reads remain available. Startup receives the same guard before optional device bootstrap writes. Each routine write uses a unique holder, renews while running, and retains its bounded Lease on uncertain completion. The IOS-XE observer also checks existing quarantine when mutation gates are disabled. This requires list-only access to both mutation CRDs even with strict RBAC and disabled mutation gates; it grants no mutation authority.

Portable coordination does not make the current IOS-XE lifecycle controller portable. A future driver must supply its own version comparison, supervisor and activation semantics, inventory observations, certificate workflow, and recovery qualification. Reuse the narrow interfaces after those requirements are demonstrated; do not make NX-OS/IOS XR inherit IOS-XE error strings or CSR policy. Recreate controls Deployment rollouts, not arbitrary Pod deletion, out-of-band changes, or ownership across Kubernetes clusters.

RESTCONF endpoints

Operation	Method	Endpoint
App config (create/list)	POST/GET	`/restconf/data/Cisco-IOS-XE-app-hosting-cfg:app-hosting-cfg-data/apps`
App config (delete)	DELETE	`/restconf/data/Cisco-IOS-XE-app-hosting-cfg:app-hosting-cfg-data/apps/app={id}`
App oper-data	GET	`/restconf/data/Cisco-IOS-XE-app-hosting-oper:app-hosting-oper-data/app`
App-hosting lifecycle and verification RPCs	POST	`/restconf/operations/Cisco-IOS-XE-rpc:app-hosting`
Device version	GET	`/restconf/data/Cisco-IOS-XE-native:native/version`
CDP neighbors	GET	`/restconf/data/Cisco-IOS-XE-cdp-oper:cdp-neighbor-details`
OSPF neighbors	GET	`/restconf/data/Cisco-IOS-XE-ospf-oper:ospf-oper-data`
Interfaces	GET	`/restconf/data/ietf-interfaces:interfaces`
ARP table	GET	`/restconf/data/Cisco-IOS-XE-arp-oper:arp-data`

YANG models

Cisco-IOS-XE-app-hosting-cfg — app config
Cisco-IOS-XE-app-hosting-oper — app runtime state
Cisco-IOS-XE-rpc — app-hosting lifecycle and verification RPCs
Cisco-IOS-XE-cdp-oper — CDP neighbor discovery
Cisco-IOS-XE-ospf-oper — OSPF neighbor state
Cisco-IOS-XE-arp-oper — ARP-based IP discovery

Project layout

cisco-virtual-kubelet/
├── api/v1alpha1/              # CiscoDevice API types (DeviceSpec, XEConfig, OTELConfig)
├── api/config/v1alpha1/       # IOS-XE config, telemetry, diagnostic API types
├── api/ops/v1alpha1/          # DeviceOperation, gNOI action, and software-upgrade API types
├── cmd/cisco-vk/              # Unified binary
│   ├── main.go                # cobra root
│   ├── run.go                 # 'run' subcommand — the VK provider
│   └── manager.go             # 'manager' subcommand — the controller
├── charts/cisco-virtual-kubelet/  # Helm chart (controller + RBAC + CRDs)
├── config/crd/                # Generated CRD manifests
├── internal/
│   ├── config/                # YAML/viper loader
│   ├── controller/            # CiscoDevice reconciler
│   ├── provider/              # AppHostingProvider, metrics, OTEL topology
│   └── drivers/               # Driver implementations
│       ├── common/            # Shared types (DeviceInfo, CDPNeighbor, etc.)
│       ├── iosxe/             # IOS-XE driver (see layering table above)
│       └── fake/              # Mock driver for tests
├── examples/                  # Example device configs and pod manifests
└── docs/                      # This documentation