Operations Runbook
Choose a guide for the task you need to complete. Maintaining the CVK installation changes Kubernetes controllers and schemas; maintaining a device can change its configuration or running operating system. Plan those changes separately so the expected result and recovery path are clear.
| Task | Start here | Expected result |
|---|---|---|
| Upgrade or roll back CVK | Installation maintenance | CRDs, chart and controller image match the selected release. |
| Inspect a device without changing it | Device diagnostics | An auditable DeviceOperation contains output or a concrete failure. |
| Provision a certificate, reboot, or perform another device action | Opt-in device actions | A separately authorized action records its invocation and outcome. |
| Upgrade or downgrade IOS-XE | IOS-XE upgrade and downgrade runbook | The device's running version is independently verified. |
| Investigate health, metrics, or failures | Observability and Troubleshooting | Evidence identifies the affected controller, worker, or device. |
For a new installation, begin with Getting started. For managed software campaigns, use the managed topology overview before creating lifecycle requests. The single-device procedure and the campaign procedure have different authorization and coordination requirements.
Read-only diagnostics and mutating requests use separate CRDs and RBAC. The
examples require a working CiscoDevice and its per-device worker; requests
must live in the target device's namespace. Keep the mutation gates disabled
when collecting diagnostics. Device actions and software upgrades remain Beta,
require explicit gates, and need a maintenance window and a tested recovery
path. A successful reboot acknowledgement alone does not prove device recovery.
Links from earlier versions of this guide
The former combined runbook is now split by operator task. Existing section links below lead to the corresponding procedure or reference.
Upgrading CRDs: back up, review, apply, then roll the controller.
After a CRD upgrade: existing resources and schema compatibility.
Rolling back: check stored data before reverting schemas.
DeviceOperation: prerequisites and a complete example.
Supported read-only kinds: commands, captures and gNOI probes.
Implementation boundary: shared read-only processing and separate mutation handling.
Write-class actions: gates, confirmation and one-shot dispatch.
Shared device-mutation Lease: serialization, quarantine and deletion behavior.
Software upgrade API details: image sources, strategies and migration compatibility.
RBAC: diagnostic permissions and separate grants for mutations.
Admin exec wrapper: transient operations through the local admin endpoint.
Status: outputs, artifacts, terminal phases and cleanup.
DeviceOperation output: inspect results and events.
Action output: acceptance versus convergence.
Upgrade output: phases and verification evidence.
Remaining boundaries: admission, API evolution, artifact storage and rollback scope.