Recover from an interrupted job or damaged state

Recover a deployment rotation while the Console is stopped

For an installer-owned Docker, split Docker or Kubernetes deployment, connect to the installer host over SSH. Use the same private state directory as the maintenance worker:

sudo irisctl maintenance --state-dir /var/lib/iris-installer/iris status

Find the backup, rotation or restore marked recovery-required. Check its operation ID, backup or key type, and recorded results. Resume that operation:

sudo irisctl maintenance --state-dir /var/lib/iris-installer/iris recover \
  --job-id <operation-id> --allow-downtime

Confirm when asked, then run status again to check completion. The worker uses the saved operation record. An accepted request is not proof of completion. For an interrupted backup, recovery restarts the original services and checks whether both archives completed. It keeps incomplete output and never starts another capture. Review the result before requesting a new backup. If the attempt stopped before service changes began, recovery closes it as failed without restarting services. Missing or changed recovery records require inspection; keep the original records and files in place.

The installer manages the worker as a host service outside the containers or pods. If it is unavailable, check irisctl worker-status and use irisctl worker-service --action restart with sudo. Pass the same --state-dir to both. Reconnect recorded backup storage before restarting a worker that reports missing or changed storage.

Restore recovery resumes the recorded replacements and verifies server health and authenticated Console access. A failed restart may have written newer state; the worker preserves it when resuming. Use the same operation ID and retain its original data directories. For a new restore, follow Restore a backup.

For local k3s installations, backup, credential rotation, and restore protect the recorded server and Console images from automatic image cleanup before stopping services. If an image is missing, restore recovers its exact bytes from the authenticated backup.

Replace the independent recovery recipient

Create the replacement with iris-key-setup. Keep an independent private copy off the deployment host. Put a protected copy of this recovery key on the installer host, outside deployment and backup folders. Then run:

sudo irisctl maintenance --state-dir /var/lib/iris-installer/iris replace-recovery \
  --identity /protected/path/recovery.age --independent-copy --allow-downtime

Compare the public recipient with your independent copy and confirm when asked. The worker checks the matching host-prepared key before backup or downtime and checks it again before changing encrypted data. Use status to check the operation. This private recovery key stays outside the Console. Private signing roots must remain on their holders' machines.

Finish an active rotation before preparing another recipient. Existing backups still require their original recovery keys. Retain those keys and your independent copies after the new recipient is active.

What this is for

A job that stops partway through, or storage that comes back with the wrong file permissions, leaves IRIS stuck. Each procedure here clears one state.

What you see

Recovering an IOx attempt cut off mid-run

This covers Industrial Ethernet switches with app hosting, Catalyst 8000 series routers, and Catalyst 9000 series switches with app-hosting storage, where the IOx app replaces the Guest Shell agent. A cut-off onboard or undeploy leaves the device's IOx verification journal in phase indeterminate. Every later job on that device, Force included, then ends in error with result code 3 and the category reconciliation_required. Its job log starts with IOx controller: predecessor recovery failed and names the values to reconcile.

1. Read the binding. Take the record id, transaction id and revision from that job-log line, or from record.iox_verification in GET /api/v1/devices/<id>/deployment. Use the journal's current revision; an older number is refused as stale.

2. Restore verification on the device. In privileged EXEC:

show app-hosting infra | include App signature verification
app-hosting verification enable
show app-hosting infra | include App signature verification

The second read must report App signature verification: enabled. Verification is device-wide, so check the other IOx applications on the device first. See Prepare Industrial Ethernet switches with app hosting, Catalyst 9000 and 8000 devices for the IOx app.

1. Reconcile the journal from inside the server container, as the service user:

docker compose -f server/docker-compose.yml exec -w /opt/iris/server iris \
  python3 iox_verification.py reconcile-enabled \
    --record-id <record-id> --transaction-id <transaction-id> \
    --revision <revision> --acknowledge-external-resolution \
    --wait --wait-timeout 600

It reads the device once over SSH and closes the journal when the device reports enabled. See Helper commands.

1. Clear what the attempt left behind. Run Undeploy with Force in the Console, or POST /api/v1/devices/<id>/undeploy with {"force": true}. Then onboard the device again.

What the exit status means

Job error categories

A job ends with the category reconciliation_required when the journal is indeterminate, or when two unresolved journals claim one device. For HTTP errors, see API error codes (problem types).

Recovering an interrupted IOS-XR teardown

This covers Cisco 8000 series routers on IOS-XR, validated on a Cisco 8201. NCS-540 registration and app startup are validated; undeploy is not yet validated on NCS-540. An interrupted undeploy leaves the deployment record in needs-reconcile, a red badge in the Console. Run undeploy again from that state, record-backed or with Force: each step re-probes the router, including the appmgr application, before acting.

IRIS_XR_SESSION_TIMEOUT bounds every command session to the router, so a wedged router fails the job with a real exit code; 0 turns the bound off. See Server configuration for the default.

A finished teardown may leave an empty iris-work/ directory on harddisk:, which a later onboarding reuses. Undeploy, retire and clean up devices lists what it removes.

Recover an interrupted Guest Shell root-hash job

This covers Catalyst 9000 series switches, where the agent runs in Guest Shell; the IOx app is the alternative on switches with app-hosting storage. When an image of the same name already sits in the IOS root file system, the agent hashes it in place with a native Embedded Event Manager (EEM) policy, IRIS-ROOT-HASH. A lost result reports copy_failed and asks you to look at the device.

1. Remove the IRIS-ROOT-HASH applet from the running configuration.
2. Wait at least 625 seconds for any job already started to finish.
3. Inspect the EEM job history.
4. Remove only <stage_dir>/.iris-root-hash.json. The next check-in retries.

Repair a volume whose file permissions were changed

A storage driver that rewrites permissions on mount widens the private files holding the deployment authority and the IOx transcripts, and the server refuses to start. Install on Kubernetes has the mount settings that prevent this. Keep the failure evidence, then repair the storage:

1. Stop new job admission, wait for all onboard, undeploy and scheduled work to finish, then stop the server pod and any maintenance pods on the volume.
2. Inspect ownership, modes, inode and link metadata against trusted backup evidence, without printing file contents. A change to content or identity needs investigation instead.
3. Restore only the verified, explicitly identified paths, each owned by 10001:10001:

Startup rejects group-writable modes such as 0660 or 0664. Check other authority files and instruction roots against their own rules. 4. Verify the volume root is 10001:10001 with mode 2770, apply the OnRootMismatch policy to the Deployment, then start the server. 5. Recheck the private modes, then verify management API health and deployment-record access before you admit jobs.

Inspect retained per-device migration evidence

The server now keeps one file per device instead of one whole-fleet JSON document per store. See Data formats and states. The original document keeps a .migrated suffix. An invalid placeholder at its old path names the retained document and current data directory. Keep both as evidence.

1. Stop every reader and writer of the selected store.
2. Back up the whole state volume: per-device files, revision metadata, placeholders, and .migrated files.
3. Check the original deployment's recorded source and state paths. Its per-device readers use the current files, not the retired whole-fleet path. For same-deployment recovery, follow Restore a backup with current security records and a consistent backup.
4. Verify inventory and policy together. An empty store is not a successful recovery.

For another release, follow Deploy a new release: remove the old deployment, install its replacement and onboard devices to the new identity.

Replace the two root keys

Replacing the two offline keys that sign every instruction key, and recovering a lost one, are on Replace or recover signing keys.

A server restart also marks any deployment record for a job that was still running as unknown, not active or failed. Do not retry the device operation blindly: read the record first and reconcile from it, the same way as for a cut-off IOx attempt above.

With the API

GET /api/v1/devices/<id>/deployment returns the deployment record, with record.iox_verification and the iox_verification_obligations summary. POST /api/v1/devices/<id>/undeploy runs a teardown; send {"force": true} for Force. Both are described in Console API.

Certificate renewal and backup jobs

Related

• Troubleshoot: symptoms and first steps
• Undeploy, retire and clean up devices
• Back up and restore
• API error codes (problem types)