Upgrades¶

Forge upgrades are mainly release metadata and tenant rollout work.

What Changes¶

Area	File or repo
Forge module refs	`release_versions.yml` or `release_versions.yaml`
Tenant runner settings	`runner_settings.hcl`
AMI IDs	image repo output and tenant specs
ARC controller and scale sets	platform release and EKS compatibility
Integration modules	integration release file and secrets
Reusable actions	operations repo workflow refs

Upgrade Flow¶

Update one non-production release file to the new Forge ref.
Run plan for helpers, infra, platform, and integrations.
Apply one tenant or one example deployment.
Run smoke workflows on EC2 and ARC if both lanes are enabled.
Roll forward by environment.
Keep the previous Forge ref and AMI IDs available for rollback.

ARC/EKS Blue-Green Upgrade¶

Use the blue-green flow when the change touches the EKS foundation or ARC runtime: EKS version, node AMIs, Karpenter, Calico, EBS CSI, Pod Identity, ARC controller, ARC scale-set templates, or cluster-level access integrations.

The operating model is:

two EKS folders per region, usually blue and green
one cluster marked active with is_active: true
one cluster marked inactive with is_active: false
tenant configs pointing arc_cluster_name at the active cluster
tenant configs keeping migrate_arc_cluster: false except during an intentional tenant move

Before starting:

Disable or pause normal promotion and regression workflows for the tenant and infra repos so another apply cannot race the upgrade.
Discover the active and inactive EKS folders and fail if there is not exactly one active and one inactive cluster.
Validate every ARC tenant in the target environment and region points at the active cluster.
Confirm the scripts are available: scripts/reinstall-eks-with-deps.sh and scripts/migrate-tenant.sh.

Run the region in this order:

Destroy the inactive EKS cluster with dependents:

cd terraform/environments/prod/regions/eu-west-1/eks/green
/path/to/forge/scripts/reinstall-eks-with-deps.sh destroy

Recreate the inactive cluster from the updated module refs and config:

yq e -i '.is_active = true' config.yaml
/path/to/forge/scripts/reinstall-eks-with-deps.sh create

Move tenants to the rebuilt cluster one at a time:

/path/to/forge/scripts/migrate-tenant.sh \
  --tf-dir terraform/environments/prod/regions/eu-west-1/vpcs/main/tenants/acme

Run ARC smoke workflows for each moved tenant.
Destroy and recreate the previous active cluster with the same dependency script.
Move tenants again only when your operating model requires tenants to finish on the active color after both clusters have been rebuilt.
Reapply cluster access integrations such as Teleport if your installation deploys them.
Re-enable the paused workflows.

For multi-environment operations, run the same reusable workflow sequentially: dev first, then production regions one at a time. Keep tenant moves max-parallel: 1 so each tenant has a narrow, observable runner gap.

See Move ARC Tenants for the per-tenant mechanics and the workflow automation shape.

Maintenance Cadence¶

Cadence	Check	Where
Weekly	Example deployments still apply and destroy in dependency order.	`forge-examples-iac-aws` workflow
Weekly	Cloud Custodian reports no stale runner instances, old AMIs, or unowned snapshots.	`cloud-custodian` repo
Monthly	Forge module refs, reusable action refs, and container tags have Renovate coverage.	`release_versions.yml`, workflows, Dockerfiles
Monthly	Runner AMIs still have an active and rollback version.	image repos and tenant `runner_settings.hcl`
Monthly	ECR lifecycle policies preserve rollback tags and remove unreferenced tags.	`modules/helpers/ecr` or external registry policy
Quarterly	ARC/EKS blue-green upgrade path is still executable in non-production.	Move ARC Tenants
Quarterly	Tenant IAM roles, ECR access, GitHub App installations, and runner labels still match current tenants.	tenant IaC repo

Use these commands as the starting point for a release review:

rg 'ref: v|ref=v|module_path:|source: git::' release_versions.y*ml terraform examples