Skip to content

Move ARC Tenants Between EKS Clusters

Use this when ARC tenants need to move between blue and green EKS clusters. The same mechanics support EKS and ARC upgrades: rebuild the cluster that is not serving tenants, move tenants to it, then rebuild the other cluster.

This is not a normal tenant onboarding step. Run it only for planned cluster work or recovery from a cluster-level issue.

Preconditions

  • arc_cluster_name ends in -blue or -green.
  • migrate_arc_cluster is false before the move starts.
  • The source and target EKS clusters exist in Terraform/Terragrunt config.
  • The operator or workflow can update kubeconfig for both clusters.
  • The operator or workflow can run targeted Terragrunt applies for the tenant.
  • No normal promotion workflow is applying the same tenant at the same time.

Move Steps

  1. Identify Current and Target Clusters

  2. Read the current cluster name from the tenant config file, for example arc_cluster_name: forge-euw1-prod-green.

  3. Determine the target cluster by switching the suffix from -green to -blue or vice versa.

  4. Scale Down Runner Sets in the Source Cluster

  5. List all runner sets defined under arc_runner_specs.

  6. For each runner set, scale down both minimum and maximum runners to zero on the source cluster to stop new jobs, then wait for running jobs and ARC runner resources to drain.

  7. Disable ARC on the Source Cluster

  8. Update the tenant's config by setting migrate_arc_cluster: true, which disables ARC resources on the source cluster.

  9. Apply this config change so the Terraform/Terragrunt deployment removes ARC for this tenant in the source cluster.

  10. Enable ARC on the Target Cluster

  11. Keep migrate_arc_cluster: true, point arc_cluster_name at the target, and apply once to prove the target has no old ARC footprint.

  12. Set migrate_arc_cluster: false and apply again to enable ARC on the target.

  13. Verify the Handoff

  14. Verify that the tenant ARC footprint is absent from the source cluster.

  15. Confirm the target controller, listeners, NodePool, and EC2NodeClass are healthy.

Automation Script

To simplify and standardize the cluster move, use:

./scripts/migrate-tenant.sh --tf-dir /full/path/to/tenant_dir \
  --from-cluster forge-euw1-prod-green --to-cluster forge-euw1-prod-blue \
  --expected-account-id 123456789012

The script performs these steps:

  • Requires explicit source and destination clusters and validates the authenticated AWS account.
  • Renders Terragrunt inputs to find the AWS profile, region, and ARC cluster name.
  • Updates kubeconfig for both clusters.
  • Validates the source runner-set inventory, stops source scheduling, and waits up to six hours per runner set for running jobs to finish.
  • Sets migrate_arc_cluster: true and applies module.arc_runners against the source cluster to remove the tenant ARC footprint there.
  • Points arc_cluster_name at the target cluster and applies module.arc_runners while migrate_arc_cluster is still true, which keeps the target side clean before enabling it.
  • Sets migrate_arc_cluster: false and applies module.arc_runners again to create the tenant ARC resources on the target cluster.
  • Verifies destination health and source cleanup from the live clusters.
  • Leaves the tenant config pointing at the target cluster with migrate_arc_cluster: false.

Usage Example

Run the script with the tenant directory, source and destination clusters, and expected AWS account ID:

./scripts/migrate-tenant.sh --tf-dir /full/path/to/tenant_dir \
  --from-cluster forge-euw1-prod-green --to-cluster forge-euw1-prod-blue \
  --expected-account-id 123456789012

After the script finishes, run a tenant ARC smoke workflow when one is available, and verify:

kubectl get autoscalingrunnersets -n <tenant>
kubectl get pods -n <tenant>
terragrunt plan --working-dir /full/path/to/tenant_dir

The plan should not try to recreate the tenant on the source cluster.

Workflow Automation Pattern

For production upgrades, put the tenant move behind a GitHub Actions workflow instead of running every command locally. The reusable workflow should do this:

  1. Discover the active and inactive EKS folders from is_active in each cluster config.yaml.
  2. Fail if there is not exactly one active and one inactive cluster.
  3. Validate all tenants currently point at the active cluster.
  4. Run validate-arc-cluster-clean.sh, then destroy and recreate it with scripts/reinstall-eks-with-deps.sh.
  5. Build a tenant matrix from the tenant directories.
  6. Move tenants with scripts/migrate-tenant.sh, using max-parallel: 1.
  7. Prove the previous active cluster is empty with validate-arc-cluster-clean.sh, then destroy and recreate it.
  8. Optionally move tenants back to the final active cluster.
  9. Reapply cluster access integrations such as Teleport when used.

Use a top-level workflow to sequence environments and regions. A practical order is:

dev/eu-west-1 -> prod/eu-west-1 -> prod/us-east-1 -> prod/us-west-2

Keep a workflow-level concurrency group and temporarily disable normal promotion workflows for the affected IaC repos while the blue-green run is in progress.

Safety Rules

  • Do not leave migrate_arc_cluster: true in a tenant config.
  • Do not move tenants in parallel until the process is proven for your cluster and runner capacity model.
  • Do not destroy the source cluster until destination health verification succeeds - run tenant smoke workflows first when one is available.
  • Do not run the normal tenant promotion workflow during the cluster move.
  • Keep the previous runner container image and EKS module refs available until the target cluster is proven.
  • If a move fails after source cleanup, fix the target cluster and rerun the tenant script from the same tenant directory.

Use Upgrades for the full ARC/EKS blue-green upgrade sequence.