Operations¶
This section is for the platform team after the first Forge tenant is running. It covers tenant support, image management, artifacts, cleanup, secrets, upgrades, and troubleshooting.
Day-2 Loop¶
| Cadence | Action | Doc |
|---|---|---|
| Every tenant request | Collect tenant values, add config, run plan, run smoke workflow. | Tenant Onboarding |
| Every image release | Build base/custom images, share AMIs, update tenant runner specs. | Runner Images |
| Weekly | Run example apply/destroy for helpers, infra, platform, integrations. | Workflow Blueprints |
| Weekly | Run cleanup and policy jobs. | Cloud Custodian |
| Monthly | Review module refs, Renovate output, AMI age, stale ECR tags, and secrets. | Upgrades |
| Planned ARC upgrade | Rebuild blue/green EKS clusters and move tenants one at a time. | Move ARC Tenants |
| Incident | Triage queued jobs, failed runner registration, IAM, webhooks, or ARC. | Troubleshooting |
| Incident | Debug Terraform, OpenTofu, or Terragrunt plan/apply that appears stuck. | Terraform/Terragrunt Stuck Runbook |
| Incident | Use Splunk dashboards to identify the failing subsystem and severity. | Splunk Dashboard Runbook |
Operating Repos¶
If you need an end-to-end operating model, copy from Operations Repo Blueprints. The blueprints include real folders for Packer, Ansible, containers, Renovate, Cloud Custodian, Terragrunt, reusable actions, and weekly example deployments.
For Splunk-based operations, start with the Splunk Dashboard Runbook. Use the panel reference when you need to map a dashboard panel back to its operational question.
For installations that do not deploy Splunk, use Troubleshooting Without Splunk as the baseline support runbook.