Splunk Dashboard Panel Reference¶
This page lists the Forge Splunk dashboards and the panels operators should expect to see. Use it with the Splunk Dashboard Runbook.
The exact SPL and SignalFlow live in the Terraform dashboard definitions:
- Splunk Cloud:
modules/integrations/splunk_cloud_conf_shared - Splunk Observability:
modules/integrations/splunk_o11y_conf_shared
Splunk Cloud Log Dashboards¶
| Dashboard | What it answers | Panels |
|---|---|---|
| Forge ARC DIND Runner Lifecycle | Are ARC DIND runners, hooks, init containers, versions, and log volumes healthy? | ARC DIND and Hook Categories; ARC DIND and Hook Trend; Hook Sidecar Signals; Init Container Failures; Runner Versions; DIND Runner Log Volume |
| Forge ARC K8S Runner Lifecycle | Are ARC K8S runner pods, hooks, PVCs, versions, and log volumes healthy? | Hook Sidecar Signals; Hook Sidecar Trend; K8s Pod and PVC Events; K8s Pod and PVC Events by Scale Set; K8s Runner Versions; K8s Runner Log Volume |
| Forge CI Job Details | Which jobs ran, queued, failed, and consumed runner time? | Queued Jobs by Time Window; Busiest Windows by Runner Type and Repo; Runner Type Timing; Runner Type Usage by Time Window; Average Workflow Run Duration by Time Window; Workflow Run Duration by Workflow; Busiest Windows by Runner Type and Workflow Job; Workflow Job Runner Usage; CI Job Details |
| Forge EC2 Fleet Scale-Up Failures | Why did EC2 CreateFleet scale-up fail? |
AMI and Launch Template Failures; Failed CreateFleet Requests by Error Code; Failed Overrides by Instance Type and Subnet; Temporary vs Repeating Fleet Failure Windows; Recent CreateFleet Request Drilldown; Fleet Tenant Error Summary |
| Forge EC2 RunInstances Scale-Up Failures | Why did non-fleet RunInstances scale-up fail? |
Temporary vs Repeating RunInstances Failure Windows; Recent RunInstances Request Drilldown; Failed RunInstances Requests by Error Code; SQS Retry Pressure for RunInstances; RunInstances Tenant Error Summary |
| Forge EC2 Runner Lifecycle | Where did the EC2 runner lifecycle fail? | AMI and SSM Update Failures; Runner Hook Activity; EC2 Lifecycle Lambda Signals; EC2 Lifecycle Lambda Trend; Runner Instance Log Symptoms; Runner Versions; Scale Capacity and AWS API Errors |
| Forge GitHub Webhook Workflow Job Events | Did Forge receive the expected GitHub workflow_job events? |
Per-Tenant Completed Job Outcomes; EC2 Queued Jobs > 15 Minutes; Workflow and Branch Activity; Non EC2 Queued Jobs > 15 Minutes; Failed Jobs; Canceled Jobs; Forge GitHub Webhook Workflow Job Events |
| Forge Ingestion Quality | Is Splunk data fresh, structured, and searchable? | Runner Log Ingestion Quality; Missing Structured Fields; Top Sources; Sourcetype Inventory; High-Volume Sourcetype Trend |
| Forge Kubernetes Storage and Network | Are Kubernetes storage, identity, scheduling, CNI, and EKS API paths healthy? | EKS API Storage and Controller Errors; EBS Provisioning Waits; EBS CSI Health; EKS Pod Identity Health; Calico and Tigera Health; Karpenter Controller Health; Storage, Scheduler, and CNI Signals; Storage, Scheduler, and CNI Trend; PVC Still Attached |
| Forge Lambda Operations | Are Forge support Lambdas failing or retrying? | Splunk S3 Runner Log Ingestion Retries; Lambda Error Trend; Lambda Errors by Category; Latest Lambda Error Samples; EC2 Runner Tagging Failures |
| Forge Runner Capacity | Are jobs waiting for capacity? | ARC Runner Set State; EC2 Runner Pool Symptoms; Queue Pressure by Runner Group; Queue p95 Trend |
| Forge Runner Control Plane Health | Are scale-up, top-up, cleanup, GitHub API, and SQS control-plane paths healthy? | GitHub and SQS API Warnings; Pool Top-Up Activity; Pool Top-Up Trend; Scale-Down Cleanup Signals; Scale-Down Cleanup Trend; Scale-Up Registration and Post-Launch Signals; Scale-Up Registration Trend |
| Forge Runner Dispatcher Rejections | Are workflow jobs rejected because labels or policies do not match? | Latest Rejection Samples; Rejection Summary; Rejected Labels; Rejected Repositories; Rejection Trend |
| Forge Stuck Workflow Job Dispatcher Health | Is stuck-job redelivery healthy overall? | Alert Quality and Receiver Decisions; Receiver Decision Trend; Failures and Warnings; Tenant and Repository Hot Spots; Worker Outcomes; Worker Outcome Trend |
| Forge Stuck Workflow Job Dispatcher Debug | What happened to one stuck-job redelivery key? | GitHub Delivery Attempts; Failure Samples; End-to-End Redelivery Lifecycle; Latest Log Samples; Receiver Decision Detail; Runner Group Capacity Decisions |
| Forge Tenant Logs | What raw logs exist for one tenant? | Tenant log search controls and raw event results |
| Forge Troubleshooting | Which subsystem should the operator inspect next? | ARC Listener Activity; ARC Runner Set State; EC2 Runner Symptom Samples; Actionable EC2 Runner Symptoms; EKS API 403 Authorization Errors; EKS API Errors by Cluster and Resource; Ingestion Trend by Sourcetype; Failure Hot Spots; GitHub Job Result Overview; Kubernetes Event Categories; Lambda Errors by Category; Lambda Error Samples; Negative Queue Time Data Quality; Queue and Duration by Runner Group; Event Volume by Sourcetype; Top Sources; Trust Validation AssumeRole Failures; Latest Trust Validation Detail |
| Forge Trust Failures | Which tenant role trust relationships are failing? | Basic AssumeRole Failures; Latest Validation Detail; Trust Failure Trend; Validator AccessDenied Errors |
| Forge Webhook Job Log Pipeline | Where did job-log capture or ingestion break? | Archiver Errors; Dispatcher Enqueues by Repo; Dispatcher to Archived JSON Counts; Pipeline Stage Summary; Pipeline Trend; DLQ Redrive Activity |
Splunk Observability Dashboards¶
| Dashboard | What it answers | Charts |
|---|---|---|
| Billing | Which tenants and services drive AWS cost? | Cost per service; Net Cost per service; Net Cost per tenant; Cost per tenant; Total Cost; Total Net Cost; Runner-related net cost; Top tenant/service net cost |
| DynamoDB | Are DynamoDB tables throttled, slow, or returning errors? | Write throttle events; System errors; Percentage of read capacity consumed; Returned item count; Average request latency; Throttled requests; User errors; Read throttle events; Percentage of write capacity consumed |
| EBS | Are EBS volumes saturated, slow, or unavailable? | Byte utilization; Write latency; Read ops; Write throughput; Read/write bytes breakdown; Read latency; Read throughput; State; Total read time; Latency per op; Total write time; Read vs write ops; Average queue length; Idle time; Write ops |
| Forge Impact | How much EC2 and K8S runner capacity is being used by tenant? | Total runners by runtime; Runner-minutes by runtime; Active EC2 runners by tenant and instance type; Active EC2 runners by tenant; Total EC2 runners by tenant; EC2 runner-minutes by tenant; EC2 runner-minutes by tenant and instance type; Active K8S runners by tenant; Total K8S runners by tenant; K8S runner-minutes by tenant |
| Lambda | Are Lambda functions erroring, throttled, slow, or spilling over concurrency? | Provisioned concurrent executions; Provisioned concurrency invocations; Provisioned concurrency spillover invocations; Total spillover invocations; Percent invocations by version; Errors by version; Total throttles; Average duration; Throttles by version; Invocations; Total errors; Provisioned concurrency utilization; Total invocations |
| OpenCost | Which Kubernetes tenants and pods drive compute cost? | Tenant hourly compute cost; Tenant monthly compute run rate; Tenant compute cost trend; Tenant CPU cost; Tenant memory cost; Top pod compute cost |
| Runner EC2 | Are EC2 runner hosts healthy at the resource layer? | Disk ops; Memory overview; Network out; Top instances by CPU utilization; Disk utilization; Disk I/O; Network errors; Memory page swaps; Active hosts per instance type; CPU utilization; Active hosts by AZ; Hosts with agent installed; EC2 status check failures |
| Runner K8S | Are Kubernetes runner pods healthy at the resource layer? | Available pods by deployment; Top CPU usage per pod; Network bytes per second; Desired pods by deployment; Network errors per second; Memory usage; Active pods; Top pods by memory; Pods by phase; Pod phase trend; Container restarts; Splunk OTel collector pod health |
| SQS | Are queues backing up or sending work to DLQs? | Queues; Top queues by message sent; Sent message size; Messages by state; Oldest message age; Empty receives; Top queues by message received; Message processing trend; Messages deleted; DLQ backlog trend; DLQ oldest message age; DLQ visible messages |
Panel Reading Rules¶
- Start with freshness before severity.
- Prefer tenant, repository, runner group, region, and time-window filters before reading a table as a platform-wide signal.
- Treat failure counts without denominator as suspicious. Compare failures to total events, jobs, runners, or tenants.
- Treat missing
forgecicd_tenantas a data-quality issue unless the sourcetype is intentionally platform-wide. - Use trends for broad health and tables for ownership or drilldown.
- Use Splunk Cloud logs to explain what happened, and Splunk Observability metrics to explain resource pressure.