GitHub Runners

How This Fits Our End-to-End Pipeline

GitHub Runners is documented here as part of one connected build, not as an isolated tutorial. The goal is to explain how code moved from a developer laptop into a monitored Kubernetes service.

Developer laptop
  | git add / commit / push
  v
GitHub repository
  | pull request + branch rules
  v
GitHub Actions workflow
  | test -> docker build -> tag -> push
  v
Docker Hub image registry
  | immutable image tag
  v
GitOps manifests repository/path
  | ArgoCD watches desired state
  v
Kubernetes cluster
  | app pods + services + ingress
  v
Prometheus scrapes metrics -> Grafana dashboards

Theory

A runner is the machine that executes a GitHub Actions job. GitHub-hosted runners are created on demand by GitHub. Self-hosted runners are machines we register and maintain ourselves.

In our CI/CD journey, hosted runners were enough to build and push Docker images. Self-hosted runners become useful when jobs need private cluster access, special tools, persistent caches, or controlled network paths.

Runner Decision Diagram

Need standard build/test only? ---- yes ---> GitHub-hosted runner
        |
        no
        v
Need private network, custom hardware, or local cluster access?
        |
        yes
        v
Self-hosted runner with hardening, labels, and restricted repository access

Implementation

1. Start with runs-on: ubuntu-latest for normal CI jobs.
2. Add a self-hosted runner only when a job has a clear requirement that hosted runners cannot satisfy.
3. Register the runner at repository, organization, or enterprise scope.
4. Use labels such as self-hosted, linux, and k8s-access to target the right runner.
5. Protect self-hosted runners from untrusted pull request code.

Workflow Usage

jobs:
  deploy-check:
    runs-on: [self-hosted, linux, k8s-access]
    steps:
      - uses: actions/checkout@v4
      - name: Check cluster access
        run: kubectl get ns

Operational Notes

• Keep the runner service updated and monitor disk usage because Docker builds can fill disks quickly.
• Use separate runners or runner groups for production-sensitive work.
• Never expose a powerful self-hosted runner to arbitrary fork pull requests.
• Clean workspace and Docker artifacts regularly if jobs build images locally.

Commands

Check service

./svc.sh status

On Linux self-hosted runners, verifies whether the runner service is active.

Start runner manually

./run.sh

Useful during initial setup and debugging.

Inspect labels

# GitHub UI -> Settings -> Actions -> Runners

Runner labels determine which jobs can use the machine.

Security Checklist

• Use runner groups for sensitive runners.
• Do not share production credentials with general build runners.
• Patch the runner host regularly.
• Clean workspaces after jobs.
• Prefer ephemeral runners for high-security workloads.

Interview Notes

• A runner executes jobs; it is not the workflow itself.
• Hosted runners are ephemeral and reduce maintenance burden.
• Self-hosted runners trade control for responsibility and security risk.
• Runner labels are how workflows target specific execution environments.
• Production deployment credentials should be isolated from general CI jobs.

FAQs

Do we need a self-hosted runner for ArgoCD?

Not usually. ArgoCD pulls from Git and syncs itself, so GitHub Actions can remain registry-and-Git focused.

Can self-hosted runners run Docker builds?

Yes, but disk cleanup, daemon access, and privilege boundaries must be planned.

Where should runners be registered?

Repository scope is simplest. Organization runner groups are better for shared controlled infrastructure.

Memorize vs Look-up

Project Implementation Journal

Baseline

We first made sure the application or configuration worked before introducing GitHub Runners. A broken baseline makes every later automation failure harder to understand.

Local proof

We validated the GitHub Runners workflow locally where possible, because local feedback is faster than waiting for CI or a cluster reconciliation loop.

Repository proof

We committed the GitHub Runners change as a reviewable unit so the reason for the pipeline change was visible in Git history.

Automation handoff

We connected GitHub Runners to the next tool in the chain instead of treating it as a standalone exercise.

Failure check

We intentionally inspected the common failure signals for GitHub Runners: logs, status output, permissions, names, tags, and configuration paths.

Rollback thinking

We asked how to return to the previous working state if the GitHub Runners change caused a bad deployment.

Interview compression

We reduced GitHub Runners into a few sentences that explain purpose, implementation, and failure modes clearly.

Operations note

We documented what someone should check the day after the deployment, not only what to run during setup.

Decision Records

• Keep GitHub Runners configuration in Git where it can be reviewed.
• Prefer explicit names over clever names: repository, image, namespace, workflow, and application names should be searchable.
• Use immutable versions for anything that can be deployed or rolled back.
• Put secrets in the platform secret store, not in code, documentation screenshots, or shell history.
• Automate only after the manual path is understood.
• Make the happy path visible, then document the first five things to check when it fails.
• Separate staging and production concerns before the project becomes too large.
• Choose boring defaults unless there is a real operational reason to customize.
• Write commands so they can be pasted into a terminal after replacing obvious placeholders.
• Treat dashboards, manifests, and workflow files as production code once people rely on them.

Failure Modes We Learned To Recognize

Wrong name

The most ordinary failures came from mismatched names: image repository, namespace, service selector, branch, workflow file, or dashboard variable.

Wrong permission

Automation failed when a token could read but not write, push but not pull, or access staging but not production.

Wrong version

A deployment looked successful while the cluster still ran an old image tag or an image tag that had been overwritten.

Wrong assumption

A command that worked locally failed in CI because the runner had a different shell, path, network, or credential context.

Missing feedback

Without logs, status commands, metrics, or dashboards, the system gave no quick answer about what changed.

Manual drift

Manual cluster edits solved a momentary problem but made the GitOps source of truth inaccurate.

Interview Drill Questions

• What problem does GitHub Runners solve in this pipeline?
• What artifact or state does GitHub Runners produce?
• Which tool consumes the output of GitHub Runners next?
• What is the most likely beginner mistake with GitHub Runners?
• How would you prove GitHub Runners worked without guessing?
• How would you roll back a bad change involving GitHub Runners?
• What should be memorized versus looked up for GitHub Runners?
• Which security boundary matters most for GitHub Runners?
• How would you explain GitHub Runners to someone who only knows basic Linux?
• What metric, log, status, or command would you check first during an incident?

Glossary For This Stage

Artifact

A build output or configuration object that can be handed to another stage.

Desired state

The state declared in Git or YAML that controllers try to make real.

Reconciliation

The loop where a tool compares desired state with actual state and fixes differences.

Immutable version

A version reference that should never change meaning after publication.

Rollback

A controlled return to a previously known working state.

Drift

A difference between what Git says should exist and what is actually running.

Health

A status signal that says whether the service is ready and operating correctly.

Traceability

The ability to connect a running system back to a commit, workflow run, image, and manifest change.

Practical Runbook

Confirm source

Identify the exact repository, branch, commit, file path, or dashboard connected to GitHub Runners.

Confirm identity

Check the account, token, kube context, registry namespace, or runner label before assuming the tool is broken.

Confirm version

Write down the version or tag you expected and compare it with the version the platform reports.

Confirm status

Use the native status command or UI first; it usually tells you whether the failure is configuration, permission, or runtime.

Confirm logs

Logs explain what happened after the tool accepted the configuration but the process still failed.

Confirm network

Many CI/CD failures are actually DNS, registry, cluster, firewall, or service discovery failures.

Confirm ownership

Know whether the application team, platform team, security team, or repository owner controls the failing setting.

Confirm rollback

Before changing more things, decide whether the quickest safe move is to revert, resync, rebuild, or redeploy.

Confirm documentation

After fixing the issue, add the command, symptom, and fix to the project notes so the next run is faster.

Confirm automation

If the GitHub Runners step is repeated manually more than twice, turn it into a workflow, manifest, script, or checklist.

Verification Checklist

• Can I point to the exact Git commit connected to this GitHub Runners change?
• Can I explain what changed in one sentence?
• Can I prove the change worked with a command or status screen?
• Can I identify the next tool that consumes this output?
• Can I identify the secret, token, or permission that would break this step?
• Can I roll back without manually editing production state?
• Can I tell whether the failure is build-time, deploy-time, or runtime?
• Can I show the relevant logs or metrics?
• Can I repeat the setup on a fresh machine or cluster?
• Can I teach this stage without reading every command from the page?