Prometheus

How This Fits Our End-to-End Pipeline

Prometheus is documented here as part of one connected build, not as an isolated tutorial. The goal is to explain how code moved from a developer laptop into a monitored Kubernetes service.

Developer laptop
  | git add / commit / push
  v
GitHub repository
  | pull request + branch rules
  v
GitHub Actions workflow
  | test -> docker build -> tag -> push
  v
Docker Hub image registry
  | immutable image tag
  v
GitOps manifests repository/path
  | ArgoCD watches desired state
  v
Kubernetes cluster
  | app pods + services + ingress
  v
Prometheus scrapes metrics -> Grafana dashboards

Theory

Prometheus collects time-series metrics by scraping HTTP endpoints. Instead of applications pushing metrics into a central server, Prometheus pulls metrics from configured targets on a schedule.

In our pipeline, Prometheus answers operational questions after deployment: are pods up, are restarts increasing, is latency rising, and did the latest rollout change system behavior?

Metrics Architecture

Application /metrics endpoint
        |
        | scraped every N seconds
        v
Prometheus server
        |
        +-- stores time series
        +-- evaluates alert rules
        +-- serves PromQL queries
        v
Grafana dashboards and alerts

Implementation

• Expose application metrics at /metrics when the app supports it.
• Use Kubernetes service discovery or ServiceMonitor resources when using the Prometheus Operator.
• Scrape Kubernetes components and kube-state-metrics for cluster-level visibility.
• Query metrics with PromQL and visualize them in Grafana.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: devops-demo
    static_configs:
      - targets: ["devops-demo.staging.svc.cluster.local:80"]

Useful PromQL

Pod restarts

increase(kube_pod_container_status_restarts_total[15m])

Shows restart growth over a time window.

HTTP request rate

sum(rate(http_requests_total[5m])) by (status)

Shows request throughput grouped by status.

CPU usage

sum(rate(container_cpu_usage_seconds_total[5m])) by (pod)

Estimates CPU usage by pod.

Memory usage

container_memory_working_set_bytes

Shows working set memory in bytes.

What We Monitored First

• Pod readiness and restarts.
• HTTP request rate and status codes.
• Container CPU and memory.
• Deployment replica availability.
• ArgoCD sync and health status when available.

Alert Thinking

• Alert on user impact and sustained symptoms, not every small metric movement.
• A pod restarting once may be useful context; a deployment crash-looping for five minutes is alert-worthy.
• Dashboards are for exploration; alerts are for action.
• Every alert should have an owner and a next step.

Interview Notes

• Prometheus uses a pull model and stores time-series data with labels.
• Labels are powerful but high-cardinality labels can hurt performance.
• PromQL is the query language for aggregating and filtering metrics.
• Prometheus is not a log system; pair it with logging tools when needed.
• Alertmanager handles routing, grouping, silencing, and notification delivery.

FAQs

Is Prometheus enough for monitoring?

It is excellent for metrics. Real production observability also includes logs, traces, alerts, and runbooks.

What should the app expose?

Request count, latency, errors, build info, and business-specific health metrics.

Why do labels matter?

Labels let you slice metrics by pod, namespace, route, status code, and version.

Memorize vs Look-up

Project Implementation Journal

Baseline

We first made sure the application or configuration worked before introducing Prometheus. A broken baseline makes every later automation failure harder to understand.

Local proof

We validated the Prometheus workflow locally where possible, because local feedback is faster than waiting for CI or a cluster reconciliation loop.

Repository proof

We committed the Prometheus change as a reviewable unit so the reason for the pipeline change was visible in Git history.

Automation handoff

We connected Prometheus to the next tool in the chain instead of treating it as a standalone exercise.

Failure check

We intentionally inspected the common failure signals for Prometheus: logs, status output, permissions, names, tags, and configuration paths.

Rollback thinking

We asked how to return to the previous working state if the Prometheus change caused a bad deployment.

Interview compression

We reduced Prometheus into a few sentences that explain purpose, implementation, and failure modes clearly.

Operations note

We documented what someone should check the day after the deployment, not only what to run during setup.

Decision Records

• Keep Prometheus configuration in Git where it can be reviewed.
• Prefer explicit names over clever names: repository, image, namespace, workflow, and application names should be searchable.
• Use immutable versions for anything that can be deployed or rolled back.
• Put secrets in the platform secret store, not in code, documentation screenshots, or shell history.
• Automate only after the manual path is understood.
• Make the happy path visible, then document the first five things to check when it fails.
• Separate staging and production concerns before the project becomes too large.
• Choose boring defaults unless there is a real operational reason to customize.
• Write commands so they can be pasted into a terminal after replacing obvious placeholders.
• Treat dashboards, manifests, and workflow files as production code once people rely on them.

Failure Modes We Learned To Recognize

Wrong name

The most ordinary failures came from mismatched names: image repository, namespace, service selector, branch, workflow file, or dashboard variable.

Wrong permission

Automation failed when a token could read but not write, push but not pull, or access staging but not production.

Wrong version

A deployment looked successful while the cluster still ran an old image tag or an image tag that had been overwritten.

Wrong assumption

A command that worked locally failed in CI because the runner had a different shell, path, network, or credential context.

Missing feedback

Without logs, status commands, metrics, or dashboards, the system gave no quick answer about what changed.

Manual drift

Manual cluster edits solved a momentary problem but made the GitOps source of truth inaccurate.

Interview Drill Questions

• What problem does Prometheus solve in this pipeline?
• What artifact or state does Prometheus produce?
• Which tool consumes the output of Prometheus next?
• What is the most likely beginner mistake with Prometheus?
• How would you prove Prometheus worked without guessing?
• How would you roll back a bad change involving Prometheus?
• What should be memorized versus looked up for Prometheus?
• Which security boundary matters most for Prometheus?
• How would you explain Prometheus to someone who only knows basic Linux?
• What metric, log, status, or command would you check first during an incident?

Glossary For This Stage

Artifact

A build output or configuration object that can be handed to another stage.

Desired state

The state declared in Git or YAML that controllers try to make real.

Reconciliation

The loop where a tool compares desired state with actual state and fixes differences.

Immutable version

A version reference that should never change meaning after publication.

Rollback

A controlled return to a previously known working state.

Drift

A difference between what Git says should exist and what is actually running.

Health

A status signal that says whether the service is ready and operating correctly.

Traceability

The ability to connect a running system back to a commit, workflow run, image, and manifest change.

Practical Runbook

Confirm source

Identify the exact repository, branch, commit, file path, or dashboard connected to Prometheus.

Confirm identity

Check the account, token, kube context, registry namespace, or runner label before assuming the tool is broken.

Confirm version

Write down the version or tag you expected and compare it with the version the platform reports.

Confirm status

Use the native status command or UI first; it usually tells you whether the failure is configuration, permission, or runtime.

Confirm logs

Logs explain what happened after the tool accepted the configuration but the process still failed.

Confirm network

Many CI/CD failures are actually DNS, registry, cluster, firewall, or service discovery failures.

Confirm ownership

Know whether the application team, platform team, security team, or repository owner controls the failing setting.

Confirm rollback

Before changing more things, decide whether the quickest safe move is to revert, resync, rebuild, or redeploy.

Confirm documentation

After fixing the issue, add the command, symptom, and fix to the project notes so the next run is faster.

Confirm automation

If the Prometheus step is repeated manually more than twice, turn it into a workflow, manifest, script, or checklist.

Verification Checklist

• Can I point to the exact Git commit connected to this Prometheus change?
• Can I explain what changed in one sentence?
• Can I prove the change worked with a command or status screen?
• Can I identify the next tool that consumes this output?
• Can I identify the secret, token, or permission that would break this step?
• Can I roll back without manually editing production state?
• Can I tell whether the failure is build-time, deploy-time, or runtime?
• Can I show the relevant logs or metrics?
• Can I repeat the setup on a fresh machine or cluster?
• Can I teach this stage without reading every command from the page?