Module 16 - Monitoring with Prometheus

This repository contains a demo project created as part of my DevOps studies in the TechWorld with Nana – DevOps Bootcamp.

Demo Project: Configure Alerting for our Application

Technologies used: Prometheus, Kubernetes, Linux

Project Description:

• Configure our Monitoring Stack to notify us whenever CPU usage > 50% or Pod cannot start
• Configure Alert Rules in Prometheus Server
• Configure Alertmanager with Email Receiver

---

Prerequisites

A running Kubernetes cluster with the kube-prometheus-stack Helm chart installed in the monitoring namespace. See the previous demo project for the setup:

https://github.com/explicit-logic/prometheus-module-16.1

---

Configure Alert Rules in Prometheus Server

Goal: get notified whenever CPU usage exceeds 50% or a Pod cannot start (crash looping).

Inspect the default rules

The kube-prometheus-stack already ships with a large set of alert rules. Each rule group is defined as a PrometheusRule custom resource, which the Prometheus Operator converts into native Prometheus rule files:

# list all rule resources
kubectl get prometheusrule -n monitoring

# inspect one of them
kubectl get prometheusrule -n monitoring monitoring-kube-prometheus-alertmanager.rules -o yaml

The generated rule files are mounted into the Prometheus pod. They are fully managed by the Operator, so editing them directly is not allowed — changes would be overwritten on the next sync:

Create custom alert rules

Instead of editing the generated files, we define our own rules in a separate PrometheusRule resource — alert-rules.yaml:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: main-rules
  namespace: monitoring
  labels:
    app: kube-prometheus-stack
    release: monitoring

spec:
  groups:
  - name: main.rules
    rules:
    - alert: HostHighCpuLoad
      expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 50
      for: 2m
      labels:
        severity: warning
        namespace: monitoring
      annotations:
        description: "CPU load on host is over 50%\n Value = {{ $value }}\n Instance = {{ $labels.instance }} \n"
        summary: "Host CPU load high"

    - alert: KubernetesPodCrashLooping
      expr: kube_pod_container_status_restarts_total > 5
      for: 0m
      labels:
        severity: critical
        namespace: monitoring
      annotations:
        description: "Pod {{ $labels.pod }} is crash looping\n Value = {{ $value }}"
        summary: "Kubernetes pod crash looping "

Key points:

• The release: monitoring label is required — the Prometheus Operator only picks up PrometheusRule resources that match its rule selector (derived from the Helm release name).
• HostHighCpuLoad — calculates the CPU usage as 100 - idle%, averaged per node over a 2-minute window. for: 2m means the expression must stay above 50% for 2 minutes before the alert fires; until then it is in pending state.
• KubernetesPodCrashLooping — fires as soon as a container has restarted more than 5 times (for: 0m = no delay).
• The namespace: monitoring label on each alert matters later: AlertmanagerConfig routing (configured below) is namespace-scoped and matches alerts on this label.
• annotations support templating — {{ $value }} and {{ $labels.<name> }} are rendered into the notification text.

PrometheusRule API reference: https://docs.redhat.com/en/documentation/openshift_container_platform/4.22/html/monitoring_apis/prometheusrule-monitoring-coreos-com-v1

Apply and verify

Apply the rules:

kubectl apply -f alert-rules.yaml

Check that the resource was created:

kubectl get prometheusrule -n monitoring

The config-reloader sidecar detects the new rule file and triggers a live reload of Prometheus — no restart needed:

kubectl logs prometheus-monitoring-kube-prometheus-prometheus-0 -n monitoring -c config-reloader
kubectl logs prometheus-monitoring-kube-prometheus-prometheus-0 -n monitoring -c prometheus

The new main.rules group appears in the Prometheus UI under Status → Rules:

Test the Alert Rules

To watch the CPU usage while testing, open Grafana → Dashboards → Kubernetes / Compute Resources / Cluster and edit the query of the CPU Utilisation panel to match the alert expression:

100 - (avg (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100)

Set the unit to Percent (0 - 100):

Simulate a CPU spike

Use the containerstack/cpustress image to stress 4 CPU cores for 30 seconds:

kubectl run cpu-test --image=containerstack/cpustress -- --cpu 4 --timeout 30s --metrics-brief
kubectl get pod

The CPU utilisation starts increasing on the Grafana panel.

Go to the Prometheus UI → Alerts: the status of HostHighCpuLoad changes from inactive to pending — the expression is true, but the 2-minute for window has not elapsed yet:

Alertmanager Configuration File

Prometheus evaluates the rules and fires alerts; Alertmanager is the component that turns firing alerts into notifications (email, Slack, PagerDuty, …).

Forward the Alertmanager service port to access its UI:

kubectl get svc -n monitoring

# runs in the background (&)
kubectl port-forward service/monitoring-kube-prometheus-alertmanager -n monitoring 9093:9093 &

The current Alertmanager configuration is visible at http://127.0.0.1:9093/#/status

The effective configuration file itself lives in a generated Secret, gzip-compressed. To decode it:

kubectl get secret alertmanager-monitoring-kube-prometheus-alertmanager-generated -n monitoring -o yaml

# copy the value of the `alertmanager.yaml.gz` key and decode it
# (on Linux use `base64 -d` instead of `-D`)
echo -n '<value of alertmanager.yaml.gz>' | base64 -D | gunzip

The default config routes everything to a "null" receiver — alerts arrive but no notifications go out. We will add an email receiver next.

Configure Email Notification

We use Resend as the SMTP provider.

Create a Resend API key

Go to https://resend.com/api-keys?new=true and add a new API key:

• Name: Prometheus
• Permission: Sending Access
• Domain: All domains

Store the API key in a Secret

Alertmanager reads the SMTP password from a Kubernetes Secret, so the credential never appears in the config in plain text. Create resend-secret.yaml:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: resend-secret
  namespace: monitoring
data:
  token: <base64 encoded Resend API key>

Encode the key with echo -n 're_xxxxxxxx' | base64. The -n flag matters — a trailing newline would corrupt the credential.

Create the secret:

kubectl apply -f resend-secret.yaml

Create the AlertmanagerConfig

Just like alert rules, the Alertmanager configuration is managed declaratively through a custom resource — the Operator merges every AlertmanagerConfig into the generated config. Create alert-manager-configuration.yaml:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: main-rules-alert-config
  namespace: monitoring
spec:
  route:
    receiver: 'email'
    repeatInterval: 30m
    routes:
    - matchers:
      - name: alertname
        value: HostHighCpuLoad
    - matchers:
      - name: alertname
        value: KubernetesPodCrashLooping
      repeatInterval: 10m
  receivers:
  - name: 'email'
    emailConfigs:
    - to: 'delivered@resend.dev'
      from: 'delivered@resend.dev'
      smarthost: 'smtp.resend.com:587'
      authUsername: 'resend'
      authIdentity: 'resend'
      authPassword:
        name: resend-secret
        key: token

Key points:

• route — matches our two alerts by alertname and sends both to the email receiver. The Operator additionally injects a namespace="monitoring" matcher, which is why the alert rules carry the namespace: monitoring label.
• repeatInterval — how often a still-firing alert is re-sent: every 30 minutes by default, every 10 minutes for the more critical KubernetesPodCrashLooping.
• emailConfigs — SMTP settings for Resend. The username is the literal string resend; the password is pulled from the resend-secret Secret created above. delivered@resend.dev is Resend's sandbox address, so no domain verification is required for this demo.

Apply the configuration:

kubectl apply -f alert-manager-configuration.yaml

Verify

kubectl get alertmanagerconfig -n monitoring

# the config-reloader picks up the change and hot-reloads Alertmanager
kubectl logs alertmanager-monitoring-kube-prometheus-alertmanager-0 -n monitoring -c config-reloader

The merged routes and the new email receiver appear in the UI at http://127.0.0.1:9093/#/status:

Test Email Notification

CPU spike → HostHighCpuLoad

Simulate a CPU spike again, this time long enough for the alert to fire:

kubectl run cpu-test --image=containerstack/cpustress -- --cpu 4 --timeout 60s --metrics-brief
kubectl get pod

If the cpu-test pod still exists from the earlier test, delete it first: kubectl delete pod cpu-test

The alert stays in pending state for 2 minutes (the for duration), then transitions to firing:

Firing alerts are pushed to Alertmanager — verify via its API at http://127.0.0.1:9093/api/v2/alerts:

Alertmanager sends the email notification. Check the inbox at https://resend.com/emails:

If no email arrives, check the Alertmanager logs for SMTP errors:

kubectl logs alertmanager-monitoring-kube-prometheus-alertmanager-0 -n monitoring -c alertmanager

CrashLoopBackOff → KubernetesPodCrashLooping

To emulate a CrashLoopBackOff, run a pod whose container exits immediately. With --restart=Always, Kubernetes keeps restarting it — wait until the restart counter exceeds 5 (the alert threshold):

kubectl run crash-test --image=busybox --restart=Always -- sh -c "exit 1"

# watch the restarts
kubectl get pod -w

The KubernetesPodCrashLooping alert fires immediately after the 6th restart (for: 0m) and the email notification arrives:

Cleanup

kubectl delete pod cpu-test crash-test