How to create a ServiceMonitor in Microk8s: Observability Tutorial

Hello,

In this quick tutorial we will explore how to create a ServiceMonitor in microk8s when you have the observability addon enabled.

When you enable the observability addon with:

microk8s enable observability

It will create the obserability namespace and install the Kubernetes Prometheus Stack.

 k get pods -n observability
NAME                                                     READY   STATUS    RESTARTS      AGE
kube-prom-stack-prometheus-node-exporter-htzw9           1/1    
tempo-0                                                  2/2     Running   4 (16h ago)   27h
alertmanager-kube-prom-stack-kube-prome-alertmanager-0   2/2  
kube-prom-stack-kube-prome-operator-cbf896985-fb5fc      1/1     
kube-prom-stack-kube-state-metrics-57c8c84df6-hflgj      1/1   
kube-prom-stack-grafana-74d8979894-sjv77                 3/3     
prometheus-kube-prom-stack-kube-prome-prometheus-0       2/2    
loki-0                                                   1/1   
loki-promtail-sg8bv                                      1/1     Running   

You will get a Grafana configured with a Prometheus data source and other data sources as well.

To monitor a custom service, let’s say a prometheus metrics exporter for Apache Kafka you will need to create a ServiceMonitor. For the service monitor to be discovered you’ll need to add some labels to it.

To find out the labels you need to add describe the prometheus operator:

k describe prometheus/kube-prom-stack-kube-prome-prometheus -n observability

[...]
  Pod Monitor Namespace Selector:
  Pod Monitor Selector:
    Match Labels:
      Release:  kube-prom-stack
  Service Account Name:  kube-prom-stack-kube-prome-prometheus
  Service Monitor Namespace Selector:
  Service Monitor Selector:
    Match Labels:
      Release:  kube-prom-stack

Then create the ServiceMonitor with the corresponding labels: kube-prom-stack.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-exporter-kafka-monitor
  labels:
    team: observability
    release: kube-prom-stack
spec:
  selector:
    matchLabels:
      app: prometheus-kafka-exporter
      app.kubernetes.io/managed-by: Helm
      chart: prometheus-kafka-exporter-1.8.0
      heritage: Helm
      release: prometheus-exporter
  endpoints:
  - port: exporter-port

Thanks for reading!