If your nodes cannot communicate with eachother via TCP or UDP, but ping works, check that you’re not using VMXNET3 interfaces on vmware with Calico as documented in this GitHub issue

You need to edit your FelixConfiguration and add this simple line:

spec:
  featureDetectOverride: ChecksumOffloadBroken=true

Wait a couple seconds, and you should be good to go.

Unfortunately Traefik does not support, by default, websockets, which my application uses.
To enable them we have to add a Header to all requests to our backend, adding a customRequestHeaders Middleware.

We do so by specifying a Middleware CRD, or “Custom Resource Definition”, which we then recall from a Traefik IngressRoute

This is not supported with traditional Kubernetes Ingress specifications, so we can scrap all of that hard work!

First, we need to tell Kuberneted what an IngressRoute is

kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v2.9/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml

And now we can create our Middleware!

# CRD to allow wss protocol through
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: metamonitor-pass-wss
spec:
  headers:
    customRequestHeaders:
      X-Forwarded-Proto: "https,ws"

Which we can recall from our IngressRoute

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: metamonitor-web-ingressroute
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`metamonitor.k8s`) && PathPrefix(`/`)
    kind: Rule
    services:
    - name: metamonitor-web
      port: 80

HTTPS

This is fine and everything, but so far everything is in plaintext.
I don’t think traffic inside the cluster will be much of a security issue (for now), so we’ll concentrate on securing traffic from clients to Traefik.

Cert-manager

Certificates have to be issued, and Kubernetes’ native way to do so is centralising everything to Cert-manager, having it create a Secret with key and signed certificate inside, and then importing those in your application either using the Kubernetes native API or passing them to your Deployments as files from secrets.

On microk8s it can be enabled with microk8s enable cert-manager

Loading the certificate

There are many ways to configure cert-manager, this time I’ll configure it as an internal CA, for which we will need to generate an appropriate certificate.
Since I am not skilled in the black arts of openssl, I’ll be laming it with XCA

Generate a new self-signed certificate using the “CA” model, create a private key in the next tab and give it a name.

Export the certificate and the key to PEM format, then convert them to base64

cat kuca.crt | base64 -w0
cat kuca.key | base64 -w0

Put the relative outputs in this Secret definition.
Make sure to use the same namespace from now on, as everything will be namespace-specific, thus, I’ll be putting everything inside the traefik namespace so traefik can access it

apiVersion: v1
kind: Secret
metadata:
  name: kuca-key-pair
  namespace: traefik
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tC....
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktL....

Now that we’ve loaded our certificate and key, we need to tell cert-manager it can use them to issue a certificate

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: kuca-issuer
  namespace: traefik
spec:
  ca:
    secretName: kuca-key-pair

Request a certificate

When requesting a certificate, cert-manager will create a Secret containing our key, our certificate, and the CA’s certificate.
The Certificate object, thus, needs to contain the domain names we need this certificate for, the name of the secret where to save the generated data, and whom we want the certificate issued by

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: metamonitor.k8s
  namespace: traefik
spec:
  dnsNames:
    - metamonitor.k8s
  secretName: metamonitor.k8s
  issuerRef:
    name: kuca-issuer
    kind: Issuer

Wait a couple moments for the magic to happen and…

$ microk8s kubectl -n traefik describe secrets metamonitor.k8s
Name:         metamonitor.k8s
Namespace:    traefik
Labels:       <none>
Annotations:  cert-manager.io/alt-names: metamonitor.k8s
              cert-manager.io/certificate-name: metamonitor.k8s
              cert-manager.io/common-name: 
              cert-manager.io/ip-sans: 
              cert-manager.io/issuer-group: 
              cert-manager.io/issuer-kind: Issuer
              cert-manager.io/issuer-name: kuca-issuer
              cert-manager.io/uri-sans: 

Type:  kubernetes.io/tls

Data
====
tls.crt:  1541 bytes
tls.key:  1675 bytes
ca.crt:   2021 bytes

Signed!

Cert-manager will make sure to keep this certificate valid and will refresh it automatically when the renewal time comes

$ microk8s kubectl -n traefik describe certificate metamonitor.k8s 
Name:         metamonitor.k8s
Namespace:    traefik
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2023-01-07T10:50:01Z
  Generation:          1
  Resource Version:  485968
  UID:               3c3e3395-58c9-49af-8c4c-5a7afbb7200f
Spec:
  Dns Names:
    metamonitor.k8s
  Issuer Ref:
    Kind:       Issuer
    Name:       kuca-issuer
  Secret Name:  metamonitor.k8s
Status:
  Conditions:
    Last Transition Time:  2023-01-07T10:50:01Z
    Message:               Certificate is up to date and has not expired
    Observed Generation:   1
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2023-04-07T10:50:01Z
  Not Before:              2023-01-07T10:50:01Z
  Renewal Time:            2023-03-08T10:50:01Z
  Revision:                1
Events:                    <none>

Make traefik use it

We can go back to our IngressRoute and change it to use https (entrypoint websecure) and adding the tls.secretName property

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: metamonitor-web-ingressroute
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(`metamonitor.k8s`) && PathPrefix(`/`)
    kind: Rule
    services:
    - name: metamonitor-web
      port: 443
  tls:
    secretName: metamonitor.k8s

annnnnndddd

Time to let it out.

Kubernetes keeps applications inside of itself by default, there are a few different ways to expose them.

LoadBalancer

Load Balancers expose applications with their own IP.
Unfortunately most loadbalancers are external services that run in the cloud, and it seems like “on-prem” Kubernetes clusters are kinda B-class citizens.

Luckily, we have MetalLb

When installed as a microk8s addon, it prompts to give it one or more address ranges to assign IPs from.

I gave it the range of 192.168.1.50-192.168.1.70, and the install script did the rest.

Let’s start by exposing the dashboard on its own address.

apiVersion: v1
metadata:
  name: kube-dash-lb
  namespace: kube-system
kind: Service
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard
  type: LoadBalancer

Apply with kubectl apply -f <file>, then check the service status with kubectl -n kube-system get services

NAME         TYPE         CLUSTER-IP    EXTERNAL-IP  PORT(S)       AGE
kube-dash-lb LoadBalancer 10.152.183.63 192.168.1.52 443:30114/TCP 10s

Now our dashboard is exposed on https://192.168.1.52

This is nice and everything, but what if we don’t have as many addresses as the services we want to expose? Afterall, virtual hosts have been a thing since at least 1999

Ingress

An API object that manages external access to the services in a cluster, typically HTTP.
Ingress may provide load balancing, SSL termination and name-based virtual hosting.

Kubernetes documentation

Unfortunately Kubernetes does not have a native, standard Ingress, so we’re going to need an:

Ingress controller

In this case we’re going to use Træfik Proxy, because I have previous experience with it on Docker. There’s a number of other controllers that you can use instead of the one I’m going with.

Installation is just another Helm chart, for a simple matter of keeping things organised we’ll deploy it in its own namespace

microk8s kubectl create namespace traefik
microk8s helm repo add traefik https://traefik.github.io/charts
microk8s helm install -n traefik traefik traefik/traefik

Check that it’s assigned an IP address from the LoadBalancer

NAME      TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)        AGE
traefik   LoadBalancer   10.152.183.53   192.168.1.53   443:32020/TCP  13m

And then let’s re-deploy our Helm chart after setting

ingress:
  enabled: true
  hosts:
    - host: metamonitor.k8s
      paths:
        - path: /
          pathType: ImplementationSpecific

Moments later, Kubernetes is happy

And so is Traefik, on the dashboard

After adding “metamonitor.k8s” to our DNS server (or adding it to localhost file) we can finally visit the page

To recap:

• a Container runs a process
• a Pod is a set of Containers
• managed by a ReplicaSet
• spawned by a Deployment

• a Service matches a certain subset of Pods and creates a hostname inside the cluster
• while an Ingress is an external-facing service-equivalent served by an Ingress Controller
• an Ingress Controller is comprised of one or more Pods interacting with the Kubernetes API to discover Ingresses
• they each expose one or more Services
• which are assigned an IP by a Load Balancer

Makes total sense, right?
…right?

helm create chart

that was easy.
Oh , right, we have to actually _do_ something with it.

As always, all code will be available in this repo.

Helm Charts start deploying an nginx container. That’s fine so far, as we still don’t have an image for our app – that will come later.

For now I want to focus on installing dependencies for my application – Postgres, Redis, InfluxDB.
Make sure you’ve installed the Bitnami repo, then search for your desired package

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm search repo bitnami/postgres
NAME                    CHART VERSION   APP VERSION     DESCRIPTION                                       
bitnami/postgresql      12.1.6          15.1.0          PostgreSQL 
bitnami/postgresql-ha   10.0.7          15.1.0          This PostgreSQL

We’re going to add postgresql. Add the following code to Chart.yaml

dependencies:
  - name: postgresql
    version: '12.1.6'
    repository: https://charts.bitnami.com/bitnami

Rinse and repeat for the other dependencies:

dependencies:
  - name: postgresql
    version: '12.1.6'
    repository: https://charts.bitnami.com/bitnami
  - name: redis
    version: '17.4.0'
    repository: https://charts.bitnami.com/bitnami
  - name: influxdb
    version: '5.4.14'
    repository: https://charts.bitnami.com/bitnami

Finally, run helm dependency update to download helm charts locally.
Let’s try it!

$ microk8s helm -n metamonitor install metamonitor .

Except that now we have a Redis master and THREE replicas! A little overkill for what I need.

Editing configuration

These charts are nice and everything but they are a little overkill for what we need. Luckily, we can customise them.
Just like we would customise them in the “values.yaml” file if we were dealing with them directly, so we can do here, adding a section named after the repo in the “values.yaml” file

redis:
  architecture: standalone

Since this is just the beginning, uninstall, install, and we’re done!

Back to my application

Until now we’ve been preparing all the side-dishes, but what about the main course?
Just need to fill out some data in Chart.yaml. For this, I’ll be using the local repository in k8s, “localhost:32000” which can be enabled with microk8s enable registry

Configuration is quite straightforward, just need to fill out the values:

image:
  repository: localhost:32000/metamonitor
  # Overrides the image tag whose default is the chart appVersion.
  tag: "latest"

service:
  type: ClusterIP
  port: 8000

Environment variables

My application is configured through about 30 environment variables which I now pass through a Docker env file.

Kubernetes of course supports passing environment variables but we can’t do it via a file.
We do it through Config Maps

Add the following variables to the values.yaml file

metamonitor:
  allowed_hosts:
  csrf:
    trusted_origins:
  django:
    debug:
    secret_key:
  ldap:
    bind:
      dn:
      password:
    group:
      base_search:
      filter:
      is_active:
      is_staff:
      is_superuser:
    server_uri:
    ssl_ignore_certificate:
    start_tls:
    user:
      query:
      search:

And then, add a configmap.yaml file in the templates/ folder, as such

kind: ConfigMap
apiVersion: v1
metadata:
  name: {{ include "chart.fullname" . }}-config
  labels: {{- include "common.labels.standard" . | nindent 4 }}
    {{- if .Values.commonLabels }}
    {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }}
    {{- end }}
  {{- if .Values.commonAnnotations }}
  annotations: {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }}
  {{- end }}
data:
  {{- if .Values.metamonitor.allowed_hosts }}
  ALLOWED_HOSTS: localhost,{{ .Values.metamonitor.allowed_hosts }}
  {{- else }}
  ALLOWED_HOSTS: "*"
  {{- end }}
  AUTH_LDAP_BIND_DN: {{ .Values.metamonitor.ldap.bind.dn }}
  AUTH_LDAP_BIND_PASSWORD: {{ .Values.metamonitor.ldap.bind.password }}
  AUTH_LDAP_GROUP_BASE_SEARCH: {{ .Values.metamonitor.ldap.group.base_search }}
  AUTH_LDAP_GROUP_FILTER: {{ .Values.metamonitor.ldap.group.filter }}
  AUTH_LDAP_GROUP_IS_ACTIVE: {{ .Values.metamonitor.ldap.group.is_active }}
  AUTH_LDAP_GROUP_IS_STAFF: {{ .Values.metamonitor.ldap.group.is_staff }}
  AUTH_LDAP_GROUP_IS_SUPERUSER: {{ .Values.metamonitor.ldap.group.is_superuser }}
  AUTH_LDAP_SERVER_URI: {{ .Values.metamonitor.ldap.server_uri }}
  AUTH_LDAP_SSL_IGNORE_CERTIFICATE: {{ .Values.metamonitor.ldap.ssl_ignore_certificate }}
  AUTH_LDAP_START_TLS: {{ .Values.metamonitor.ldap.start_tls }}
  AUTH_LDAP_USER_QUERY: {{ .Values.metamonitor.ldap.user.query }}
  AUTH_LDAP_USER_SEARCH: {{ .Values.metamonitor.ldap.user.search }}
  {{- if .Values.metamonitor.csrf.trusted_origins }}
  CSRF_TRUSTED_ORIGINS: {{ .Values.metamonitor.csrf.trusted_origins }}
  {{- end }}
  DB_ENGINE: django.db.backends.postgresql_psycopg2
  DB_HOST: {{ .Release.Name }}-postgresql-hl
  DB_NAME: {{ .Values.postgresql.auth.database }}
  DB_USER: {{ .Values.postgresql.auth.username }}
  DJANGO_DEBUG: {{ .Values.metamonitor.django.debug }}
  INFLUXDB_BUCKET: {{ .Values.influxdb.auth.admin.bucket }}
  INFLUXDB_ORG: {{ .Values.influxdb.auth.admin.org }}
  INFLUXDB_URL: http://{{ .Release.Name }}-influxdb:8086
  STATIC_ROOT: /metamonitor/static/

Ignoring the upper part of the template, let’s checl the main features of this template:

• Most variables are just taken from values.yaml. You do so by calling them like so: {{.Values.metamonitor.ldap.bind.dn}}
• Other variables are built from variables generated by the subcharts, like postgresql and influxdb.
  Since the postgres chart will always generate a Service called <name>-postgresql-hl, where “name” is the value we pass to the install command ("helm install <name> .“) we can just set the DB hostname as so:
  DB_HOST: {{ .Release.Name }}-postgresql-hl, and same for Influxdb

Secrets

Django needs a “secret key” to generate cookies from.
I want to be able to pass it via config, or to have it autogenerated at deploy time.
So I’ve added templates/secrets.yaml

apiVersion: v1
kind: Secret
metadata:
  name: {{ include "common.names.fullname" . }}-web
type: Opaque
data:
  DJANGO_SECRET_KEY: {{ include "common.secrets.passwords.manage" (dict "secret" (include "common.names.fullname" .) "key" "DJANGO_SECRET_KEY" "providedValues" (list "metamonitor.django.secret_key") "length" 50 "strong" true "context" $) }}

This uses Bitnami’s “common” helm chart providing a function to fetch data from either “metamonitor.django.secret_key”, if available, or by an existing Secret inside Kubernetes ((dict "secret" (include "common.names.fullname" .) "key" "DJANGO_SECRET_KEY").
Should neither of these be available, a default value will be generated and stored ("length" 50 "strong"

Now we need to pass all of these variables to our container, including the postgres password and the influxdb token, which are stored in the respective secrets:

containers:
- name: {{ .Chart.Name }}
  # more data
  envFrom:
  - configMapRef:
      name: {{ .Release.Name }}-web-config
  env:
  - name: DJANGO_SECRET_KEY
    valueFrom:
      secretKeyRef:
        name: {{ include "common.names.fullname" . }}-web
        key: DJANGO_SECRET_KEY
        optional: false # same as default; "mysecret" must exist
                        # and include a key named "password"
  - name: INFLUXDB_TOKEN
    valueFrom:
      secretKeyRef:
        name: {{ .Release.Name }}-influxdb
        key: admin-user-token
        optional: false # same as default; "mysecret" must exist
                        # and include a key named "password"
  - name: DB_PASSWORD
    valueFrom:
        secretKeyRef:
          name: {{ .Release.Name }}-postgresql
          key: password

helm install metamonitor .

Fingers crossed… give it time….

Hooray!

Let’s try a port-forward with kubectl port-forward <pod-name> 8000:8000 andddddd

I can’t show you any further than the login screen, but next time we’ll look into Ingress Controllers to expose this application through a hostname

As mentioned in part 1, my application has not one, but three backends!

• Postgres for relational data
• Influxdb for performance and time-series data
• Redis to pass data in queues between Celery workers

Neither of these applications is “stateless”, so we can’t treat them like cattle and scale them up or down as needed

Stateful Sets

Kubernetes introduces the idea of Stateful Set for services that need to be in a defined state with defined names.

Let’s create a Stateful Set for postgres, defining a few labels and a PersistentVolumeClaim to reserve up to 20Gb for the database data:

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  labels:
    app.kubernetes.io/name: postgres
    app.kubernetes.io/component: postgres
    app.kubernetes.io/part-of: metamonitor
spec:
  selector:
    matchLabels:
      app: metamonitor # has to match .spec.template.metadata.labels
      component: postgres
  serviceName: "postgres"
  replicas: 1 # by default is 1
  minReadySeconds: 10 # by default is 0
  template:
    metadata:
      labels:
        app: metamonitor
        component: postgres
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: postgres
        image: docker.io/postgres:15
        ports:
        - containerPort: 5432
          name: sql
        volumeMounts:
        - name: postgres-data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: postgres-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi

We can load this file either via web or with microk8s kubectl apply -f filename.yaml and we’ll see that a Stateful Set is created, which in turn creates a Pod with a Postgres container in it.
But it will fail, as we haven’t passed any information such as password or DB name

The Official Postgres image says we have to pass three environment variables: POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB

Let’s do it, then.

template:
  spec:
    containers:
    - name: postgres
      env:
        - name: POSTGRES_USER
          value: "antani"
        - name: POSTGRES_PASSWORD
          value: "antanipass"
        - name: POSTGRES_DB
          value: "antanidb"

Save, apply

LOG:  database system is ready to accept connections

Hooray!
But now our password is in plain sight in the config file. That’s bad.

Secrets

Kubernetes has a way to store data (although the default setting is quite unsafe) in a separate object, which can be recalled from many other places.
For example, we will create a Secret containing Postgres’ username, password and DB, which we will recall from both Postgres’ config, and Django’s.

microk8s kubectl create secret generic mm-database \
  --from-literal=username=antani \
  --from-literal=password=antanipass \
  --from-literal=db=antanidb

Now we can replace the “env” section in our yaml file to recall this Secret, destroy the ReplicaSet and recreate.

Reachability

Postgres is running, but there is no “postgres” service running. There’s a “postgres” Service Set, but that generates a “postgres-0” pod.

So we need to set up a Service pointing to port 5432, matching all Pods with the tags we have set above, “app: metamonitor” and “service: postgres“

---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: metamonitor
  labels:
    app.kubernetes.io/name: postgres
    app.kubernetes.io/component: postgres
    app.kubernetes.io/part-of: metamonitor
spec:
  type: ClusterIP
  selector:
    app: metamonitor
    component: postgres
  ports:
    - port: 5432
      targetPort: sql

Now, “just” do the same for Redis, and then set up “alive” checks to make sure we’re not routing traffic to dead pods, and then…

Helm Charts

Or maybe not.

That was quite long and boring, wasn’t it?
Feels like installing by hand every single software and setting up initscripts (or systemd services, for you modern penguin hathchlings).
We could all use a package manager. Well, there’s one.

Helm charts are just that, curated (some more, some less) installation scripts with advanced configuration. For example, Bitnami’s Postgres helm chart can even set up replication on the fly.

All you gotta do is literally

microk8s helm repo add bitnami https://charts.bitnami.com/bitnami
microk8s helm install -n metamonitor postgres bitnami/postgres

This will take care of setting everything up, generating random passwords, saving them in secrets…

In case you want to pass values to the install, either do so with flags or for a more revision-control-friendly way download the “values.yaml” file and pass it with “-f values.yaml”.

Now do the same for Redis and InfluxDB.
Much easier, right?

Since our Managed Services team has a lot of customers, each with their own monitoring system, an ex-colleague had built (and I’ve since completely rewritten and expanded) a Django application against which monitoring systems fire webhooks, centralizing event-management.

The architecture

Metamonitor already runs under docker-compose, and is built as such:

version: '3'

services:
  db:
    image: postgres

  web:
    image: metamonitor:latest

  nginx:
    image: nginx:alpine

  migration:
    image: metamonitor:latest
    command: bash -c "cd /code && python manage.py collectstatic --noinput && python manage.py migrate --noinput"

  celery:
    image: metamonitor:latest
    command: celery -A metasedano.celery worker --loglevel=info

  celerybeat:
    image: metamonitor:latest
    command: celery -A metasedano.celery beat --loglevel=info

  grafana:
    image: grafana/grafana:main

  traefik:
    image: traefik

  influxdb:
    image: influxdb:2.5-alpine

  telegraf:
    image: telegraf:1.23-alpine

  redis:
    image: redis:7
    restart: unless-stopped

• The core is built out of a Django app (web) with Postgres (db) backend. The migration container runs Django database migrations and collects static files to serve through nginx
• Background async tasks and scheduled tasks are run by the celery and celerybeat containers, which talk among themselves and with the main app through redis
• influxdb for long-term statistics, both of application performance and ticketing metrics
• grafana for nice dashboards on the NOC ledwall (because a NOC without giant monitors is just an open space)
• træfix to route it all

Installing Kubernetes

For this test, we’re going to use Canonical’s Microk8s. Yes, in a previous post I said I was going to use Red Hat-based stuff. Yes I changed my mind. This is a lab afterall.

To install Microk8s you can follow the installation guide or just copy-paste the following commands:

sudo snap install --classic microk8s
microk8s start

This will take a while on the first run as it has to download some data off the internet

Check everything is fine with the microk8s status command, then let’s install some extra components with this command:

microk8s enable dashboard dns hostpath-storage registry

Now we need to expose the web interface.
This is where it gets tricky, as for now it’s an internal service and we need to expose it on our host machine, consider this like doing an old-fashioned “port forward”, just like the command says

microk8s kubectl port-forward -n kube-system service/kubernetes-dashboard 10443:443

Leave the terminal open, go to https://localhost:10443 and accept the certificate

We’re going to need a token to log in.
This is tricky again, but bear with me.

..get it? bearer token?
ok, fiiiine

microk8s kubectl describe secret -n kube-system microk8s-dashboard-token

Copy the “token” part (yes, all >900 characters of it) and paste it in the token field

Hello Kubernetes!

It’s tradition that when we play with a new technology we first have it say Hello World.
In Docker, it’s docker run --rm hello-world, what’s the Kubernetes equivalent?

microk8s kubectl run -i -t hello-world --image=docker.io/hello-world --restart=Never

Hello World!

Now if we check the Pods section in the web dashboard we’ll see a new Pod called “hello-world”

In the Kubernetes world, “Pods” are groups of containers, just like Podman’s “Pods”. For now, our Pod only has one container inside.

Delete the pod, either via web or via cli with microk8s kubectl delete pods hello-world

Something a little more complex?

Hello world is fun, but Kubernetes is built for microservices talking with eachother, we want some networking dammit.
We’re going to run traefik/whoami, a simple web server printing out its hostname.

This time though, we’re going to do it via YAML, which can be either copy-pasted in the web interface or saved in a file and loaded via cli with the command microk8s kubectl apply -f filename.yaml

Create a Pod running whoami:

---
kind: Pod
apiVersion: v1
metadata:
  name: whoami
  labels:
    app: whoami
spec:
  containers:
    - name: whoami
      image: traefik/whoami
      ports:
        - containerPort: 80
          protocol: TCP
  restartPolicy: Always

And check it’s running:

But we can’t reach it yet!
We could port forward, but we’re going to do something better

Services

In a dynamic and constantly-changing environment such as Kubernetes’, the only certainty is chaos.
Our app is only one container now, but what if it was two or more? Running on different hosts? How would we ever find it?

Kuberntes has the concept of services, that is a common service exposed by multiple pods. Let’s create a new one.

---
apiVersion: v1
kind: Service
metadata:
  name: whoami-web
spec:
  type: NodePort
  selector:
    app: whoami
  ports:
    - port: 80
      targetPort: 80

And here it is!
Kubernetes has assigned port 31951 for external reachability. Open a browser, go to localhost:31951 and…

But how does Kubernetes know to send our request to our Pod? Well, we told it.
When creating our Pod we gave it a label “app: whoami”, and when we created our Service we gave it a Selector “app: whoami”

This is reflected in the dashboard, if we open the Service details we can see it’s linked to our Pod

Spice it up

I mentioned we could have more than one Pod running at a time running the same service, so we’re going to do that.

You can try by deleting the Pod and creating a second one using the YAML above and only changing the name tag (not app!), then refreshing the page in your browser. The hostname will match what you’ve written.

But we want Kubernetes to manage how many Pods to create, this is where Deployments come to help

---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: whoami
  labels:
    app: whoami
spec:
  replicas: 5
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: traefik/whoami
          ports:
            - containerPort: 80
              protocol: TCP
      restartPolicy: Always
      terminationGracePeriodSeconds: 30

This will create one Deployment with FIVE Pods, each running the same service and each grouped under our previous Service.

Try to delete a Pod – it will immediately spawn a new one!

Conclusion

We’ve learned what Pods, Deployments and Services are.
There’s still a long way to go, but by now I know that my application will need the following services:

• web=80
• db=5432
• nginx=80
• grafana=3000
• influxdb=8086
• redis=6379

Ever heard of streaming telemetry?
No?
Would you like to?

We no longer have to poll network devices for thousands of data points every few seconds to monitor them, we can just have them report back to us whatever data we want, either on a timer, or when it changes.

Configuration is disarmingly easy.

On your Cisco IOS-XE device you will need to add the following line to enable YANG-based configuration:

netconf-yang

And then to get a report of incoming and outgoing traffic, every 60 seconds, for all interfaces, just add:

telemetry ietf subscription 2
 encoding encode-kvgpb
 filter xpath /interfaces/interface/statistics/rx-kbps
 stream yang-push
 update-policy periodic 6000
 receiver ip address <telegraf_server_ip> 57000 protocol grpc-tcp
telemetry ietf subscription 3
 encoding encode-kvgpb
 filter xpath /interfaces/interface/statistics/tx-kbps
 stream yang-push
 update-policy periodic 6000
 receiver ip address <telegraf_server_ip> 57000 protocol grpc-tcp

Then on your Influxdb web page create a new Telegraf config for “Cisco Model-driven telemetry”, copy the two lines you’re prompted for in a terminal – and you’re done!

No, really. That’s it.
And you know the best part? This is an open standard, so you will be able to use this across all your machines that support YANG-based streaming telemetry.

Where to get monitoring paths

While “/interfaces/interface/statistics/rx-kbps” is definitely less cryptic than SNMP’s 1.3.180.12.67.30… equivalent, one might wonder where you can get those paths from.

Enter Cisco YANG suite.

Install it locally (you need Docker, it won’t work on Podman, I tried) following the guides

Create a new device, create a new repo, and either import data directly from your own machine or from the public Git repo for YANG, like so:

For example, let’s say we want to monitor power supply status.
Go under Explore -> Yang.
We’re looking for something called power so… power?

Yes!

..no.
Alright, this is not an exact science. What about going letter by letter… “po”?

Hey look!

Let’s say we want to get all the juicy data from “poe-switch”.
Well it’s just a matter of getting its xpath…

And putting it in the conf

telemetry ietf subscription 5
 encoding encode-kvgpb
 filter xpath /poe-oper-data/poe-switch
 stream yang-push
 update-policy periodic 100
 receiver ip address 10.164.85.152 57777 protocol grpc-tcp

And after a little while…

This looks _very_ promising. Maybe we could also set it to send data “on-change” instead of every 10 seconds, although even with all this, CPU usage is still below 1%