Time to let it out.

Kubernetes keeps applications inside of itself by default, there are a few different ways to expose them.

LoadBalancer

Load Balancers expose applications with their own IP.
Unfortunately most loadbalancers are external services that run in the cloud, and it seems like “on-prem” Kubernetes clusters are kinda B-class citizens.

Luckily, we have MetalLb

When installed as a microk8s addon, it prompts to give it one or more address ranges to assign IPs from.

I gave it the range of 192.168.1.50-192.168.1.70, and the install script did the rest.

Let’s start by exposing the dashboard on its own address.

apiVersion: v1
metadata:
  name: kube-dash-lb
  namespace: kube-system
kind: Service
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard
  type: LoadBalancer

Apply with kubectl apply -f <file>, then check the service status with kubectl -n kube-system get services

NAME         TYPE         CLUSTER-IP    EXTERNAL-IP  PORT(S)       AGE
kube-dash-lb LoadBalancer 10.152.183.63 192.168.1.52 443:30114/TCP 10s

Now our dashboard is exposed on https://192.168.1.52

This is nice and everything, but what if we don’t have as many addresses as the services we want to expose? Afterall, virtual hosts have been a thing since at least 1999

Ingress

An API object that manages external access to the services in a cluster, typically HTTP.
Ingress may provide load balancing, SSL termination and name-based virtual hosting.

Kubernetes documentation

Unfortunately Kubernetes does not have a native, standard Ingress, so we’re going to need an:

Ingress controller

In this case we’re going to use Træfik Proxy, because I have previous experience with it on Docker. There’s a number of other controllers that you can use instead of the one I’m going with.

Installation is just another Helm chart, for a simple matter of keeping things organised we’ll deploy it in its own namespace

microk8s kubectl create namespace traefik
microk8s helm repo add traefik https://traefik.github.io/charts
microk8s helm install -n traefik traefik traefik/traefik

Check that it’s assigned an IP address from the LoadBalancer

NAME      TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)        AGE
traefik   LoadBalancer   10.152.183.53   192.168.1.53   443:32020/TCP  13m

And then let’s re-deploy our Helm chart after setting

ingress:
  enabled: true
  hosts:
    - host: metamonitor.k8s
      paths:
        - path: /
          pathType: ImplementationSpecific

Moments later, Kubernetes is happy

And so is Traefik, on the dashboard

After adding “metamonitor.k8s” to our DNS server (or adding it to localhost file) we can finally visit the page

To recap:

• a Container runs a process
• a Pod is a set of Containers
• managed by a ReplicaSet
• spawned by a Deployment

• a Service matches a certain subset of Pods and creates a hostname inside the cluster
• while an Ingress is an external-facing service-equivalent served by an Ingress Controller
• an Ingress Controller is comprised of one or more Pods interacting with the Kubernetes API to discover Ingresses
• they each expose one or more Services
• which are assigned an IP by a Load Balancer

Makes total sense, right?
…right?

As mentioned in part 1, my application has not one, but three backends!

• Postgres for relational data
• Influxdb for performance and time-series data
• Redis to pass data in queues between Celery workers

Neither of these applications is “stateless”, so we can’t treat them like cattle and scale them up or down as needed

Stateful Sets

Kubernetes introduces the idea of Stateful Set for services that need to be in a defined state with defined names.

Let’s create a Stateful Set for postgres, defining a few labels and a PersistentVolumeClaim to reserve up to 20Gb for the database data:

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  labels:
    app.kubernetes.io/name: postgres
    app.kubernetes.io/component: postgres
    app.kubernetes.io/part-of: metamonitor
spec:
  selector:
    matchLabels:
      app: metamonitor # has to match .spec.template.metadata.labels
      component: postgres
  serviceName: "postgres"
  replicas: 1 # by default is 1
  minReadySeconds: 10 # by default is 0
  template:
    metadata:
      labels:
        app: metamonitor
        component: postgres
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: postgres
        image: docker.io/postgres:15
        ports:
        - containerPort: 5432
          name: sql
        volumeMounts:
        - name: postgres-data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: postgres-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi

We can load this file either via web or with microk8s kubectl apply -f filename.yaml and we’ll see that a Stateful Set is created, which in turn creates a Pod with a Postgres container in it.
But it will fail, as we haven’t passed any information such as password or DB name

The Official Postgres image says we have to pass three environment variables: POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB

Let’s do it, then.

template:
  spec:
    containers:
    - name: postgres
      env:
        - name: POSTGRES_USER
          value: "antani"
        - name: POSTGRES_PASSWORD
          value: "antanipass"
        - name: POSTGRES_DB
          value: "antanidb"

Save, apply

LOG:  database system is ready to accept connections

Hooray!
But now our password is in plain sight in the config file. That’s bad.

Secrets

Kubernetes has a way to store data (although the default setting is quite unsafe) in a separate object, which can be recalled from many other places.
For example, we will create a Secret containing Postgres’ username, password and DB, which we will recall from both Postgres’ config, and Django’s.

microk8s kubectl create secret generic mm-database \
  --from-literal=username=antani \
  --from-literal=password=antanipass \
  --from-literal=db=antanidb

Now we can replace the “env” section in our yaml file to recall this Secret, destroy the ReplicaSet and recreate.

Reachability

Postgres is running, but there is no “postgres” service running. There’s a “postgres” Service Set, but that generates a “postgres-0” pod.

So we need to set up a Service pointing to port 5432, matching all Pods with the tags we have set above, “app: metamonitor” and “service: postgres“

---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: metamonitor
  labels:
    app.kubernetes.io/name: postgres
    app.kubernetes.io/component: postgres
    app.kubernetes.io/part-of: metamonitor
spec:
  type: ClusterIP
  selector:
    app: metamonitor
    component: postgres
  ports:
    - port: 5432
      targetPort: sql

Now, “just” do the same for Redis, and then set up “alive” checks to make sure we’re not routing traffic to dead pods, and then…

Helm Charts

Or maybe not.

That was quite long and boring, wasn’t it?
Feels like installing by hand every single software and setting up initscripts (or systemd services, for you modern penguin hathchlings).
We could all use a package manager. Well, there’s one.

Helm charts are just that, curated (some more, some less) installation scripts with advanced configuration. For example, Bitnami’s Postgres helm chart can even set up replication on the fly.

All you gotta do is literally

microk8s helm repo add bitnami https://charts.bitnami.com/bitnami
microk8s helm install -n metamonitor postgres bitnami/postgres

This will take care of setting everything up, generating random passwords, saving them in secrets…

In case you want to pass values to the install, either do so with flags or for a more revision-control-friendly way download the “values.yaml” file and pass it with “-f values.yaml”.

Now do the same for Redis and InfluxDB.
Much easier, right?

Ever heard of streaming telemetry?
No?
Would you like to?

We no longer have to poll network devices for thousands of data points every few seconds to monitor them, we can just have them report back to us whatever data we want, either on a timer, or when it changes.

Configuration is disarmingly easy.

On your Cisco IOS-XE device you will need to add the following line to enable YANG-based configuration:

netconf-yang

And then to get a report of incoming and outgoing traffic, every 60 seconds, for all interfaces, just add:

telemetry ietf subscription 2
 encoding encode-kvgpb
 filter xpath /interfaces/interface/statistics/rx-kbps
 stream yang-push
 update-policy periodic 6000
 receiver ip address <telegraf_server_ip> 57000 protocol grpc-tcp
telemetry ietf subscription 3
 encoding encode-kvgpb
 filter xpath /interfaces/interface/statistics/tx-kbps
 stream yang-push
 update-policy periodic 6000
 receiver ip address <telegraf_server_ip> 57000 protocol grpc-tcp

Then on your Influxdb web page create a new Telegraf config for “Cisco Model-driven telemetry”, copy the two lines you’re prompted for in a terminal – and you’re done!

No, really. That’s it.
And you know the best part? This is an open standard, so you will be able to use this across all your machines that support YANG-based streaming telemetry.

Where to get monitoring paths

While “/interfaces/interface/statistics/rx-kbps” is definitely less cryptic than SNMP’s 1.3.180.12.67.30… equivalent, one might wonder where you can get those paths from.

Enter Cisco YANG suite.

Install it locally (you need Docker, it won’t work on Podman, I tried) following the guides

Create a new device, create a new repo, and either import data directly from your own machine or from the public Git repo for YANG, like so:

For example, let’s say we want to monitor power supply status.
Go under Explore -> Yang.
We’re looking for something called power so… power?

Yes!

..no.
Alright, this is not an exact science. What about going letter by letter… “po”?

Hey look!

Let’s say we want to get all the juicy data from “poe-switch”.
Well it’s just a matter of getting its xpath…

And putting it in the conf

telemetry ietf subscription 5
 encoding encode-kvgpb
 filter xpath /poe-oper-data/poe-switch
 stream yang-push
 update-policy periodic 100
 receiver ip address 10.164.85.152 57777 protocol grpc-tcp

And after a little while…

This looks _very_ promising. Maybe we could also set it to send data “on-change” instead of every 10 seconds, although even with all this, CPU usage is still below 1%