Ever heard of streaming telemetry?
No?
Would you like to?

We no longer have to poll network devices for thousands of data points every few seconds to monitor them, we can just have them report back to us whatever data we want, either on a timer, or when it changes.

Configuration is disarmingly easy.

On your Cisco IOS-XE device you will need to add the following line to enable YANG-based configuration:

netconf-yang

And then to get a report of incoming and outgoing traffic, every 60 seconds, for all interfaces, just add:

telemetry ietf subscription 2
 encoding encode-kvgpb
 filter xpath /interfaces/interface/statistics/rx-kbps
 stream yang-push
 update-policy periodic 6000
 receiver ip address <telegraf_server_ip> 57000 protocol grpc-tcp
telemetry ietf subscription 3
 encoding encode-kvgpb
 filter xpath /interfaces/interface/statistics/tx-kbps
 stream yang-push
 update-policy periodic 6000
 receiver ip address <telegraf_server_ip> 57000 protocol grpc-tcp

Then on your Influxdb web page create a new Telegraf config for “Cisco Model-driven telemetry”, copy the two lines you’re prompted for in a terminal – and you’re done!

No, really. That’s it.
And you know the best part? This is an open standard, so you will be able to use this across all your machines that support YANG-based streaming telemetry.

Where to get monitoring paths

While “/interfaces/interface/statistics/rx-kbps” is definitely less cryptic than SNMP’s 1.3.180.12.67.30… equivalent, one might wonder where you can get those paths from.

Enter Cisco YANG suite.

Install it locally (you need Docker, it won’t work on Podman, I tried) following the guides

Create a new device, create a new repo, and either import data directly from your own machine or from the public Git repo for YANG, like so:

For example, let’s say we want to monitor power supply status.
Go under Explore -> Yang.
We’re looking for something called power so… power?

Yes!

..no.
Alright, this is not an exact science. What about going letter by letter… “po”?

Hey look!

Let’s say we want to get all the juicy data from “poe-switch”.
Well it’s just a matter of getting its xpath…

And putting it in the conf

telemetry ietf subscription 5
 encoding encode-kvgpb
 filter xpath /poe-oper-data/poe-switch
 stream yang-push
 update-policy periodic 100
 receiver ip address 10.164.85.152 57777 protocol grpc-tcp

And after a little while…

This looks _very_ promising. Maybe we could also set it to send data “on-change” instead of every 10 seconds, although even with all this, CPU usage is still below 1%

In this article we will learn to configure Telegraf, influxDB and Grafana to collect and parse json-formatted Nginx logs

What’s a TIG?

TIG stands for Telegraf, InfluxDB, Grafana, and we’re going to see a lot of this in the upcoming posts

Let’s start by creating our pod

podman pod create -n site-analytics -p 8086:8086 -p 3000:3000

As usual, with Podman, we need to expose our ports when we create our pod.
We’ll expose 8086 for InfluxDB and 3000 for Grafana.

Create two containers, one for each of those services

podman run -d --pod site-analytics docker.io/influxdb:2.5-alpine
podman run -d --pod site-analytics -v grafana-data:/var/lib/grafana docker.io/grafana/grafana:latest

Next, we’ll configure influxdb.

Configure Influxdb

Go to http://localhost:8086

Fill in your data, choose a name of your liking for Org Name and Bucket

Click “configure later”.

The wizard already set us up with a bucket (in a normal DB this would be a “table”) set to never expire. It’s fine for now.

Create a new bucket called “blog-logs” and set to expire after 48 hours so we don’t fill up our disk.

Configure Grafana

Create a token in InfuxDB for Grafana. Give it permissions to “read all buckets”

Open grafana in a new tab. http://localhost:3000.
Login with admin/admin and choose a new password.

Click the cog in the lower left corner and configure a new Influxdb data source.

• Query Language: Flux
• URL: http://localhost:8086
• No Basic Auth
• Organization: what you set before
• Token: paste the token from Influxdb
• Save and test

Configure Nginx

We need to tell nginx to save logs in json format.
Open nginx.conf and add this line

log_format json escape=json '{"time": "$msec", "address": "$remote_addr", "method": "$request_method", "uri": "$request_uri", "status": $status, "resp_body_size": $body_bytes_sent, "request_length": $request_length, "user_agent": "$http_user_agent", "resp_time": "$request_time", "referer": "$http_referer"}';

With this, logs will be output in this format

{
   "time": "$msec",
   "address": "$remote_addr",
   "method": "$request_method",
   "uri": "$request_uri",
   "resp_body_size": $body_bytes_sent,
   "status": $status,
   "request_length": $request_length,
   "user_agent": "$http_user_agent",
   "resp_time": "$request_time",
   "referer": "$http_referer"
}

Edit the “access_log” setting, adding “json” as desired log format, like so:

access_log /var/log/nginx/access.log json;

Reload nginx.

Configure TeleGRAF

Telegraf does not have a web interface. We won’t even install it inside a container, we need it running on the host so it can read nginx’s log file.

Install it locally following their guide.
Generate a new token in Influxdb with write permissions for your bucket.

Let’s first configure Telegraf to output data to InfluxDB in /etc/telegraf/telegraf.conf

[global_tags]

# Configuration for telegraf agent
[agent]
  # send data every 10 seconds
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"

  debug = false
  quiet = false

# Configuration for influxdb server to send metrics to
[[outputs.influxdb_v2]]
  urls = ["http://localhost:8086"]
  token = "<token>"
  bucket = "<bucket>"
  organization = "<org>"

Next, we need Telegraf to continuously read nginx’s log file and parse it as json.
In the same file, add:

[[inputs.tail]]
  files = ["/var/log/nginx/access.log"]
  data_format = "json_v2"

  [[inputs.tail.json_v2]]
    name_override = "nginx_access_log"

    [[inputs.tail.json_v2.object]]
      name_override = "nginx_access_request"
      path = "@this"
      timestamp_key = "@this.time"
      timestamp_format = "unix_ms"

Telegraf will read each line as a json “object”. The only thing we need to tell it is where the log timestamp is (“@this.time”) and the format (“unix_ms”)

permission hell

We need to make one more change before we can get our data: nginx’s log files are owned by root and Telegraf can’t read them.

So the first thing to do is assigning group ownership of those files to telegraf’s group, so it can read them

sudo chgrp telegraf /var/log/nginx/access.log

Unfortunately this won’t last long.

On Fedora, nginx’s log files are rotated daily by logrotate who also takes care of creating a new file where Nginx can write to, and the permissions are set in logrotate’s configuration, so changing them on the current file is a fix that would only last up to the next run of logrotate.

Logrotate’s config for nginx at /etc/logrotate.d/nginx looks like this

/var/log/nginx/*.log {
    create 0640 nginx root
    daily
    rotate 10
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        /bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true
    endscript
}

We just need to swap “root” with “telegraf”, save, and that’s it!

Are we done yet?

We can finally restart telegraf. After a couple moments we should see our data coming in our Influxdb dashboard!

Next post we’ll figure out what to do with this data.
Once I’ve collected enough, of course