Virtualisation, Storage and various other ramblings.

Category: Uncategorized (Page 1 of 3)

Writing my first Prometheus exporter and scraping with Rancher

TLDR; Code repo can be found here. Huge thanks to Spencer for the original blog post that helped me, answering some questions over email, and providing a really helpful Prometheus exporter template (in Go) that I used for this example.

The monitoring framework in Rancher 2.5 was significantly changed – including, but not limited to, giving us the ability to scrape for our own application metrics and creating custom Grafana dashboards. The Monitoring stack in Rancher is a culmination of a number of open-source technologies:

  • Prometheus – For collecting and storing metrics into a time-series database.
  • Grafana – Analytics and visualisation of metrics (IE Prometheus).
  • AlertManager – An extension of Prometheus that enabling configuration of alerts and routing them to notification, paging, and automation systems.

Prometheus Exporters

Prometheus-native applications expose their own metrics which can be scraped from an HTTP endpoint. If we want to capture Prometheus metrics from a system that doesn’t natively export them in this format we need Exporters.

Exporters act as an interpretation layer, taking non-Prometheus formatted metrics and exposing them as such.

Test System – VDSL Modem

My VDSL modem (a EchoLife HG612 with unlocked firmware) does expose some metrics about my connection, but not in a format understood by Prometheus. Accessing /html/status/xdslStatus.asp on my modem returns:

var DSLCfg = new Array(new stDsl("InternetGatewayDevice.WANDevice.1.WANDSLInterfaceConfig","Up","VDSL2","","8852","42044","0","0","8852","40780","0","222","62","134","62","134","Unknown Mode"),null); var DSLStats = new Array(new stStats("InternetGatewayDevice.WANDevice.1.WANDSLInterfaceConfig.Stats.Showtime","90","4294967290","238","127","0","0","32","0","18","0","0","0"),null); var DslUpTime = "0"; var time = 0;

This information gives me some info about my connection. After some quick Googling, I found a reference table that defines what each of these fields relates to.

Writing the Exporter

As the aforementioned metrics aren’t in a format Prometheus can understand, I need to write an Exporter. Prometheus expects to scrape from HTTP-based endpoints, so writing one in go is quite trivial. Prometheus has packages you can leverage to help write your own – which I’ve used as part of the HTTP handler.

func main() {
	//Kick off collector in background
	go collector.Collect()

	//This section will start the HTTP server and expose
	//any metrics on the /metrics endpoint.
	http.Handle("/metrics", promhttp.Handler())
	log.Info("Beginning to serve on port :8080")
	log.Fatal(http.ListenAndServe(":8080", nil))
}

I wrote my own page for collector, which also includes Prometheus packages:

package collector

import (
	"github.com/prometheus/client_golang/prometheus"
...
...
)

// Stats get reset after disconnect, hence the use of Gauge Type
type myMetrics struct {
	UpstreamCurrRate      prometheus.Gauge
	DownstreamCurrRate    prometheus.Gauge
	UpstreamCurrRate2     prometheus.Gauge
	DownstreamCurrRate2   prometheus.Gauge
	UpstreamMaxRate       prometheus.Gauge
...
...

For tidiness, I decided to encapsulate all my metrics into a single instance of a myMetrics struct. Alternatively, you could simply store these in individual variables but with the number of metrics this collects, this seemed to be the more appropriate way.

The Prometheus package also exposes certain types, Guage is heavily used as over time, these values could fluctuate or even reset to 0.

The bulk of the Collector package performs the following:

  • Makes an HTTP call to the VDSL modems stats page.
  • Uses Regex to extrapolate the quoted values
  • Calls a helper function to convert these into float – which is what the prometheus.Gauge type expects
//Form Regex to extract all quoted strings
			re := regexp.MustCompile("\"(.*?)\"")
			extractedValues := re.FindAll(bodyBytes, -1)

			modemMetrics.UpstreamCurrRate.Set(convertToFloat(extractedValues[4]))
			modemMetrics.DownstreamCurrRate.Set(convertToFloat(extractedValues[5]))
			modemMetrics.UpstreamCurrRate2.Set(convertToFloat(extractedValues[6]))
			modemMetrics.DownstreamCurrRate2.Set(convertToFloat(extractedValues[7]))
			modemMetrics.UpstreamMaxRate.Set(convertToFloat(extractedValues[8]))
			modemMetrics.DownstreamMaxRate.Set(convertToFloat(extractedValues[9]))
			modemMetrics.UpstreamNoiseMargin.Set(convertToFloat(extractedValues[10]))

Running this code and navigating to http://localhost:8080 shows the metrics:

Packaging in a Container

In the aforementioned repo, a Github CI job kicks off on a push to package this application inside a docker container, which makes it easy to deploy to Kubernetes.

Scraping with Rancher

To scrape additional metrics within Rancher (after installing the Monitoring chart), we can define additional servicemonitor objects to specify what needs to be scraped. In this example, I created a simple deployment object for my Exporter container, inside a Pod, exposed by a Service of type clusterIP

Which we can then inspect the metrics for in Prometheus:

And visualise in Grafana:

Automated deployment of K3s and Rancher on vSphere with Terraform

Previously, my local Rancher installs were based on RKE. However, since K3S is now a supported distribution, I decided to rebuild my environment leveraging it. Additionally, it was a good opportunity to automate the process with Terraform.

TL;DR

https://github.com/David-VTUK/Rancher-K3s-vSphere contains the Terraform code required to do this.

Quick note on K3S with Embedded DB

This installation method is currently experimental. Do not leverage it in production (yet). Towards the end of August 2020, we (Rancher) plan to replace it with embedded etcd as per the roadmap. I’m a fan of simplicity, therefore when v1.19 does come out, I plan to simply tear down and rebuild my cluster using this Terraform code. However, one could equally modify it to leverage an external DB for a more production-ready setup.

Resources Created

The aforementioned Terraform code will create:

  • A single VM with NGINX installed acting as a Loadbalancer, forwarding TCP 80/443/6443 to the K3s Nodes
  • Three VM’s which will form the K3s cluster with an embedded DB. The first of which is used to initialise the cluster
  • Once the cluster is created, Cert-Manager and Rancher are installed which are probed for readiness.
file:///home/david/Downloads/Architecture.png

Prerequisites

  • Terraform version 0.13
  • Prior to running this script, a DNS record needs to be created to point at the Loadbalancer IP address, defined in the variable lb_address.
  • The VM template used must have the Cloud-Init Datasource for VMware GuestInfo project installed, which facilitates pulling meta, user, and vendor data from VMware vSphere’s GuestInfo interface. This can be achieved with:
curl -sSL https://raw.githubusercontent.com/vmware/cloud-init-vmware-guestinfo/master/install.sh | sh -

Or use the following Packer Template:

https://github.com/David-VTUK/Rancher-Packer/tree/master/vSphere/ubuntu_2004_cloud_init_guestinfo

Acquire Kubeconfig

  • SSH to one of the K3s nodes
  • Grab /etc/rancher/k3s/k3s.yaml
  • Replace server: https://127.0.0.1:6443 with the IP address defined in lb_address

K8s, MetalLB and Pihole

An ongoing project of mine involves the migration of home services (Unifi, Pi-hole, etc) to my Kubernetes cluster. This post explores my approach to migrating Pi-hole, with the help of MetalLB.

MetalLB Overview

MetalLB is a load balancer implementation for environments that do not natively provide this functionality. For example, with AWS, Azure, GCP and others, provisioning a “LoadBalancer” service will make API calls to the respective cloud provider to provision a load balancer. For bare-metal / on-premises and similar environments this may not work (depending on the CNI used). MetalLB bridges this functionality to these environments so services can be exposed externally.

 

 

MetalLB consists of the following components:

  • Controller Deployment – A single replica deployment responsible for IP assignment.
  • Speaker DaemonSet – Facilitates communication based on the specified protocols used for external services.
  • Controller and Speaker service accounts – RBAC permissions required for respective components.
  • Configuration ConfigMap – Specifies parameters for either L2 or BGP configuration. The former being used in this example for simplicity.

The Speaker and Controller components can be deployed by applying the MetalLB manifest:

kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.1/manifests/metallb.yaml

A configmap is used to complement the deployment by specifying the required parameters. Below is an example I’ve used.

apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 172.16.10.221-172.16.10.230

The end result is any service of type “LoadBalancer” will be provisioned from the pool of IP addresses in the above configmap.

 

PI-Hole Overview

Pi-Hole is a network-wide adblocker. It’s designed to act as a DNS resolver employing some intelligence to identify and block requests to known ad sites. An advantage of implementing it vs something like Ublock Origin, is PiHole operates at the network level, and is, therefore, device/client agnostic and requires no plugins/software on the originating device.

 

 

The makers of Pi-Hole have an official Dockerhub repo for running Pi-Hole as a container, which makes it easier to run in Kubernetes, but with some caveats, as is described below.

Storing Persistent Data with Pi-Hole

A Pi-Hole container can be fired up with relative ease and provides some effective ad-blocking functionality but if the container is deleted or restarted, any additional configuration post-creation will be lost, it would, therefore, be convenient to have a persistent location for the Pi-Hole configuration, so blocklist / regex entries / etc could be modified. The makers of Pi-Hole have documented the location and use of various configuration files. Of interest are the following:

adlists.list: a custom user-defined list of blocklist URL’s (public blocklists maintained by Pi-Hole users). Located in /etc/pihole

regex.list : file of regex filters that are compiled with each pihole-FTL start or restart. Located in /etc/pihole

 

Approach #1 – Persistent Volumes

This approach leverages a persistent volume mounted to /etc/pihole with a “Retain” policy. This would ensure that if the container terminates, the information in /etc/pihole would be retained. One disadvantage of this includes the operational overhead of implementing and managing Persistent Volumes.

Approach #2 – Config Maps

This approach leverages configmaps mounted directly to the pod, presented as files. Using this method will ensure consistency of configuration parameters without the need to maintain persistent volumes, with the added benefit of residing within the etcd database and is therefore included in etcd backups. This method also completely abstracts the configuration from the pod, which can easily facilitate updates/changes.

 

Implementation

Given the options, I felt #2 was better suited for my environment. YAML manifests can be found in https://github.com/David-VTUK/k8spihole.

 

00-namespace.yaml

Create a namespace for our application. This will be referenced later

apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 172.16.10.221-172.16.10.230

01-configmaps.yaml

This is where our persistent configuration will be stored.

Location for adlists:

apiVersion: v1
kind: ConfigMap
metadata:
name: pihole-adlists
namespace: pihole-test
data:
adlists.list: |
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
......etc

Location for regex values

apiVersion: v1
kind: ConfigMap
metadata:
name: pihole-regex
namespace: pihole-test
data:
regex.list: |
^(.+[-_.])??adse?rv(er?|ice)?s?[0-9]*[-.]
......etc

Setting environment variables for the timezone and upstream DNS servers.

apiVersion: v1
kind: ConfigMap
metadata:
name: pihole-env
namespace: pihole-test
data:
TZ: UTC
DNS1: 1.1.1.1
DNS2: 1.0.0.1

02-deployment.yaml

This manifest defines the parameters of the deployment, of significance are how the config maps are consumed. For example, the environment variables are set from the respective configmap:

containers:
- name: pihole
image: pihole/pihole
env:
- name: TZ
valueFrom:
configMapKeyRef:
name: pihole-env
key: TZ

The files are mounted from the aforementioned configmaps as volumes:

volumeMounts:
- name: pihole-adlists
mountPath: /etc/pihole/adlists.list
subPath: adlists.list
- name: pihole-regex
mountPath: /etc/pihole/regex.list
subPath: regex.list
volumes:
- name: pihole-adlists
configMap:
name: pihole-adlists
- name: pihole-regex
configMap:
name: pihole-regex

03-service.yaml

Currently, you cannot mix UDP and TCP services on the same Kubernetes load balancer, therefore two services are created. One for the DNS queries (UDP 53) and one for the web interface (TCP 80)

kind: Service
apiVersion: v1
metadata:
name: pihole-web-service
namespace : pihole-test
spec:
selector:
app: pihole
ports:
- protocol: TCP
port: 80
targetPort: 80
name : web
type: LoadBalancer
---
kind: Service
apiVersion: v1
metadata:
name: pihole-dns-service
namespace: pihole-test
spec:
selector:
app: pihole
ports:
- protocol: UDP
port: 53
targetPort: 53
name : dns
type: LoadBalancer

Deployment

After configuring the configmaps, the manifests can be deployed:

david@david-desktop:~/pihole$ kubectl apply -f .
namespace/pihole-test created
configmap/pihole-adlists created
configmap/pihole-regex created
configmap/pihole-env created
deployment.apps/pihole-deployment created
service/pihole-web-service created
service/pihole-dns-service created

Extract the password for Pi-Hole from the container:

david@david-desktop:~/pihole$ kubectl get po -n pihole-test
NAME                                 READY   STATUS    RESTARTS   AGE
pihole-deployment-6ffb58fb8f-2mc97   1/1     Running   0          2m24s
david@david-desktop:~/pihole$ kubectl logs pihole-deployment-6ffb58fb8f-2mc97 -n pihole-test | grep random
Assigning random password: j6ddiTdS

Identify the IP address of the web service:

david@david-desktop:~/pihole$ kubectl get svc -n pihole-test
NAME                 TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)        AGE
pihole-dns-service   LoadBalancer   10.100.40.39     172.16.10.226   53:31725/UDP   3m38s
pihole-web-service   LoadBalancer   10.107.251.224   172.16.10.225   80:30735/TCP   3m38s

Access Pi-Hole on the web service external IP using the password extracted from the pod:

All that remains is to reconfigure DHCP or static settings to point to the pihole-dns-service Loadbalancer address for its DNS queries.

I’m quite surprised how much it has blocked thus far (~48 hours of usage):

 

Happy Ad Blocking!

 

« Older posts

© 2021 Virtual Thoughts

Theme by Anders NorenUp ↑

Social media & sharing icons powered by UltimatelySocial
RSS
Twitter
Visit Us
Follow Me