Virtual Thoughts

Virtualisation, Storage and various other ramblings.

Page 3 of 18

Pi-Hole and K8s v2 – Now with DNS over HTTPS

In a previous post, I went through the process of configuring Pi-Hole within a Kubernetes cluster for the purpose of facilitating a network-wide adblocking. Although helpful, I wanted to augment this with DNS over HTTPS.

Complete manifests can be found here. Shout out to visibilityspots for the cloudflared image on Dockerhub


DNS, as a protocol, is insecure and can be prone to manipulation and man-in-the-middle attacks. DNS over HTTPS helps address this by encrypting the data between the DNS over HTTPS client and the DNS over HTTPS-based DNS resolver. One of which is provided by Cloudflare.

Thankfully, Pi-Hole has some documentation on how to implement this for the traditional Pi-Hole setups. But for Kubernetes-based deployments, this requires a different approach.


The DNS over HTTPS client is facilitated by a Cloudflare daemon. In a traditional Pi-Hole setup this is simply run alongside Pi-Hole itself, but in a containerised environment there are predominantly two ways to address this:

As a separate microservice

This approach leverages two different deployments, one for the Pi-Hole service, and one for cloudflared. While workable, I felt that this was a less optimal approach

Service-To-Service communication between Pi-Hole and Cloudflared

As another container within the Pi-Hole pod

Given the tight relationship between these containers, and the fact their respective services run on different ports, this seems like a more efficient approach.

Intra-pod communication between Pi-Hole and CloudflareD

As the containers share the same network interface, one pod can access the other over either the veth interface, or simply the localhost address. For Pi-Hole, we can facilitate this via a configmap change:

apiVersion: v1
kind: ConfigMap
  name: pihole-env
  namespace: pihole-system


Once the respective manifest files have been deployed and clients are pointing to pi-hole as a DNS resolver, it can be tested by accessing As per the example below, DNS over HTTPS has been identified.

K8s, MetalLB and Pihole

An ongoing project of mine involves the migration of home services (Unifi, Pi-hole, etc) to my Kubernetes cluster. This post explores my approach to migrating Pi-hole, with the help of MetalLB.

MetalLB Overview

MetalLB is a load balancer implementation for environments that do not natively provide this functionality. For example, with AWS, Azure, GCP and others, provisioning a “LoadBalancer” service will make API calls to the respective cloud provider to provision a load balancer. For bare-metal / on-premises and similar environments this may not work (depending on the CNI used). MetalLB bridges this functionality to these environments so services can be exposed externally.



MetalLB consists of the following components:

  • Controller Deployment – A single replica deployment responsible for IP assignment.
  • Speaker DaemonSet – Facilitates communication based on the specified protocols used for external services.
  • Controller and Speaker service accounts – RBAC permissions required for respective components.
  • Configuration ConfigMap – Specifies parameters for either L2 or BGP configuration. The former being used in this example for simplicity.

The Speaker and Controller components can be deployed by applying the MetalLB manifest:

kubectl apply -f

A configmap is used to complement the deployment by specifying the required parameters. Below is an example I’ve used.

apiVersion: v1
kind: ConfigMap
namespace: metallb-system
name: config
config: |
- name: default
protocol: layer2

The end result is any service of type “LoadBalancer” will be provisioned from the pool of IP addresses in the above configmap.


PI-Hole Overview

Pi-Hole is a network-wide adblocker. It’s designed to act as a DNS resolver employing some intelligence to identify and block requests to known ad sites. An advantage of implementing it vs something like Ublock Origin, is PiHole operates at the network level, and is, therefore, device/client agnostic and requires no plugins/software on the originating device.



The makers of Pi-Hole have an official Dockerhub repo for running Pi-Hole as a container, which makes it easier to run in Kubernetes, but with some caveats, as is described below.

Storing Persistent Data with Pi-Hole

A Pi-Hole container can be fired up with relative ease and provides some effective ad-blocking functionality but if the container is deleted or restarted, any additional configuration post-creation will be lost, it would, therefore, be convenient to have a persistent location for the Pi-Hole configuration, so blocklist / regex entries / etc could be modified. The makers of Pi-Hole have documented the location and use of various configuration files. Of interest are the following:

adlists.list: a custom user-defined list of blocklist URL’s (public blocklists maintained by Pi-Hole users). Located in /etc/pihole

regex.list : file of regex filters that are compiled with each pihole-FTL start or restart. Located in /etc/pihole


Approach #1 – Persistent Volumes

This approach leverages a persistent volume mounted to /etc/pihole with a “Retain” policy. This would ensure that if the container terminates, the information in /etc/pihole would be retained. One disadvantage of this includes the operational overhead of implementing and managing Persistent Volumes.

Approach #2 – Config Maps

This approach leverages configmaps mounted directly to the pod, presented as files. Using this method will ensure consistency of configuration parameters without the need to maintain persistent volumes, with the added benefit of residing within the etcd database and is therefore included in etcd backups. This method also completely abstracts the configuration from the pod, which can easily facilitate updates/changes.



Given the options, I felt #2 was better suited for my environment. YAML manifests can be found in



Create a namespace for our application. This will be referenced later

apiVersion: v1
kind: ConfigMap
namespace: metallb-system
name: config
config: |
- name: default
protocol: layer2


This is where our persistent configuration will be stored.

Location for adlists:

apiVersion: v1
kind: ConfigMap
name: pihole-adlists
namespace: pihole-test
adlists.list: |

Location for regex values

apiVersion: v1
kind: ConfigMap
name: pihole-regex
namespace: pihole-test
regex.list: |

Setting environment variables for the timezone and upstream DNS servers.

apiVersion: v1
kind: ConfigMap
name: pihole-env
namespace: pihole-test


This manifest defines the parameters of the deployment, of significance are how the config maps are consumed. For example, the environment variables are set from the respective configmap:

- name: pihole
image: pihole/pihole
- name: TZ
name: pihole-env
key: TZ

The files are mounted from the aforementioned configmaps as volumes:

- name: pihole-adlists
mountPath: /etc/pihole/adlists.list
subPath: adlists.list
- name: pihole-regex
mountPath: /etc/pihole/regex.list
subPath: regex.list
- name: pihole-adlists
name: pihole-adlists
- name: pihole-regex
name: pihole-regex


Currently, you cannot mix UDP and TCP services on the same Kubernetes load balancer, therefore two services are created. One for the DNS queries (UDP 53) and one for the web interface (TCP 80)

kind: Service
apiVersion: v1
name: pihole-web-service
namespace : pihole-test
app: pihole
- protocol: TCP
port: 80
targetPort: 80
name : web
type: LoadBalancer
kind: Service
apiVersion: v1
name: pihole-dns-service
namespace: pihole-test
app: pihole
- protocol: UDP
port: 53
targetPort: 53
name : dns
type: LoadBalancer


After configuring the configmaps, the manifests can be deployed:

david@david-desktop:~/pihole$ kubectl apply -f .
namespace/pihole-test created
configmap/pihole-adlists created
configmap/pihole-regex created
configmap/pihole-env created
deployment.apps/pihole-deployment created
service/pihole-web-service created
service/pihole-dns-service created

Extract the password for Pi-Hole from the container:

david@david-desktop:~/pihole$ kubectl get po -n pihole-test
NAME                                 READY   STATUS    RESTARTS   AGE
pihole-deployment-6ffb58fb8f-2mc97   1/1     Running   0          2m24s
david@david-desktop:~/pihole$ kubectl logs pihole-deployment-6ffb58fb8f-2mc97 -n pihole-test | grep random
Assigning random password: j6ddiTdS

Identify the IP address of the web service:

david@david-desktop:~/pihole$ kubectl get svc -n pihole-test
NAME                 TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)        AGE
pihole-dns-service   LoadBalancer   53:31725/UDP   3m38s
pihole-web-service   LoadBalancer   80:30735/TCP   3m38s

Access Pi-Hole on the web service external IP using the password extracted from the pod:

All that remains is to reconfigure DHCP or static settings to point to the pihole-dns-service Loadbalancer address for its DNS queries.

I’m quite surprised how much it has blocked thus far (~48 hours of usage):


Happy Ad Blocking!


Creating a highly available, cross AZ, loadbalanced ETCD cluster in AWS with Terraform

Having experimented with Terraform recently, I decided to leverage this tool by creating an etcd cluster in AWS. This blog post goes through the steps used to accomplish this. For readability, I’ve only quoted pertinent code snippets, but all of the code can be found at


I do not profess to be an Etcd, Terraform or AWS expert, therefore he be dragons in the form of implementations unlikely to be best practice or production-ready. In particular, I would like to revisit this at some point and enhance it to include:

  • Securing all Etcd communication with generated certs.
  • Implement a mechanism for rotating members in/out of the cluster.
  • Leverage auto-scaling groups for both bastion and Etcd members.
  • Locking down the security groups.
  • ….and a  lot more.



The basic principles of this implementation are as follows:

  • Etcd members will reside in private subnets and accessed via an internal load-balancer.
  • Etcd members will be joined to a cluster by leveraging the Etcd discovery URL.
  • A bastion host will be used to proxy all access to the private subnets.
  • The Terraform script creates all the components, at the VPC level and beyond.

The scripts are separated into the following files:

├── deployetcd.tpl
├── terraformec2.pem
├── terraform.tfstate
└── terraform.tfstate.backup


This is a template file for Terraform, it’s used to generate a bootstrap script to run on each of our Etcd nodes.

To begin with, download Etcd, extract and do some general housekeeping with regards to access and users. It will also output two environment variables – ETCD_HOST_IP and ETCD_NAME, which is needed for the Systemd unit file.

cd /usr/local/src
sudo wget ""
sudo tar -xvf etcd-v3.3.9-linux-amd64.tar.gz
sudo mv etcd-v3.3.9-linux-amd64/etcd* /usr/local/bin/
sudo mkdir -p /etc/etcd /var/lib/etcd
sudo groupadd -f -g 1501 etcd
sudo useradd -c "etcd user" -d /var/lib/etcd -s /bin/false -g etcd -u 1501 etcd
sudo chown -R etcd:etcd /var/lib/etcd

export ETCD_HOST_IP=$(ip addr show eth0 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)
export ETCD_NAME=$(hostname -s)

Create the Systemd unit file

sudo -E bash -c 'cat << EOF > /lib/systemd/system/etcd.service
Description=etcd service

ExecStart=/usr/local/bin/etcd \\
 --data-dir /var/lib/etcd \\
 --discovery ${discoveryURL} \\
 --initial-advertise-peer-urls http://$ETCD_HOST_IP:2380 \\
 --name $ETCD_NAME \\
 --listen-peer-urls http://$ETCD_HOST_IP:2380 \\
 --listen-client-urls http://$ETCD_HOST_IP:2379, \\
 --advertise-client-urls http://$ETCD_HOST_IP:2379 \\


${discoveryURL} is a placeholder variable. The Terraform script will replace this and generate the script file with a value for this variable.

This is where the discovery token is generated, parsed into the template file to which a new file will be generated – our complete bootstrap script. Simply performing an HTTP get request to the discovery URL will return a unique identifier URL in the response body. So we grab this response and store it as a data object. Note “size=3” denotes the explicit size of the Etcd cluster.

data "http" "etcd-join" {
  url = ""

Take this data and parse it into the template file, filling it the variable ${discoveryURL} with the unique URL generated.

data "template_file" "service_template" {
  template = "${file("./deployetcd.tpl")}"

  vars = {
    discoveryURL = "${data.http.etcd-join.body}"

Take the generated output, and store it as “” in the local directory

resource "local_file" "template" {
  content  = "${data.template_file.service_template.rendered}"
  filename = ""

The purpose of this code block is to generate one script to be executed on all three Etcd members, each joining a specific 3 node etcd cluster.

This script performs the following:

  • Create a bastion host in the public subnet with a public IP.
  • Create three etcd instanced in each AZ, and execute the generated script via the bastion host:
    connection {
    host = "${aws_instance.etc1.private_ip}"
    port = "22"
    type = "ssh"
    user = "ubuntu"
    private_key = "${file("./terraformec2.pem")}"
    timeout = "2m"
    agent = false

    bastion_host = "${aws_instance.bastion.public_ip}"
    bastion_port = "22"
    bastion_user = "ec2-user"
    bastion_private_key = "${file("./terraformec2.pem")}"

  provisioner "file" {
    source      = ""
    destination = "/tmp/"

  provisioner "remote-exec" {
    inline = [
      "chmod +x /tmp/",

  • Contains access and secret keys for AWS access

Configures the following:

  • Elastic IP for NAT gateway.
  • NAT gateway so private subnets can route out to pull etcd.
  • Three subnets in the EU-West-2 region
    • EU-West-2a
    • EU-West-2b
    • EU-West-2c
  • An ALB that spans across the aforementioned AZ’s:
    • Create a target group.
    • Create a listener on Etcd port.
    • Attach EC2 instances.
    • A health check that probes :2779/version on the Etcd EC2 instances.
  • An Internet Gateway and attach to VPC

  • Creates a default security group, allowing all (don’t do this in production)

  • Creates the VPC with a subnet of


  • Key used to SSH into the bastion host and Etcd EC2 instances.


Log on to each ec2 instance and check the service via “journalctl -u etcd.service”. In particular, we’re interested in the peering and if any errors occur.

ubuntu@ip-10-0-1-100:~$ journalctl -u etcd.service
Sep 08 15:34:22 ip-10-0-1-100 etcd[1575]: found peer b8321dfeb5729811 in the cluster
Sep 08 15:34:22 ip-10-0-1-100 etcd[1575]: found peer 7e245ce888bd3e1f in the cluster
Sep 08 15:34:22 ip-10-0-1-100 etcd[1575]: found self b4ccf12cb29f9dda in the cluster
Sep 08 15:34:22 ip-10-0-1-100 etcd[1575]: discovery URL=
 cluster 29d1ca9a6b0f3f87
Sep 08 15:34:23 ip-10-0-1-100 systemd[1]: Started etcd service.
Sep 08 15:34:23 ip-10-0-1-100 etcd[1575]: ready to serve client requests
Sep 08 15:34:23 ip-10-0-1-100 etcd[1575]: set the initial cluster version to 3.3
Sep 08 15:34:23 ip-10-0-1-100 etcd[1575]: enabled capabilities for version 3.3

We can also check they’re registered and healthy with the ALB:

Next, run a couple of commands against the ALB address:

[ec2-user@ip-10-0-4-242 ~]$ etcdctl --endpoints member list
7e245ce888bd3e1f: name=ip-10-0-3-100 peerURLs= clientURLs= isLeader=false
b4ccf12cb29f9dda: name=ip-10-0-1-100 peerURLs= clientURLs= isLeader=false
b8321dfeb5729811: name=ip-10-0-2-100 peerURLs= clientURLs= isLeader=true
[ec2-user@ip-10-0-4-242 ~]$ etcdctl --endpoints cluster-health
member 7e245ce888bd3e1f is healthy: got healthy result from
member b4ccf12cb29f9dda is healthy: got healthy result from
member b8321dfeb5729811 is healthy: got healthy result from
cluster is healthy

Splendid. Now this cluster is ready to start serving clients.

« Older posts Newer posts »

© 2020 Virtual Thoughts

Theme by Anders NorenUp ↑

Social media & sharing icons powered by UltimatelySocial
Visit Us
Follow Me