Virtual Thoughts

Virtualisation, Storage and various other ramblings.

Author: David (page 1 of 5)

Creating a highly available, cross AZ, loadbalanced ETCD cluster in AWS with Terraform

Having experimented with Terraform recently, I decided to leverage this tool by creating an etcd cluster in AWS. This blog post goes through the steps used to accomplish this. For readability, I’ve only quoted pertinent code snippets, but all of the code can be found at https://github.com/David-VTUK/terraformec2.

Disclaimer

I do not profess to be an Etcd, Terraform or AWS expert, therefore he be dragons in the form of implementations unlikely to be best practice or production-ready. In particular, I would like to revisit this at some point and enhance it to include:

  • Securing all Etcd communication with generated certs.
  • Implement a mechanism for rotating members in/out of the cluster.
  • Leverage auto-scaling groups for both bastion and Etcd members.
  • Locking down the security groups.
  • ….and a  lot more.

 

Architecture

The basic principles of this implementation are as follows:

  • Etcd members will reside in private subnets and accessed via an internal load-balancer.
  • Etcd members will be joined to a cluster by leveraging the Etcd discovery URL.
  • A bastion host will be used to proxy all access to the private subnets.
  • The Terraform script creates all the components, at the VPC level and beyond.

The scripts are separated into the following files:

.
├── deployetcd.tpl
├── etcdBootstrapScript.tf
├── etcdEC2Instances.tf
├── etcdMain.tf
├── etcdNetworking.tf
├── etcdSecurityGroups.tf
├── etcdVPC.tf
├── terraformec2.pem
├── terraform.tfstate
└── terraform.tfstate.backup

deployEtcd.tpl

This is a template file for Terraform, it’s used to generate a bootstrap script to run on each of our Etcd nodes.

To begin with, download Etcd, extract and do some general housekeeping with regards to access and users. It will also output two environment variables – ETCD_HOST_IP and ETCD_NAME, which is needed for the Systemd unit file.

cd /usr/local/src
sudo wget "https://github.com/coreos/etcd/releases/download/v3.3.9/etcd-v3.3.9-linux-amd64.tar.gz"
sudo tar -xvf etcd-v3.3.9-linux-amd64.tar.gz
sudo mv etcd-v3.3.9-linux-amd64/etcd* /usr/local/bin/
sudo mkdir -p /etc/etcd /var/lib/etcd
sudo groupadd -f -g 1501 etcd
sudo useradd -c "etcd user" -d /var/lib/etcd -s /bin/false -g etcd -u 1501 etcd
sudo chown -R etcd:etcd /var/lib/etcd

export ETCD_HOST_IP=$(ip addr show eth0 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)
export ETCD_NAME=$(hostname -s)

Create the Systemd unit file

sudo -E bash -c 'cat << EOF > /lib/systemd/system/etcd.service
[Unit]
Description=etcd service
Documentation=https://github.com/coreos/etcd

[Service]
User=etcd
Type=notify
ExecStart=/usr/local/bin/etcd \\
 --data-dir /var/lib/etcd \\
 --discovery ${discoveryURL} \\
 --initial-advertise-peer-urls http://$ETCD_HOST_IP:2380 \\
 --name $ETCD_NAME \\
 --listen-peer-urls http://$ETCD_HOST_IP:2380 \\
 --listen-client-urls http://$ETCD_HOST_IP:2379,http://127.0.0.1:2379 \\
 --advertise-client-urls http://$ETCD_HOST_IP:2379 \\

[Install]
WantedBy=multi-user.target
EOF'

${discoveryURL} is a placeholder variable. The Terraform script will replace this and generate the script file with a value for this variable.

etcdBootstrapScript.tf

This is where the discovery token is generated, parsed into the template file to which a new file will be generated – our complete bootstrap script. Simply performing an HTTP get request to the discovery URL will return a unique identifier URL in the response body. So we grab this response and store it as a data object. Note “size=3” denotes the explicit size of the Etcd cluster.

data "http" "etcd-join" {
  url = "https://discovery.etcd.io/new?size=3"
}

Take this data and parse it into the template file, filling it the variable ${discoveryURL} with the unique URL generated.

data "template_file" "service_template" {
  template = "${file("./deployetcd.tpl")}"

  vars = {
    discoveryURL = "${data.http.etcd-join.body}"
  }
}

Take the generated output, and store it as “script.sh” in the local directory

resource "local_file" "template" {
  content  = "${data.template_file.service_template.rendered}"
  filename = "script.sh"
}

The purpose of this code block is to generate one script to be executed on all three Etcd members, each joining a specific 3 node etcd cluster.

etcdEC2Instances.tf

This script performs the following:

  • Create a bastion host in the public subnet with a public IP.
  • Create three etcd instanced in each AZ, and execute the generated script via the bastion host:
    connection {
    host = "${aws_instance.etc1.private_ip}"
    port = "22"
    type = "ssh"
    user = "ubuntu"
    private_key = "${file("./terraformec2.pem")}"
    timeout = "2m"
    agent = false

    bastion_host = "${aws_instance.bastion.public_ip}"
    bastion_port = "22"
    bastion_user = "ec2-user"
    bastion_private_key = "${file("./terraformec2.pem")}"
}

  provisioner "file" {
    source      = "script.sh"
    destination = "/tmp/script.sh"
  }

  provisioner "remote-exec" {
    inline = [
      "chmod +x /tmp/script.sh",
      "/tmp/script.sh",
    ]
  }

etcdMain.tf

  • Contains access and secret keys for AWS access

etcdNetworking.tf

Configures the following:

  • Elastic IP for NAT gateway.
  • NAT gateway so private subnets can route out to pull etcd.
  • Three subnets in the EU-West-2 region
    • EU-West-2a
    • EU-West-2b
    • EU-West-2c
  • An ALB that spans across the aforementioned AZ’s:
    • Create a target group.
    • Create a listener on Etcd port.
    • Attach EC2 instances.
    • A health check that probes :2779/version on the Etcd EC2 instances.
  • An Internet Gateway and attach to VPC

etcdSecurityGroups.tf

  • Creates a default security group, allowing all (don’t do this in production)

etcdVPC.tf

  • Creates the VPC with a subnet of 10.0.0.0/16

terraformec2.pem

  • Key used to SSH into the bastion host and Etcd EC2 instances.

Validating

Log on to each ec2 instance and check the service via “journalctl -u etcd.service”. In particular, we’re interested in the peering and if any errors occur.

ubuntu@ip-10-0-1-100:~$ journalctl -u etcd.service
Sep 08 15:34:22 ip-10-0-1-100 etcd[1575]: found peer b8321dfeb5729811 in the cluster
Sep 08 15:34:22 ip-10-0-1-100 etcd[1575]: found peer 7e245ce888bd3e1f in the cluster
Sep 08 15:34:22 ip-10-0-1-100 etcd[1575]: found self b4ccf12cb29f9dda in the cluster
Sep 08 15:34:22 ip-10-0-1-100 etcd[1575]: discovery URL= https://discovery.etcd.io/32656e986a6a53a90b4bdda27559bf6e
 cluster 29d1ca9a6b0f3f87
Sep 08 15:34:23 ip-10-0-1-100 systemd[1]: Started etcd service.
Sep 08 15:34:23 ip-10-0-1-100 etcd[1575]: ready to serve client requests
Sep 08 15:34:23 ip-10-0-1-100 etcd[1575]: set the initial cluster version to 3.3
Sep 08 15:34:23 ip-10-0-1-100 etcd[1575]: enabled capabilities for version 3.3

We can also check they’re registered and healthy with the ALB:

Next, run a couple of commands against the ALB address:

[ec2-user@ip-10-0-4-242 ~]$ etcdctl --endpoints http://internal-terraform-example-alb-824389756.eu-west-2.alb.amazonaws.com:2379 member list
7e245ce888bd3e1f: name=ip-10-0-3-100 peerURLs=http://10.0.3.100:2380 clientURLs=http://10.0.3.100:2379 isLeader=false
b4ccf12cb29f9dda: name=ip-10-0-1-100 peerURLs=http://10.0.1.100:2380 clientURLs=http://10.0.1.100:2379 isLeader=false
b8321dfeb5729811: name=ip-10-0-2-100 peerURLs=http://10.0.2.100:2380 clientURLs=http://10.0.2.100:2379 isLeader=true
[ec2-user@ip-10-0-4-242 ~]$ etcdctl --endpoints http://internal-terraform-example-alb-824389756.eu-west-2.alb.amazonaws.com:2379 cluster-health
member 7e245ce888bd3e1f is healthy: got healthy result from http://10.0.3.100:2379
member b4ccf12cb29f9dda is healthy: got healthy result from http://10.0.1.100:2379
member b8321dfeb5729811 is healthy: got healthy result from http://10.0.2.100:2379
cluster is healthy

Splendid. Now this cluster is ready to start serving clients.

Application security with mutual TLS (mTLS) via Istio

TLS Overview

If we take an example of accessing a website such as https://www.virtualthoughts.co.uk/, these are the high-level steps of what occurs:

 

 

 

  1. The client initiates a connection to the web server requesting an HTTPS connection.
  2. The web server responds with its public key. The client validates the key with its list of known Certificate Authorities.
  3. A session key is generated by the client and encrypted with the web server’s public key and is sent back to the web server.
  4. The web server decrypts the session key with its private key. End to end encryption is established.

By default, the TLS protocol only proves the identity of the server to the client using X.509 certificate and the authentication of the client to the server is left to the application layer.  For external, public-facing websites, this is an acceptable and well-established implementation of TLS. But what about communication between different microservices?

 

As opposed to monolithic applications, microservices are usually inter-connected which allow them to be scaled/modified/etc independently. But this does raise some challenges. For example:

  • How do we ensure service-to-service communication is always encrypted?
  • How can do we do this without changing the application source code?
  • How can we automatically secure communication when we introduce a new service to an application?
  • How can we authenticate clients and servers and fully establish a “zero trust” network policy?

Istio can help us address these challenges:

Example Application

To demonstrate Istio’s mTLS capabilities a WordPress Helm chart was deployed into a namespace with automatic sidecar injection. Installing and configuring Istio can be found on a previous blog post. By default, the policy specifies no mTLS between the respective services. As such, the topology of the solution is depicted below:

 

 

We can validate this by using Istioctl:

 

All of the “testsite” services (WordPress frontend and backend) Envoy proxies are using HTTP as their transport mechanism. Therefore mTLS has not been configured yet.

Creating Istio Objects – Policy and Destination Rules

As you might expect, establishing mutual TLS (mTLS) is a two-part process, First, we must configure the clients to leverage mTLS, as well as the servers. This is accomplished with Policy and Destination rules.

Policy (AKA – what I, the server, will accept)

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
 name: "default"
 namespace: "wordpress-app"
spec:
 peers:
 - mtls: {}

This example policy strictly enforces only mTLS connections between services within the “wordpress-app” namespace

DestinationRule (AKA – what I, the client, will send out)

 apiVersion: "authentication.istio.io/v1alpha1" 
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
 name: "vt-wordpress-mariadb"
 namespace: "wordpress-app"
spec:
 host: "*.wordpress-app.svc.cluster.local"
 trafficPolicy:
 tls:
 mode: ISTIO_MUTUAL

This example enforces the use of mutual TLS when communicating with any service in the wordpress-app namespace. Applying these and re-running the previous istioctl command yields the following result:

This is accomplished largely due to Citadel – a component in the Istio control plane that manages certificate creation and distribution:

When mTLS is configured the traffic flow (from a high level) can be described as follows:

  • Citadel provides certificates to the sidecar pods and manages them.
  • WordPress pod creates a packet to query the MYSQL database.
    • WordPress Envoy sidecar pod intercepts this and establishes a connection to the destination sidecar pod and presents its certificate for authenticity.
  • MYSQL Envoy sidecar pod receives a connection request, validates the client’s certificate and sends its own back.
  • WordPress Envoy sidecar pod receives MYSQL’s certificate and checks it for authenticity.
  • Both proxies are in agreement as to each other’s identity and establish an encrypted tunnel between the two.

This is what makes it “mutual” TLS. In effect, both services are presenting, inspecting and validating each other’s certificate as a prerequisite for service-to-service communication. This differs from a standard HTTPs site described earlier on where only the client was validating the server.

Additional Comments

Some additional observations I’ve made from this exercise

  • If enforcing strict mTLS on a service that’s exposed externally from a load balancer, your clients will obviously need to send x509 certificates that can be validated by Citadel. A more flexible alternative to this is to employ an Istio gateway that provides TLS termination at the cluster boundary. This negates the need to provision x509 certs to each and every client, whilst maintaining mTLS within the cluster.
  • Envoy sidecar pods can affect liveness probes and might require you to implement
     sidecarInjectorWebhook.rewriteAppHTTPProbe=true 

    upon installing Helm 

Container Packet Inspection with NSX-T

For troubleshooting (or just being a bit nosey) we have a number of tools that allow us to inspect the traffic between two endpoints. When it comes to containers however, our approach has to be adjusted slightly. When using NSX-T as a CNI, we have some of these tools available to us out of the box.

App Overview

 

For this example, I’ve deployed a standard WordPress deployment consisting of a frontend (web) pod and a backend (DB) pod. The objective is to identify and capture the traffic between these pods.

 

Configure NSX-T Port Mirroring

Navigate to the “Port Mirroring Session” section in NSX-T via “Advanced Networking & Security” and click “Add”

Define a Local SPAN session:

Define the transport node (Source ESXI host)

Select the physical NIC(s) that participate in the n-vds configuration that we want to capture packets from.

Important: Enable “Encapsulated Packet” (Disabled by default) so we can inspect the underlying Geneve overlay packets. However, the NIC we want to mirror to must support the increased overall packet size. Therefore, adjust the MTU of the nic on the Wireshark host accordingly.

 

Select the source VNIC to capture from – This is the NIC of the Kubernetes worker node VM.

 

 

Finally, select the destination for traffic mirroring. For this example, it’s a NIC on a VM I have Wireshark installed on

Traffic Inspection

With all the hard work out of the way, we can simply provide a filter in Wireshark to show only packets originating from my WordPress Web Pod and terminating at the WordPress DB pod:

Notes:

  • The overall frame size of 1600 further validates our requirement to have the recipient of the traffic mirroring having an interface configured accordingly.
  • Because we previously defined the mirroring of encapsulated packets we can see the overlay information
  • We can see the entire packet structure consisting of:
    • Outer Ethernet (NSX-T Transport Node MAC)
    • Outer IP (NSX-T VTEP IP)
    • Geneve UDP
    • Geneve Data
    • Inner Ethernet (Pod Ethernet addresses)
    • Inner IP (Pod IP addresses)
    • Inner TCP (MYSQL connection)
    • Inner Payload (MYSQL Query) – Highlighted

 

Step by Step – Istio up and running

Service Mesh is a pretty hot topic in the Kubernetes ecosystem currently, and I wanted to get it up and running in my own lab environment. Istio’s documentation has a pre-baked solution to demonstrate some of its capabilities (a book app, if memory serves me correctly), but I wanted to deploy my own app to get more “hands-on” experience with the tech, even if it’s only very basic to start with.

Install Istio

There are a number of prerequisite steps that need to be satisfied prior to installing Istio. These are specific to my environment,  others may differ.

Install the Helm client

 sudo snap install helm --classic 

Grab Istio (1.2.0 in this example)

curl -L https://git.io/getLatestIstio | ISTIO_VERSION=1.2.0 sh - cd istio-1.2.0/

Create the Helm service account (named “tiller”)

kubectl apply -f install/kubernetes/helm/helm-service-account.yaml

Initialise Helm  using the service account specified in the previous step

helm init --service-account tiller

Create a namespace to accommodate the Istio components

kubectl create ns istio-system

Initialise Istio into the aforementioned namespace:

helm install install/kubernetes/helm/istio-init --name istio-init --namespace istio-system

Monitor the state of the pods – it will take some time for the pods to finish – these create the CRD’s required for Istio

kubectl get pods -n istio-system
NAME                      READY   STATUS              RESTARTS   AGE
istio-init-crd-10-t82m6   0/1     ContainerCreating   0          95s
istio-init-crd-11-42622   0/1     ContainerCreating   0          95s
istio-init-crd-12-65m5v   0/1     ContainerCreating   0          95s

Install Istio into the aforementioned namespace

helm install install/kubernetes/helm/istio --name istio --namespace istio-system

Configure a namespace for automatic sidecar injection

By this point, we have the internal foundations for Istio, but we’re not leveraging it. One of the fundamental workings of Istio is the use of pod sidecars. Sidecars act as the data plane, facilitating a lot of the features we want to leverage from Istio.

The overall architecture of an Istio-based application.

Istio doesn’t do this automatically, out of the box for all pods deployed into an environment, but Istio will inject sidecars into pods deployed into namespaces that have the istio-injection=enabled label set.

kubectl create ns app-with-injection
namespace/app-with-injection created
kubectl label namespace app-with-injection istio-injection=enabled
namespace/app-with-injection labeled

We can validate this by creating a pod into this namespace:

kubectl run nginx -n app-with-injection --image nginx

And checking the Pod contents (notice how this pod has two containers)

kubectl get pods -n app-with-injection
NAME                     READY   STATUS    RESTARTS   AGE
nginx-7cdbd8cdc9-96mbz   2/2     Running   0          52s

The proxy sidecar:

 

Environment Anatomy

The diagram below shows how my test environment is set up

 

 

Key considerations:

The Istio gateway will reside on the edge
80% of all traffic will be routed to v1 of my web application
20% of all traffic will be routed to v2 of my web application

The application manifest can be found at https://raw.githubusercontent.com/David-VTUK/istioexample/master/webapp.yaml

To accomplish this we need to implement two key objects:

Gateway

This is our entry point into our application. By default, Istio deploys the gateway object (we must note the external IP)

kubectl get svc -n istio-system
NAME                     TYPE           CLUSTER-IP       EXTERNAL-IP                
istio-citadel            ClusterIP      10.100.200.47                        
istio-galley             ClusterIP      10.100.200.149                       
istio-ingressgateway     LoadBalancer   10.100.200.244   10.10.20.150,100.64.80.1   
istio-pilot              ClusterIP      10.100.200.170                       
istio-policy             ClusterIP      10.100.200.3                        
istio-sidecar-injector   ClusterIP      10.100.200.169                   
istio-telemetry          ClusterIP      10.100.200.141                      
prometheus               ClusterIP      10.100.200.238                     

We configure the Gateway by deploying a gateway manifest file:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: http-gateway
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"              
  • Kind : Type of object. Gateway is a CRD (Custom Resource Definition) that Istio implements
  • Selector: What this applies to, in this case the default Ingress Gateway
  • Ports: Which ports we want to listen to on the external IP address, together with a name and protocol
  • Hosts : We can implement layer 7 load balancing on the edge, but as I’ll be testing this out via IP address, “*” will suffice. In production, this would likely be an FQDN of an external facing website

 

VirtualService

gateway object helps us define the entry point into the cluster, but we have yet to effectively tell the gateway where to route traffic to. This is where the VirtualService object type comes in. This is where we define routing intelligence.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: demoapp
spec:
  hosts:
  - "*"
  gateways:
  - http-gateway
  http:
  - route:
    - destination:
        port:
          number: 80
        host: vt-webapp-v1.app-with-injection.svc.cluster.local
      weight: 80
    - destination:
        port:
          number: 80
        host: vt-webapp-v2.app-with-injection.svc.cluster.local
      weight: 20      

What the above effectively does is listen for all HTTP requests (hence the “*” under “hosts”) and route 80% of traffic to V1 of the webapp, by directing traffic at the respective service and 20% to v2.

Testing

The “WebApp” is pretty simple. It simply displays one of the following (depending on the version)

 

What we should now see from accessing the external IP is traffic being split across both services via a 80/20 split:

Out of 10 curl commands 8 were routed to v1 of my app, 2 were routed to v2 of my app.

Conclusion

Admittedly, this is an extremely simple example of a more simple use case of Istio, but as I’m learning, I think it’s a decent start, and I hope others find it useful.

Removing Unused PKS LoadBalancer IP allocations in NSX-T

I’m constantly building, removing, destroying, breaking and constantly tinkering in my lab, which unfortunately can leave behind a bit of technical mess. Most recently I’ve been doing some messing around in PKS and consequently had a load of load balancer VIPs no longer in use due to clusters being destroyed and / or recreated.

Unfortunately, there’s no way to remove these in the UI (for now, at least). But you can remove them using the API.:


curl -k -u <NSX-USERNAME>:<NSX-PASSWORD> -X POST 'https://<NSX-IP-OR-FQDN>/api/v1/pools/ip-pools/<POOL-ID>?action=RELEASE' -H "X-Allow-Overwrite: true" -d {"allocation_id":"10.10.10.2"}' -H "Content-Type: application/json"

This is great for the odd IP, but for me, I had 10’s of IP addresses I needed releasing. A quick and easy Bash script to the rescue!


for i in {2..254}
do
curl -k -u <NSX-USERNAME>:<NSX-PASSWORD> -X POST 'https://<NSX-IP-OR-FQDN>/api/v1/pools/ip-pools/<POOL-ID>?action=RELEASE' -H "X-Allow-Overwrite: true" -d '{"allocation_id":"10.10.10.'"$i"'"}' -H "Content-Type: application/json"
done

Modify the for loop for your environment. I typically have .1 as my default gateway.

Note : After releasing an IP, it seems to take a while before it’s reflected in the UI, therefore give it a few minutes.

Introducing Velero – Backup and DR for Kubernetes Applications

Image result for velero logo heptio

 

What is Velero?

Velero (previously known as Heptio ARK) provides a suite of tools to backup Kubernetes resources and applications for two main purposes:

  • Disaster Recovery – Recover Kubernetes cluster components and applications.
  • Migration – Migrate your Kubernetes applications to another Kubernetes cluster.

Migrating Kubernetes applications is a compelling use case. One of the significant benefits of using Kubernetes is the predictability of the platform and consequently the portability of applications that reside on it. With the main exception of nuances with persistent storage, the Kubernetes API will fell almost indistinguishable whether it resides on prem, GKE, AKS, EKS and elsewhere. If you have your Kubernetes-based application on one provider and want to migrate it to another or duplicate it to run elsewhere for dev/test, this can easily be achieved. Especially with Velero.

 

Install and Configure Velero

Velero consists primarily of the server (runs in a container) a storage location (ie S3 bucket) and the CLI.

Rather than reiterate what’s already in Velero’s existing and comprehensive documentation, the straight forward instructions are located https://heptio.github.io/velero/master/.

 

Demo App

To understand and get to grips with Velero I decided to write my own application. It’s a really simple application written in Golang and does the following:

  • Every 2 seconds, output the contents of /mnt/data/data.txt with a timestamp.

The pod will mount /mnt/data as a persistent volume based on the respective claim. The file it reads is “data.txt” which contains “some data”. The PV type is hostpath, which is suitable for testing in this single worker node cluster.

The overall solution is depicted below:

After deploying the application, we can validate it’s working as expected by inspecting the stdout file descriptor with kubectl logs:

Execute a backup (Note that in production this would likely be scheduled, and this backup includes everything. Ideally, you’d probably want to back up on a per namespace or app basis).

Validate the backup:

We can also see the backup residing in the previously defined S3 bucket:

Simulating a Disaster and Restoring

To simulate a disaster, I’ll simply remove my Pod, PV and PVC:

And for good measure, issue the same command previously used to extrapolate the logs:

We can use the velero restore command to pull the backup from our S3 bucket and restore our application.

Validate the restore:

Note that because the previous backup encompassed all namespaces and all resource types, Velero employs some logic to determine which objects it should restore, based on what already exists. However, my app, its namespace, PV and PVC has been restored, which we can validate with:

Closing Thoughts

Being able to mobilise a Kubernetes-based application on virtually any Kubernetes-based cluster is one of many compelling reasons to leverage this technology. Equally as important as ensuring availability and reliability of applications is to adopt a solid backup and disaster recovery solution. Velero is designed to facilitate this, and its flexibility in where to store and retrieve backups from helps with preventing lock-in with a particular solution, provider or vendor. Whether it’s on prem, Azure, Google Cloud, AWS, Digital Ocean or others, disasters do happen, downtime occurs. Being able to lift and shift entire cluster resources in a standardised is quickly becoming a solid requirement for modern applications.

Bootstrapping Prometheus, Grafana and Alertmanager to PKS deployed K8s Clusters

PKS is a comprehensive platform for the provisioning and management of Kubernetes clusters, which can be further enhanced by leveraging its extensibility options. In this post, we will modify a plan to deploy a yaml manifest file which provisions Prometheus, Grafana, and Alertmanager backed by NSX-T load balancers.

Why Prometheus, Grafana and Alertmanager?

The Cloud Native Computing Foundation accepted Prometheus as its second incubated project, the first being Kubernetes. Originally developed by SoundCloud. It has quickly become a popular platform for the monitoring of Kubernetes platforms. Built upon a powerful analytics engine, extensive and highly flexible data modeling can be accomplished with relative ease.

Grafana is, amongst other things, a visualisation tool that enables users to graph, chart, and generally visually represent data from a wide range of sources, Prometheus being one of them.

AlertManager handles alerts that are sent by applications such as Prometheus and performs a number of operations such as deduplicating, grouping and routing.

What I wanted to do, as someone unfamiliar with these tools is to devise a way to deploy these components in an automated way in which I can destroy, and recreate with ease. The topology of the solution is depicted below:

 

 

Constructing the manifest file

TLDR; I’ve placed the entire manifest file here, which creates the following:

  • Create the “monitoring” namespace.
  • Create a service account for Prometheus.
  • Create a cluster role requires for the Prometheus service account.
  • Create a cluster role binding for the Prometheus service account and the Prometheus cluster role.
  • Create a config map for Prometheus that:
    • Defines the Alertmonitor target.
    • Defines K8S master, K8S worker, and cAdvisor scrape targets.
    • Defines where to input alert rules from.
  • Create a config map for Prometheus that:
    • Provides a template for alerting rules.
  • Create a single replica deployment for Prometheus.
    • Expose this deployment via “LoadBalancer” (NSX-T).
  • Create a single replica deployment for Grafana.
    • Expose this deployment via “LoadBalancer” (NSX-T).
  • Create a single replica deployment for Alertmonitor.
    • Expose this deployment via “Loadbalancer” (NSX-T).

 

Bootstrapping the manifest file in PKS

Log into Ops manager and select the PKS tile:

Select a plan from the left-hand side, record the plan name for later:

Scroll down and paste the aforementioned YAML file into the Add-ons section

Save and then provision a cluster using the aforementioned plan:


david@mgmt-jumpbox:~$ pks create-cluster k8s --external-hostname k8s.virtualthoughts.co.uk --plan medium-with-prometheus-grafana-alertmanager --num-nodes 2

After which execute the following to acquire the list of loadbalancer IP addresses for the respective services:

david@mgmt-jumpbox:~$ kubectl get svc -n monitoring
NAME           TYPE           CLUSTER-IP       EXTERNAL-IP                 PORT(S)        AGE
alertmanager   LoadBalancer   10.100.200.218   100.64.96.5,172.16.12.129   80:31747/TCP   100m
grafana        LoadBalancer   10.100.200.89    100.64.96.5,172.16.12.128   80:31007/TCP   100m
prometheus     LoadBalancer   10.100.200.75    100.64.96.5,172.16.12.127   80:31558/TCP   100m

Prometheus – Quick tour

Logging into http://prometheus-lb-vip/targets will list the list of scrape targets for Prometheus, which have been configured via the respective configmap and include:

  • API server (Master)
  • Nodes (Workers)
  • cAdvisor (Pods)
  • Prometheus (self)

Which can be graphed / modelled / queried:

 

Grafana – Quick Tour

Out of the box, Grafana has very limited configuration applied – I struggled a little bit with constructing a configmap that would automatically add Prometheus as a data source, so a little bit of manual configuration is required (for now). Accessing http://grafana-lb-vip will prompt for a logon: (admin/admin) is the default

 

Add a data source:

Hint : use “prometheus.monitoring.svc.cluster.local” as the source URL

After creating a dashboard (or importing one) we can validate Grafana is extracting information from Prometheus

 

 

Alertmanager – Quick Tour

Alertmanager can be accessed via http://alertmanager-lb-vip. From the YAML manifest it has a vanilla config, but Prometheus is configured to use it as a alert target via the configmap:

  prometheus.yml: |-
    global:
      scrape_interval: 5s
      evaluation_interval: 5s
      
    alerting:
      alertmanagers:
        - static_configs:
          - targets: 
            - "alertmanager.monitoring.svc.cluster.local:80"

Alerts need to be configured in Prometheus in order for Alertmanager to ingest / deduplicate / forward etc.

As an example, I tested some integration with Discord:

Practical to send alerts to a Discord sever? Probably not.

Fun? Yes!

CKA Exam Experience (Inc study & lab guide)

Introduction

Over the long bank holiday weekend, I sat and passed the Certified Kubernetes Exam (CKA). This blog post goes over my experience (With respect to the NDA) together with a lab guide I’ve made which I’ve uploaded hoping it might help others.

 

 

Format

The online exams consist of a set of performance-based items (problems) to be solved on the command line. For the CKA there are 24 questions of varying difficulty. At the time of writing, the only option to sit this exam is through remote proctoring.

This link contains the most pertinent information to assimilate.

 

Experience

I’m a huge fan of practical exams. I’m so glad the powers at be decided to go down this route. I absolutely loathe multiple choice exams for many reasons. The remote proctoring was a new experience for me, and I wasn’t completely comfortable with it. Given the choice, I would have preferred to go to a test center. I hope The Linux Foundation adds this option in the future.

I sat the exam first time around feeling relatively confident, but knew I had some weaker areas. After a painstaking wait I received the following email:

confused oh no GIF by It's Always Sunny in Philadelphia

I brushed myself off, crammed the areas I was weaker on, took the exam again and waited….

…and waited

From my experience, as a techie, I get a much more accomplished feeling when passing practical exams, such as this / VCAP’s etc. Either way, to say I was happy would be a gross understatement.

 

Takeaways

  • A lot of people say this exam is “hard”. I get really discouraged reading up on peoples exam experiences saying exams are “hard”. I would say a more accurate adjective for this exam would be “Fair”. Know the curriculum, practice your craft, and you’ll get there.
  • Lean on the documentation as much as you need to. You have access to kubernetes.io/docs during the exam.
  • It’s a practical exam, so practice, practice, and practice some more.
  • You get a free retake, so don’t worry if you don’t pass first time.
  • kubectl run somedeployment –image=nginx –replicas=5 –dry-run -o yaml. Output existing or new objects to a yaml file if you need to make finer adjustments or create objects from scratch.

 

Lab Guide

My revision approach for this exam predominantly consisted of:

  1. Reading up on the topics
  2. Apply the knowledge to practical examples
  3. Validate the approach

I ended up with three documents:

  • Revision Notes
  • Practice lab exercises
  • Practice lab exercises answers (writing this helped me commit this information to memory)

 

All of which can be found at https://github.com/David-VTUK/CKA-StudyGuide

 

Exposing the K8s dashboard via a NSX-T Load balancer

For the following to work, your k8s infrastructure needs to leverage some kind of CNI that’s able to provision load balancers. For this example I’m leveraging PKS which has native integration with NSX-T.

The default way to access the Kubernetes dashboard is to leverage the kubectl proxy command. However, this is somewhat limiting for a production environment. An alternative way is to expose the dashboard through a load balancer.

 

Modify the Dashboard service by executing : kubectl -n kube-system edit service kubernetes-dashboard and modifying the “type” field from “ClusterIP” to “LoadBalancer”

 

Afterwards, the service will be reconfigured to be presented by a load balancer external VIP.

At which point we can access it directly:

 

PKS, Harbor and the importance of container registries

What are container registries and why do we need them?

A lot of the time, particularly when individuals and organisations are evaluating, testing and experimenting with containers they will use public container registries such as Docker Hub.  These public registries provide an easy-to-use, simple way to access images. As developers, application owners, system admins etc gain familiarity and experience additional operational considerations need to be explored, such as:

  • Organisation – How can we organise container images in a meaningful way? Such as by environment state (Prod/Dev/Test) and application type?
  • RBAC – How can we implement role-based access control to a container registry?
  • Vulnerability Scanning – How can we scan container images for known vulnerabilities?
  • Efficiency – How can we centrally manage all our container images and deploy an application from them?
  • Security – Some images need to kept under lock and key, rather than using an external service like Docker Hub.

Introducing VMware Harbor Registry

VMware Harbor Registry has been designed to address these considerations as enterprise-class container registry solution with integration into PKS. In this post, We’ll have a quick primer on getting up and running with Harbor in PKS and explore some of its features. To begin, we need to download PKS Harbor from the Pivotal site and import it into ops manager.

After which the tile will be added (When doing this for the first time it will have an orange bar at the bottom. Press the tile to configure).

The following need to be defined with applicable parameters to suit your environment.

  • Availability Zone and Networks – This is where the Harbor VM will reside, and the respective configuration will be dependent on your setup.
  • General – Hostname and IP address settings
  • Certificate – Generate a self-signed certificate, or BYOC (bring your own certificate)
  • Credentials – Define the local admin password
  • Authentication – Choose between
    • Internal
    • LDAP
    • UAA in PKS
    • UAA in PAS
  • Container Registry store – Choose where to store container images pushed to Harbor
    • Local file system
    • NFS Server
    • S3 Bucket
    • Google Cloud Storage
  • Clair Proxy Settings
  • Notary settings
  • Resource Config

VMware Harbor Registry – Organisation

Harbor employs the concept of “projects”. Projects are a way of collecting images for a specific application or service. When images are pushed to Harbor, they reside within a project:

 

Projects can either be private or public and can be configured during, or after, project creation:

A project is comprised of a number of components:

 

VMware Harbor Registry – RBAC

In Harbor, we have three role types we can assign to projects:

 

rbac

Image source: https://github.com/goharbor/harbor/blob/master/docs/user_guide.md#managing-projects

  • Guest – Read-only access, can pull images
  • Developer – Read/write access, can pull and push images
  • Admin – Read/Write access, as well as project-level activities, such as modifying parameters and permissions.

As a practical example, AD groups can be created to facilitate these roles:

And these AD groups can be mapped to respective permissions within the project

 

Therefore, facilitating RBAC within our Harbor environment. Pretty handy.

VMware Harbor Registry – Vulnerability Scanning

The ability to identify, evaluate and remediate vulnerabilities is a standard operation is modern software development and deployment. Thankfully Harbor addresses this with integration with Clair – an open source project that addresses the identification, categorisation and analysis of vulnerabilities within containers. As a demonstration we need to first push an image to Harbor:

After initiating a scan, Harbor can inform us of what vulnerabilities exist within this container image

We can then explore more details about these vulnerabilities, including when they were fixed:

 

Conclusion

Harbor provides us with an enterprise level, container registry solution. This blog post has only scratched the surface, and with constant development being invested into the project, expect more features and improvements.

 

Older posts

© 2019 Virtual Thoughts

Theme by Anders NorenUp ↑

Social media & sharing icons powered by UltimatelySocial
RSS