Virtual Thoughts

Virtualisation, Storage and various other ramblings.

Page 2 of 16

Creating a highly available, cross AZ, loadbalanced ETCD cluster in AWS with Terraform

Having experimented with Terraform recently, I decided to leverage this tool by creating an etcd cluster in AWS. This blog post goes through the steps used to accomplish this. For readability, I’ve only quoted pertinent code snippets, but all of the code can be found at https://github.com/David-VTUK/terraformec2.

Disclaimer

I do not profess to be an Etcd, Terraform or AWS expert, therefore he be dragons in the form of implementations unlikely to be best practice or production-ready. In particular, I would like to revisit this at some point and enhance it to include:

  • Securing all Etcd communication with generated certs.
  • Implement a mechanism for rotating members in/out of the cluster.
  • Leverage auto-scaling groups for both bastion and Etcd members.
  • Locking down the security groups.
  • ….and a  lot more.

 

Architecture

The basic principles of this implementation are as follows:

  • Etcd members will reside in private subnets and accessed via an internal load-balancer.
  • Etcd members will be joined to a cluster by leveraging the Etcd discovery URL.
  • A bastion host will be used to proxy all access to the private subnets.
  • The Terraform script creates all the components, at the VPC level and beyond.

The scripts are separated into the following files:

.
├── deployetcd.tpl
├── etcdBootstrapScript.tf
├── etcdEC2Instances.tf
├── etcdMain.tf
├── etcdNetworking.tf
├── etcdSecurityGroups.tf
├── etcdVPC.tf
├── terraformec2.pem
├── terraform.tfstate
└── terraform.tfstate.backup

deployEtcd.tpl

This is a template file for Terraform, it’s used to generate a bootstrap script to run on each of our Etcd nodes.

To begin with, download Etcd, extract and do some general housekeeping with regards to access and users. It will also output two environment variables – ETCD_HOST_IP and ETCD_NAME, which is needed for the Systemd unit file.

cd /usr/local/src
sudo wget "https://github.com/coreos/etcd/releases/download/v3.3.9/etcd-v3.3.9-linux-amd64.tar.gz"
sudo tar -xvf etcd-v3.3.9-linux-amd64.tar.gz
sudo mv etcd-v3.3.9-linux-amd64/etcd* /usr/local/bin/
sudo mkdir -p /etc/etcd /var/lib/etcd
sudo groupadd -f -g 1501 etcd
sudo useradd -c "etcd user" -d /var/lib/etcd -s /bin/false -g etcd -u 1501 etcd
sudo chown -R etcd:etcd /var/lib/etcd

export ETCD_HOST_IP=$(ip addr show eth0 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)
export ETCD_NAME=$(hostname -s)

Create the Systemd unit file

sudo -E bash -c 'cat << EOF > /lib/systemd/system/etcd.service
[Unit]
Description=etcd service
Documentation=https://github.com/coreos/etcd

[Service]
User=etcd
Type=notify
ExecStart=/usr/local/bin/etcd \\
 --data-dir /var/lib/etcd \\
 --discovery ${discoveryURL} \\
 --initial-advertise-peer-urls http://$ETCD_HOST_IP:2380 \\
 --name $ETCD_NAME \\
 --listen-peer-urls http://$ETCD_HOST_IP:2380 \\
 --listen-client-urls http://$ETCD_HOST_IP:2379,http://127.0.0.1:2379 \\
 --advertise-client-urls http://$ETCD_HOST_IP:2379 \\

[Install]
WantedBy=multi-user.target
EOF'

${discoveryURL} is a placeholder variable. The Terraform script will replace this and generate the script file with a value for this variable.

etcdBootstrapScript.tf

This is where the discovery token is generated, parsed into the template file to which a new file will be generated – our complete bootstrap script. Simply performing an HTTP get request to the discovery URL will return a unique identifier URL in the response body. So we grab this response and store it as a data object. Note “size=3” denotes the explicit size of the Etcd cluster.

data "http" "etcd-join" {
  url = "https://discovery.etcd.io/new?size=3"
}

Take this data and parse it into the template file, filling it the variable ${discoveryURL} with the unique URL generated.

data "template_file" "service_template" {
  template = "${file("./deployetcd.tpl")}"

  vars = {
    discoveryURL = "${data.http.etcd-join.body}"
  }
}

Take the generated output, and store it as “script.sh” in the local directory

resource "local_file" "template" {
  content  = "${data.template_file.service_template.rendered}"
  filename = "script.sh"
}

The purpose of this code block is to generate one script to be executed on all three Etcd members, each joining a specific 3 node etcd cluster.

etcdEC2Instances.tf

This script performs the following:

  • Create a bastion host in the public subnet with a public IP.
  • Create three etcd instanced in each AZ, and execute the generated script via the bastion host:
    connection {
    host = "${aws_instance.etc1.private_ip}"
    port = "22"
    type = "ssh"
    user = "ubuntu"
    private_key = "${file("./terraformec2.pem")}"
    timeout = "2m"
    agent = false

    bastion_host = "${aws_instance.bastion.public_ip}"
    bastion_port = "22"
    bastion_user = "ec2-user"
    bastion_private_key = "${file("./terraformec2.pem")}"
}

  provisioner "file" {
    source      = "script.sh"
    destination = "/tmp/script.sh"
  }

  provisioner "remote-exec" {
    inline = [
      "chmod +x /tmp/script.sh",
      "/tmp/script.sh",
    ]
  }

etcdMain.tf

  • Contains access and secret keys for AWS access

etcdNetworking.tf

Configures the following:

  • Elastic IP for NAT gateway.
  • NAT gateway so private subnets can route out to pull etcd.
  • Three subnets in the EU-West-2 region
    • EU-West-2a
    • EU-West-2b
    • EU-West-2c
  • An ALB that spans across the aforementioned AZ’s:
    • Create a target group.
    • Create a listener on Etcd port.
    • Attach EC2 instances.
    • A health check that probes :2779/version on the Etcd EC2 instances.
  • An Internet Gateway and attach to VPC

etcdSecurityGroups.tf

  • Creates a default security group, allowing all (don’t do this in production)

etcdVPC.tf

  • Creates the VPC with a subnet of 10.0.0.0/16

terraformec2.pem

  • Key used to SSH into the bastion host and Etcd EC2 instances.

Validating

Log on to each ec2 instance and check the service via “journalctl -u etcd.service”. In particular, we’re interested in the peering and if any errors occur.

ubuntu@ip-10-0-1-100:~$ journalctl -u etcd.service
Sep 08 15:34:22 ip-10-0-1-100 etcd[1575]: found peer b8321dfeb5729811 in the cluster
Sep 08 15:34:22 ip-10-0-1-100 etcd[1575]: found peer 7e245ce888bd3e1f in the cluster
Sep 08 15:34:22 ip-10-0-1-100 etcd[1575]: found self b4ccf12cb29f9dda in the cluster
Sep 08 15:34:22 ip-10-0-1-100 etcd[1575]: discovery URL= https://discovery.etcd.io/32656e986a6a53a90b4bdda27559bf6e
 cluster 29d1ca9a6b0f3f87
Sep 08 15:34:23 ip-10-0-1-100 systemd[1]: Started etcd service.
Sep 08 15:34:23 ip-10-0-1-100 etcd[1575]: ready to serve client requests
Sep 08 15:34:23 ip-10-0-1-100 etcd[1575]: set the initial cluster version to 3.3
Sep 08 15:34:23 ip-10-0-1-100 etcd[1575]: enabled capabilities for version 3.3

We can also check they’re registered and healthy with the ALB:

Next, run a couple of commands against the ALB address:

[ec2-user@ip-10-0-4-242 ~]$ etcdctl --endpoints http://internal-terraform-example-alb-824389756.eu-west-2.alb.amazonaws.com:2379 member list
7e245ce888bd3e1f: name=ip-10-0-3-100 peerURLs=http://10.0.3.100:2380 clientURLs=http://10.0.3.100:2379 isLeader=false
b4ccf12cb29f9dda: name=ip-10-0-1-100 peerURLs=http://10.0.1.100:2380 clientURLs=http://10.0.1.100:2379 isLeader=false
b8321dfeb5729811: name=ip-10-0-2-100 peerURLs=http://10.0.2.100:2380 clientURLs=http://10.0.2.100:2379 isLeader=true
[ec2-user@ip-10-0-4-242 ~]$ etcdctl --endpoints http://internal-terraform-example-alb-824389756.eu-west-2.alb.amazonaws.com:2379 cluster-health
member 7e245ce888bd3e1f is healthy: got healthy result from http://10.0.3.100:2379
member b4ccf12cb29f9dda is healthy: got healthy result from http://10.0.1.100:2379
member b8321dfeb5729811 is healthy: got healthy result from http://10.0.2.100:2379
cluster is healthy

Splendid. Now this cluster is ready to start serving clients.

Application security with mutual TLS (mTLS) via Istio

TLS Overview

If we take an example of accessing a website such as https://www.virtualthoughts.co.uk/, these are the high-level steps of what occurs:

 

 

 

  1. The client initiates a connection to the web server requesting an HTTPS connection.
  2. The web server responds with its public key. The client validates the key with its list of known Certificate Authorities.
  3. A session key is generated by the client and encrypted with the web server’s public key and is sent back to the web server.
  4. The web server decrypts the session key with its private key. End to end encryption is established.

By default, the TLS protocol only proves the identity of the server to the client using X.509 certificate and the authentication of the client to the server is left to the application layer.  For external, public-facing websites, this is an acceptable and well-established implementation of TLS. But what about communication between different microservices?

 

As opposed to monolithic applications, microservices are usually inter-connected which allow them to be scaled/modified/etc independently. But this does raise some challenges. For example:

  • How do we ensure service-to-service communication is always encrypted?
  • How can do we do this without changing the application source code?
  • How can we automatically secure communication when we introduce a new service to an application?
  • How can we authenticate clients and servers and fully establish a “zero trust” network policy?

Istio can help us address these challenges:

Example Application

To demonstrate Istio’s mTLS capabilities a WordPress Helm chart was deployed into a namespace with automatic sidecar injection. Installing and configuring Istio can be found on a previous blog post. By default, the policy specifies no mTLS between the respective services. As such, the topology of the solution is depicted below:

 

 

We can validate this by using Istioctl:

 

All of the “testsite” services (WordPress frontend and backend) Envoy proxies are using HTTP as their transport mechanism. Therefore mTLS has not been configured yet.

Creating Istio Objects – Policy and Destination Rules

As you might expect, establishing mutual TLS (mTLS) is a two-part process, First, we must configure the clients to leverage mTLS, as well as the servers. This is accomplished with Policy and Destination rules.

Policy (AKA – what I, the server, will accept)

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
name: "default"
namespace: "wordpress-app"
spec:
peers:
- mtls: {}

This example policy strictly enforces only mTLS connections between services within the “wordpress-app” namespace

DestinationRule (AKA – what I, the client, will send out)

 apiVersion: "authentication.istio.io/v1alpha1"
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
name: "vt-wordpress-mariadb"
namespace: "wordpress-app"
spec:
host: "*.wordpress-app.svc.cluster.local"
trafficPolicy:
tls:
mode: ISTIO_MUTUAL

This example enforces the use of mutual TLS when communicating with any service in the wordpress-app namespace. Applying these and re-running the previous istioctl command yields the following result:

This is accomplished largely due to Citadel – a component in the Istio control plane that manages certificate creation and distribution:

When mTLS is configured the traffic flow (from a high level) can be described as follows:

  • Citadel provides certificates to the sidecar pods and manages them.
  • WordPress pod creates a packet to query the MYSQL database.
    • WordPress Envoy sidecar pod intercepts this and establishes a connection to the destination sidecar pod and presents its certificate for authenticity.
  • MYSQL Envoy sidecar pod receives a connection request, validates the client’s certificate and sends its own back.
  • WordPress Envoy sidecar pod receives MYSQL’s certificate and checks it for authenticity.
  • Both proxies are in agreement as to each other’s identity and establish an encrypted tunnel between the two.

This is what makes it “mutual” TLS. In effect, both services are presenting, inspecting and validating each other’s certificate as a prerequisite for service-to-service communication. This differs from a standard HTTPs site described earlier on where only the client was validating the server.

Additional Comments

Some additional observations I’ve made from this exercise

  • If enforcing strict mTLS on a service that’s exposed externally from a load balancer, your clients will obviously need to send x509 certificates that can be validated by Citadel. A more flexible alternative to this is to employ an Istio gateway that provides TLS termination at the cluster boundary. This negates the need to provision x509 certs to each and every client, whilst maintaining mTLS within the cluster.
  • Envoy sidecar pods can affect liveness probes and might require you to implement
     sidecarInjectorWebhook.rewriteAppHTTPProbe=true 

    upon installing Helm 

Container Packet Inspection with NSX-T

For troubleshooting (or just being a bit nosey) we have a number of tools that allow us to inspect the traffic between two endpoints. When it comes to containers however, our approach has to be adjusted slightly. When using NSX-T as a CNI, we have some of these tools available to us out of the box.

App Overview

 

For this example, I’ve deployed a standard WordPress deployment consisting of a frontend (web) pod and a backend (DB) pod. The objective is to identify and capture the traffic between these pods.

 

Configure NSX-T Port Mirroring

Navigate to the “Port Mirroring Session” section in NSX-T via “Advanced Networking & Security” and click “Add”

Define a Local SPAN session:

Define the transport node (Source ESXI host)

Select the physical NIC(s) that participate in the n-vds configuration that we want to capture packets from.

Important: Enable “Encapsulated Packet” (Disabled by default) so we can inspect the underlying Geneve overlay packets. However, the NIC we want to mirror to must support the increased overall packet size. Therefore, adjust the MTU of the nic on the Wireshark host accordingly.

 

Select the source VNIC to capture from – This is the NIC of the Kubernetes worker node VM.

 

 

Finally, select the destination for traffic mirroring. For this example, it’s a NIC on a VM I have Wireshark installed on

Traffic Inspection

With all the hard work out of the way, we can simply provide a filter in Wireshark to show only packets originating from my WordPress Web Pod and terminating at the WordPress DB pod:

Notes:

  • The overall frame size of 1600 further validates our requirement to have the recipient of the traffic mirroring having an interface configured accordingly.
  • Because we previously defined the mirroring of encapsulated packets we can see the overlay information
  • We can see the entire packet structure consisting of:
    • Outer Ethernet (NSX-T Transport Node MAC)
    • Outer IP (NSX-T VTEP IP)
    • Geneve UDP
    • Geneve Data
    • Inner Ethernet (Pod Ethernet addresses)
    • Inner IP (Pod IP addresses)
    • Inner TCP (MYSQL connection)
    • Inner Payload (MYSQL Query) – Highlighted

 

« Older posts Newer posts »

© 2020 Virtual Thoughts

Theme by Anders NorenUp ↑

Social media & sharing icons powered by UltimatelySocial
RSS
Twitter
Visit Us