Virtual Thoughts

Virtualisation, Storage and various other ramblings.

Page 2 of 16

Application security with mutual TLS (mTLS) via Istio

TLS Overview

If we take an example of accessing a website such as https://www.virtualthoughts.co.uk/, these are the high-level steps of what occurs:

 

 

 

  1. The client initiates a connection to the web server requesting an HTTPS connection.
  2. The web server responds with its public key. The client validates the key with its list of known Certificate Authorities.
  3. A session key is generated by the client and encrypted with the web server’s public key and is sent back to the web server.
  4. The web server decrypts the session key with its private key. End to end encryption is established.

By default, the TLS protocol only proves the identity of the server to the client using X.509 certificate and the authentication of the client to the server is left to the application layer.  For external, public-facing websites, this is an acceptable and well-established implementation of TLS. But what about communication between different microservices?

 

As opposed to monolithic applications, microservices are usually inter-connected which allow them to be scaled/modified/etc independently. But this does raise some challenges. For example:

  • How do we ensure service-to-service communication is always encrypted?
  • How can do we do this without changing the application source code?
  • How can we automatically secure communication when we introduce a new service to an application?
  • How can we authenticate clients and servers and fully establish a “zero trust” network policy?

Istio can help us address these challenges:

Example Application

To demonstrate Istio’s mTLS capabilities a WordPress Helm chart was deployed into a namespace with automatic sidecar injection. Installing and configuring Istio can be found on a previous blog post. By default, the policy specifies no mTLS between the respective services. As such, the topology of the solution is depicted below:

 

 

We can validate this by using Istioctl:

 

All of the “testsite” services (WordPress frontend and backend) Envoy proxies are using HTTP as their transport mechanism. Therefore mTLS has not been configured yet.

Creating Istio Objects – Policy and Destination Rules

As you might expect, establishing mutual TLS (mTLS) is a two-part process, First, we must configure the clients to leverage mTLS, as well as the servers. This is accomplished with Policy and Destination rules.

Policy (AKA – what I, the server, will accept)

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
name: "default"
namespace: "wordpress-app"
spec:
peers:
- mtls: {}

This example policy strictly enforces only mTLS connections between services within the “wordpress-app” namespace

DestinationRule (AKA – what I, the client, will send out)

 apiVersion: "authentication.istio.io/v1alpha1"
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
name: "vt-wordpress-mariadb"
namespace: "wordpress-app"
spec:
host: "*.wordpress-app.svc.cluster.local"
trafficPolicy:
tls:
mode: ISTIO_MUTUAL

This example enforces the use of mutual TLS when communicating with any service in the wordpress-app namespace. Applying these and re-running the previous istioctl command yields the following result:

This is accomplished largely due to Citadel – a component in the Istio control plane that manages certificate creation and distribution:

When mTLS is configured the traffic flow (from a high level) can be described as follows:

  • Citadel provides certificates to the sidecar pods and manages them.
  • WordPress pod creates a packet to query the MYSQL database.
    • WordPress Envoy sidecar pod intercepts this and establishes a connection to the destination sidecar pod and presents its certificate for authenticity.
  • MYSQL Envoy sidecar pod receives a connection request, validates the client’s certificate and sends its own back.
  • WordPress Envoy sidecar pod receives MYSQL’s certificate and checks it for authenticity.
  • Both proxies are in agreement as to each other’s identity and establish an encrypted tunnel between the two.

This is what makes it “mutual” TLS. In effect, both services are presenting, inspecting and validating each other’s certificate as a prerequisite for service-to-service communication. This differs from a standard HTTPs site described earlier on where only the client was validating the server.

Additional Comments

Some additional observations I’ve made from this exercise

  • If enforcing strict mTLS on a service that’s exposed externally from a load balancer, your clients will obviously need to send x509 certificates that can be validated by Citadel. A more flexible alternative to this is to employ an Istio gateway that provides TLS termination at the cluster boundary. This negates the need to provision x509 certs to each and every client, whilst maintaining mTLS within the cluster.
  • Envoy sidecar pods can affect liveness probes and might require you to implement
     sidecarInjectorWebhook.rewriteAppHTTPProbe=true 

    upon installing Helm 

Container Packet Inspection with NSX-T

For troubleshooting (or just being a bit nosey) we have a number of tools that allow us to inspect the traffic between two endpoints. When it comes to containers however, our approach has to be adjusted slightly. When using NSX-T as a CNI, we have some of these tools available to us out of the box.

App Overview

 

For this example, I’ve deployed a standard WordPress deployment consisting of a frontend (web) pod and a backend (DB) pod. The objective is to identify and capture the traffic between these pods.

 

Configure NSX-T Port Mirroring

Navigate to the “Port Mirroring Session” section in NSX-T via “Advanced Networking & Security” and click “Add”

Define a Local SPAN session:

Define the transport node (Source ESXI host)

Select the physical NIC(s) that participate in the n-vds configuration that we want to capture packets from.

Important: Enable “Encapsulated Packet” (Disabled by default) so we can inspect the underlying Geneve overlay packets. However, the NIC we want to mirror to must support the increased overall packet size. Therefore, adjust the MTU of the nic on the Wireshark host accordingly.

 

Select the source VNIC to capture from – This is the NIC of the Kubernetes worker node VM.

 

 

Finally, select the destination for traffic mirroring. For this example, it’s a NIC on a VM I have Wireshark installed on

Traffic Inspection

With all the hard work out of the way, we can simply provide a filter in Wireshark to show only packets originating from my WordPress Web Pod and terminating at the WordPress DB pod:

Notes:

  • The overall frame size of 1600 further validates our requirement to have the recipient of the traffic mirroring having an interface configured accordingly.
  • Because we previously defined the mirroring of encapsulated packets we can see the overlay information
  • We can see the entire packet structure consisting of:
    • Outer Ethernet (NSX-T Transport Node MAC)
    • Outer IP (NSX-T VTEP IP)
    • Geneve UDP
    • Geneve Data
    • Inner Ethernet (Pod Ethernet addresses)
    • Inner IP (Pod IP addresses)
    • Inner TCP (MYSQL connection)
    • Inner Payload (MYSQL Query) – Highlighted

 

Step by Step – Istio up and running

Service Mesh is a pretty hot topic in the Kubernetes ecosystem currently, and I wanted to get it up and running in my own lab environment. Istio’s documentation has a pre-baked solution to demonstrate some of its capabilities (a book app, if memory serves me correctly), but I wanted to deploy my own app to get more “hands-on” experience with the tech, even if it’s only very basic to start with.

Install Istio

There are a number of prerequisite steps that need to be satisfied prior to installing Istio. These are specific to my environment,  others may differ.

Install the Helm client

 sudo snap install helm --classic 

Grab Istio (1.2.0 in this example)

curl -L https://git.io/getLatestIstio | ISTIO_VERSION=1.2.0 sh - cd istio-1.2.0/

Create the Helm service account (named “tiller”)

kubectl apply -f install/kubernetes/helm/helm-service-account.yaml

Initialise Helm  using the service account specified in the previous step

helm init --service-account tiller

Create a namespace to accommodate the Istio components

kubectl create ns istio-system

Initialise Istio into the aforementioned namespace:

helm install install/kubernetes/helm/istio-init --name istio-init --namespace istio-system

Monitor the state of the pods – it will take some time for the pods to finish – these create the CRD’s required for Istio

kubectl get pods -n istio-system
NAME                      READY   STATUS              RESTARTS   AGE
istio-init-crd-10-t82m6   0/1     ContainerCreating   0          95s
istio-init-crd-11-42622   0/1     ContainerCreating   0          95s
istio-init-crd-12-65m5v   0/1     ContainerCreating   0          95s

Install Istio into the aforementioned namespace

helm install install/kubernetes/helm/istio --name istio --namespace istio-system

Configure a namespace for automatic sidecar injection

By this point, we have the internal foundations for Istio, but we’re not leveraging it. One of the fundamental workings of Istio is the use of pod sidecars. Sidecars act as the data plane, facilitating a lot of the features we want to leverage from Istio.

The overall architecture of an Istio-based application.

Istio doesn’t do this automatically, out of the box for all pods deployed into an environment, but Istio will inject sidecars into pods deployed into namespaces that have the istio-injection=enabled label set.

kubectl create ns app-with-injection
namespace/app-with-injection created
kubectl label namespace app-with-injection istio-injection=enabled
namespace/app-with-injection labeled

We can validate this by creating a pod into this namespace:

kubectl run nginx -n app-with-injection --image nginx

And checking the Pod contents (notice how this pod has two containers)

kubectl get pods -n app-with-injection
NAME                     READY   STATUS    RESTARTS   AGE
nginx-7cdbd8cdc9-96mbz   2/2     Running   0          52s

The proxy sidecar:

 

Environment Anatomy

The diagram below shows how my test environment is set up

 

 

Key considerations:

The Istio gateway will reside on the edge
80% of all traffic will be routed to v1 of my web application
20% of all traffic will be routed to v2 of my web application

The application manifest can be found at https://raw.githubusercontent.com/David-VTUK/istioexample/master/webapp.yaml

To accomplish this we need to implement two key objects:

Gateway

This is our entry point into our application. By default, Istio deploys the gateway object (we must note the external IP)

kubectl get svc -n istio-system
NAME                     TYPE           CLUSTER-IP       EXTERNAL-IP                
istio-citadel            ClusterIP      10.100.200.47                        
istio-galley             ClusterIP      10.100.200.149                       
istio-ingressgateway     LoadBalancer   10.100.200.244   10.10.20.150,100.64.80.1   
istio-pilot              ClusterIP      10.100.200.170                       
istio-policy             ClusterIP      10.100.200.3                        
istio-sidecar-injector   ClusterIP      10.100.200.169                   
istio-telemetry          ClusterIP      10.100.200.141                      
prometheus               ClusterIP      10.100.200.238                     

We configure the Gateway by deploying a gateway manifest file:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: http-gateway
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"              
  • Kind : Type of object. Gateway is a CRD (Custom Resource Definition) that Istio implements
  • Selector: What this applies to, in this case the default Ingress Gateway
  • Ports: Which ports we want to listen to on the external IP address, together with a name and protocol
  • Hosts : We can implement layer 7 load balancing on the edge, but as I’ll be testing this out via IP address, “*” will suffice. In production, this would likely be an FQDN of an external facing website

 

VirtualService

gateway object helps us define the entry point into the cluster, but we have yet to effectively tell the gateway where to route traffic to. This is where the VirtualService object type comes in. This is where we define routing intelligence.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: demoapp
spec:
  hosts:
  - "*"
  gateways:
  - http-gateway
  http:
  - route:
    - destination:
        port:
          number: 80
        host: vt-webapp-v1.app-with-injection.svc.cluster.local
      weight: 80
    - destination:
        port:
          number: 80
        host: vt-webapp-v2.app-with-injection.svc.cluster.local
      weight: 20      

What the above effectively does is listen for all HTTP requests (hence the “*” under “hosts”) and route 80% of traffic to V1 of the webapp, by directing traffic at the respective service and 20% to v2.

Testing

The “WebApp” is pretty simple. It simply displays one of the following (depending on the version)

 

What we should now see from accessing the external IP is traffic being split across both services via a 80/20 split:

Out of 10 curl commands 8 were routed to v1 of my app, 2 were routed to v2 of my app.

Conclusion

Admittedly, this is an extremely simple example of a more simple use case of Istio, but as I’m learning, I think it’s a decent start, and I hope others find it useful.

« Older posts Newer posts »

© 2019 Virtual Thoughts

Theme by Anders NorenUp ↑

Social media & sharing icons powered by UltimatelySocial
RSS
Twitter
Visit Us