Virtualisation, Storage and various other ramblings.

On-prem K8s clusters with Rancher, Terraform and Ubuntu

One of the attractive characteristics of Kubernetes is how it can run pretty much anywhere – in the cloud, in the data center, on the edge, on your local machine and much more. Leveraging existing investments in datacenter resources can be logical when deciding where to place new Kubernetes clusters, and this post goes into automating this with Rancher and Terraform.

Primer

For this exercise the following is leveraged:

  • Rancher 2.3
  • vSphere 6.7
  • Ubuntu 18.04 LTS

An Ubuntu VM will be created and configured into a template to spin up Kubernetes nodes.

Step 1 – Preparing a Ubuntu Server VM

In Rancher 2.3 Node templates for vSphere can leverage either of the following:

For the purposes of this demo, "Deploy from template" will be used, given its simplicity.

To create a new VM template, we must first create a VM. Right-click an appropriate object in vCenter and select "New Virtual Machine"

Select a source:

Give it a name:

Give it a home (compute):

Give it a home (storage):

Specify the VM hardware version:

Specify the guest OS:

Configure the VM properties, ensure the Ubuntu install CD is mounted:

After this, power up the VM and walk through the install steps. After which it can be turned into a template:

Rancher doesn’t have much in the way of requirements for the VM. For this install method a VM needs to have:

  • Cloud-Init (Installed by default on Ubuntu 18.04).
  • SSH connectivity (Rancher will provide its own SSH certificates as per Cloud-Init bootstrap) – Ensure SSH server has been installed.

A Note on Cloud-Init

For Vanilla Ubuntu Server installs, it uses Cloud-Init as part of the general Installation process. As such, cloud-init can not be re-invoked on startup by default. To get around this for templating purposes, the VM must be void of the existing cloud-init configuration prior to being turned into a template. To accomplish this, run the following:

sudo rm -rf /var/lib/cloud/instances

Before shutting down the VM and converting it into a template.

Constructing the Terraform Script

Now the VM template has been created it can be leveraged by a Terraform script:

Specify the provider: (Note – insecure = "true" Is required for vCenter servers leveraging an untrusted certificate, such as self-signed.

provider "rancher2" {
  api_url    = "https://rancher.virtualthoughts.co.uk"
  access_key = #ommited - reference a Terraform varaible/environment variable/secret/etc
  secret_key = #ommited - reference a Terraform varaible/environment variable/secret/etc
  insecure = "true"
}

Specify the Cloud Credentials:

# Create a new rancher2 Cloud Credential
resource "rancher2_cloud_credential" "vsphere-terraform" {
  name = "vsphere-terraform"
  description = "Terraform Credentials"
  vsphere_credential_config {
    username = "Terraform@vsphere.local"
    password = #ommited - reference a Terraform varaible/environment variable/secret/etc
    vcenter = "svr-vcs-01.virtualthoughts.co.uk"
  }
}

Specify the Node Template settings:

Note we can supply extra cloud-config options to further customise the VM, including adding additional SSH keys for users.

resource "rancher2_node_template" "vSphereTestTemplate" {
  name = "vSphereTestTemplate"
  description = "Created by Terraform"
  cloud_credential_id = rancher2_cloud_credential.vsphere-terraform.id
   vsphere_config {
   cfgparam = ["disk.enableUUID=TRUE"]
   clone_from = "/Homelab/vm/Ubuntu1804WithCloudInit"
   cloud_config = "#cloud-config\nusers:\n  - name: demo\n    ssh-authorized-keys:\n      - ssh-rsa [SomeKey]
   cpu_count = "4"
   creation_type = "template"
   disk_size = "20000"
   memory_size = "4096"
   datastore = "/Homelab/datastore/NFS-500"
   datacenter = "/Homelab"
   pool = "/Homelab/host/MGMT/Resources"
   network = ["/Homelab/network/VDS-MGMT-DEFAULT"]
   }
}

Specify the cluster settings:

resource "rancher2_cluster" "vsphere-test" {
  name = "vsphere-test"
  description = "Terraform created vSphere Cluster"
  rke_config {
    network {
      plugin = "canal"
    }
  }
}

Specify the Node Pool:

resource "rancher2_node_pool" "nodepool" {

  cluster_id =  rancher2_cluster.vsphere-test.id
  name = "all-in-one"
  hostname_prefix =  "vsphere-cluster-0"
  node_template_id = rancher2_node_template.vSphereTestTemplate.id
  quantity = 1
  control_plane = true
  etcd = true
  worker = true
}

After which the script can be executed.

What’s going on?

From a high level the following activities are being executed:

  1. Rancher requests VM’s from vSphere using supplied Cloud Credentials.
  2. vSphere clones the VM Templateeverywhere with the specified configuration parameters.
  3. An ISO image is mounted to the VM, which contains certificates and configuration generated by Rancher in the cloud-init format.
  4. Cloud-Init on startup reads this ISO image and applies the configuration.
  5. Rancher builds the Kubernetes cluster by Installing Docker and pulling down the images.

After which, a shiny new cluster will be created!

11 Comments

  1. DANIEL FERNANDES VILAR CARDOSO

    Hi, pretty good post, where should i put the terraform script? at rancher or run from my machine?
    thanks!!!

    • David

      Hi Daniel,

      Run from your local machine that has access to the rancher URL.

  2. anon

    Hi David,
    what steps should be taken with the VM to generalize it before turning it into a template?

    • David

      Not much, sudo rm -rf /var/lib/cloud/instances before turning it into a VM should be enough.

  3. Mike

    Hi
    Thanks for your post!
    Sorry to ask what is probably a stupid question, but I thought Rancher could deploy nodes/clusters from i.e. an ‘unmodified’ Ubuntu installed vSphere template. I thought Rancher could, using Cloud Init or equivalent, modify the VM (hostname, certificates, kubernetes packages, etc) without the need for customization using Teraform, Ansible or other tools, but I havn’t worked with Rancher yet, so I’ve probably misunderstood.
    I’m having trouble figuring out precisely how to prepare the vSphere templates for Rancher vSphere cluster deployment and hoped you could point me to some descriptions.
    Thanks πŸ™‚

    • David

      Hi Mike,

      Not a stupid question at all. Rancher can indeed deploy nodes/clusters from an unmodified Ubuntu template or similar that’s provisioned in vSphere. It will modify the VM post-cloning by passing its own SSH key as part of the cloud-init userdata bundle. Docker-Machine is actually what (under the hood) changes the hostname and does a lot of the legwork setting docker up on the node.

      I’m going to write an article on setting up a “golden image for vSphere templates + Rancher”, but from a high level, a template should:

      – Have the HW version set to that of the vSphere version
      – Cloud init installed
      – Cloud init reset to a state that’s accepting of additional config (sudo rm -rf /var/lib/cloud/instances)
      – VM Tools installed and up to date

      PS – feel free to join the #vSphere channel on https://slack.rancher.io/

      • Mike

        Hi David,

        Thanks for taking the time πŸ™‚ I’ve joined the #vSphere channelπŸ‘
        So apart from open-vm-tools, DHCP and sudo rm -rf /var/lib/cloud/instances, there is nothing I need i guess.
        But do I then need to configure the cloud init script in the ‘inline editor’ in Ranncher Node Template, and are you aware of some examples?
        Thanks again πŸ™‚

        • Olivier

          Dear,

          Would like to see more example too !
          Struggling to master all the interaction between template & Rancher & docker setup in order to run rke.

          Nice article !

          Thanks

  4. SC

    Hi,
    Can we leverage VMware NSX-T for Rancher k8s, if not what would be best options for Container Networking in Rancher

  5. vasya

    Very useful article! Thanks a lot!

  6. Sameh

    Hello,
    thanks a lot for this article, it’s very helpful
    But I wish know more if this will provision nodes that have vsphere CPI & CSI installed by default or should I after provisioning this cluster install it on my master nodes ?

Leave a Reply to Mike Cancel reply

Your email address will not be published. Required fields are marked *

© 2024 Virtual Thoughts

Theme by Anders NorenUp ↑

Social media & sharing icons powered by UltimatelySocial
RSS
Twitter
Visit Us
Follow Me