Virtualisation, Storage and various other ramblings.

Rancher, vSphere Network Protocol Profiles and static IP addresses for k8s nodes

Note : The following currently works only with v2.3.6-rc6 v2.3.6 of Rancher and newer.

Note: As mentioned by Jonathan in the comments, disabling cloud-init’s initial network configuration is recommended. To do this, create a file:

/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg

To contain:

network: {config: disabled}

In your VM template.

How networking configuration is applied to k8s nodes (or VM’s in general) in the on-premises environment is usually achieved by one of two ways – DHCP or static. For some, DHCP is not a popular option and static addresses can be time-consuming to manage, particularly when there’s no IPAM feature in Rancher. In this blog post I go through how to leverage vSphere Network Protocol Profiles in conjunction with Rancher and Cloud-Init to reliably, and predictably apply static IP addresses to deployed nodes using a single node template.

Create the vSphere Network Protocol Profile

Navigate to Datacenter > Configure > Network Protocol Profiles. and click “Add”.

Provide a name for the profile and assign it to one, or a number of port groups.

Next define the network parameters for this port group. The IP Pool and IP Pool Range are of particular importance here – we will use this pool of addresses to assign to our Rancher depoyed K8s nodes.

After adding any other network configuration items the profile will be created and associated with the previously specified port group.

Create the Rancher Node Template

In Rancher, navigate to User > Node Templates > vSphere and configure the parameters to match your environment.

In the cloud-init config, we add a script to extrapolate the ovf environment that vSphere will provide via the Network Profile and configure the underlying os. In this case, Ubuntu 18.04 using Netplan:

Code snippet:

#cloud-config
write_files:
  - path: /root/test.sh
    content: |
        #!/bin/bash
        vmtoolsd --cmd 'info-get guestinfo.ovfEnv' > /tmp/ovfenv
        IPAddress=$(sed -n 's/.*Property oe:key="guestinfo.interface.0.ip.0.address" oe:value="\([^"]*\).*/\1/p' /tmp/ovfenv)
        SubnetMask=$(sed -n 's/.*Property oe:key="guestinfo.interface.0.ip.0.netmask" oe:value="\([^"]*\).*/\1/p' /tmp/ovfenv)
        Gateway=$(sed -n 's/.*Property oe:key="guestinfo.interface.0.route.0.gateway" oe:value="\([^"]*\).*/\1/p' /tmp/ovfenv)
        DNS=$(sed -n 's/.*Property oe:key="guestinfo.dns.servers" oe:value="\([^"]*\).*/\1/p' /tmp/ovfenv)

        cat > /etc/netplan/01-netcfg.yaml <<EOF
        network:
          version: 2
          renderer: networkd
          ethernets:
            ens160:
              addresses: 
                - $IPAddress/24
              gateway4: $Gateway
              nameservers:
                addresses : [$DNS]
        EOF

        sudo netplan apply
runcmd:
  - bash /root/test.sh

What took me a little while to figure out is the application of this feature is essentially a glorified transport mechanism for a bunch of key/value pairs – how they are leveraged is down to external scripting/tooling. VMTools will not do this magic for us.

Next, we configure the vApp portion of the Node Template (how we consume the Network Protocol Profile:

the format is param:portgroup. ip:VDS-MGMT-DEFAULT will be an IP address from the pool we defined earlier – vSphere will take an IP out of the pool and assign it to each VM associated with this template. This can be validated from the UI:

What we essentially do with the cloud-init script is extract this and apply it as a configuration to the VM.

This could be seen as the best of both worlds – Leveraging vSphere Network Profiles for predictable IP assignment whilst avoiding DHCP and the need to implement many Node Templates in Rancher.

12 Comments

  1. Jonathan

    I think it’s important to mention that when using the cloud image of ubuntu 18.04, in order for the above to work it is important to disable the cloud-init mode of netplan in advance, as explained in https://graspingtech.com/ubuntu-server-18.04-static-ip/

    • David

      Good point. The default netplan config could interfere with this.

  2. John Black

    Followed this document, and while I can see the vApp fields being populated in VCenter, the VM OS never gets anything changed. IPs never get applied. I did the cloud init disable as well.

    -Ubuntu 18.04
    -VCenter 6.7

    Also would be nice to know – how to set the hostname of the guest to the VM name. Tried many attempts, but figure it’s the same issue with the networking not applying from these scripts.

    • David

      Hi John,

      The script above works with netplan. Which Linux distro are you using? You may need to tweak it slightly to match the network manager used in the VM.

      • John Black

        Thank you for your reply. I am using https://www.osboxes.org/ubuntu/#ubuntu-1804-vmware – the Bionic Beaver image. If I run your script manually on the image, it applies and works – so I don’t think it’s a script problem, rather an issue getting Vapp properties to apply to Ubuntu. I hear there are issues with VMware customization on Ubuntu distros – but you stated that you did use Ubuntu 18.04 correct? Is there something that has to be done to allow the VMware customization?

        • David

          Hi John,

          The only real requirement is cloud-init and vmtools. The fact you say it works manually implies that vmtools is working as expected. Can you confirm cloud-init is installed?

          • John Black

            Correct, the latest apt updates of open-vm-tools and cloud-init are installed on this version of Ubuntu.
            Anything to do with the 99-disable-network-config.cfg script? should I remove this? Not sure what else to try. Is there something that needs to be done in Rancher or VMware that might not have been so obvious here?

          • John Black

            Hi David – I reached out to VMWare to see if I could get a list of the variables associated with guestinfo. So far I haven’t been successful and haven’t been able to find anything except one obscure API document. Do you happen to have this information you could share to help us add additional functionality here? That would be awesome. Thank you so much for this post. It is great!

          • David

            Hi John,

            Thanks for your kind comments :). My only reference has also been this API document you’ve mentioned.

  3. John Black

    Solved by disabling VMware customization in cloud-init

    • John Black

      Sorry, solved by following VMware KB to set disable_vmware_customizations to false in cloud-init. One final question – is there anyway to set a vApp variable to grab the VM name ? doesn’t look like rancher is changing the cloned template to match the vmname – or this has to be figured out – outside of the vApp?

      • David

        Good question. Rancher will change the name of the VM but not via cloud-init – it’s done via docker machine and the name is based on the node pool in Rancher. If you’re quick enough to log into the VM quickly after it’s cloned it will have the template name briefly. Give it a minute or so and you should see the name change to match the VM name.

Leave a Reply

Your email address will not be published. Required fields are marked *

© 2020 Virtual Thoughts

Theme by Anders NorenUp ↑

Social media & sharing icons powered by UltimatelySocial
RSS
Twitter
Visit Us
Follow Me