Virtualisation, Storage and various other ramblings.

Rancher, vSphere Network Protocol Profiles and static IP addresses for k8s nodes

Note : The following currently works only with v2.3.6-rc6 v2.3.6 of Rancher and newer.

Note: As mentioned by Jonathan in the comments, disabling cloud-init’s initial network configuration is recommended. To do this, create a file:

/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg

To contain:

network: {config: disabled}

In your VM template.

How networking configuration is applied to k8s nodes (or VM’s in general) in the on-premises environment is usually achieved by one of two ways – DHCP or static. For some, DHCP is not a popular option and static addresses can be time-consuming to manage, particularly when there’s no IPAM feature in Rancher. In this blog post I go through how to leverage vSphere Network Protocol Profiles in conjunction with Rancher and Cloud-Init to reliably, and predictably apply static IP addresses to deployed nodes using a single node template.

Create the vSphere Network Protocol Profile

Navigate to Datacenter > Configure > Network Protocol Profiles. and click “Add”.

Provide a name for the profile and assign it to one, or a number of port groups.

Next define the network parameters for this port group. The IP Pool and IP Pool Range are of particular importance here – we will use this pool of addresses to assign to our Rancher depoyed K8s nodes.

After adding any other network configuration items the profile will be created and associated with the previously specified port group.

Create the Rancher Node Template

In Rancher, navigate to User > Node Templates > vSphere and configure the parameters to match your environment.

In the cloud-init config, we add a script to extrapolate the ovf environment that vSphere will provide via the Network Profile and configure the underlying os. In this case, Ubuntu 18.04 using Netplan:

Code snippet:

#cloud-config
write_files:
  - path: /root/test.sh
    content: |
        #!/bin/bash
        vmtoolsd --cmd 'info-get guestinfo.ovfEnv' > /tmp/ovfenv
        IPAddress=$(sed -n 's/.*Property oe:key="guestinfo.interface.0.ip.0.address" oe:value="\([^"]*\).*/\1/p' /tmp/ovfenv)
        SubnetMask=$(sed -n 's/.*Property oe:key="guestinfo.interface.0.ip.0.netmask" oe:value="\([^"]*\).*/\1/p' /tmp/ovfenv)
        Gateway=$(sed -n 's/.*Property oe:key="guestinfo.interface.0.route.0.gateway" oe:value="\([^"]*\).*/\1/p' /tmp/ovfenv)
        DNS=$(sed -n 's/.*Property oe:key="guestinfo.dns.servers" oe:value="\([^"]*\).*/\1/p' /tmp/ovfenv)

        cat > /etc/netplan/01-netcfg.yaml <<EOF
        network:
          version: 2
          renderer: networkd
          ethernets:
            ens160:
              addresses: 
                - $IPAddress/24
              gateway4: $Gateway
              nameservers:
                addresses : [$DNS]
        EOF

        sudo netplan apply
runcmd:
  - bash /root/test.sh

What took me a little while to figure out is the application of this feature is essentially a glorified transport mechanism for a bunch of key/value pairs – how they are leveraged is down to external scripting/tooling. VMTools will not do this magic for us.

Next, we configure the vApp portion of the Node Template (how we consume the Network Protocol Profile:

the format is param:portgroup. ip:VDS-MGMT-DEFAULT will be an IP address from the pool we defined earlier – vSphere will take an IP out of the pool and assign it to each VM associated with this template. This can be validated from the UI:

What we essentially do with the cloud-init script is extract this and apply it as a configuration to the VM.

This could be seen as the best of both worlds – Leveraging vSphere Network Profiles for predictable IP assignment whilst avoiding DHCP and the need to implement many Node Templates in Rancher.

17 Comments

  1. Jonathan

    I think it’s important to mention that when using the cloud image of ubuntu 18.04, in order for the above to work it is important to disable the cloud-init mode of netplan in advance, as explained in https://graspingtech.com/ubuntu-server-18.04-static-ip/

    • David

      Good point. The default netplan config could interfere with this.

  2. John Black

    Followed this document, and while I can see the vApp fields being populated in VCenter, the VM OS never gets anything changed. IPs never get applied. I did the cloud init disable as well.

    -Ubuntu 18.04
    -VCenter 6.7

    Also would be nice to know – how to set the hostname of the guest to the VM name. Tried many attempts, but figure it’s the same issue with the networking not applying from these scripts.

    • David

      Hi John,

      The script above works with netplan. Which Linux distro are you using? You may need to tweak it slightly to match the network manager used in the VM.

      • John Black

        Thank you for your reply. I am using https://www.osboxes.org/ubuntu/#ubuntu-1804-vmware – the Bionic Beaver image. If I run your script manually on the image, it applies and works – so I don’t think it’s a script problem, rather an issue getting Vapp properties to apply to Ubuntu. I hear there are issues with VMware customization on Ubuntu distros – but you stated that you did use Ubuntu 18.04 correct? Is there something that has to be done to allow the VMware customization?

        • David

          Hi John,

          The only real requirement is cloud-init and vmtools. The fact you say it works manually implies that vmtools is working as expected. Can you confirm cloud-init is installed?

          • John Black

            Correct, the latest apt updates of open-vm-tools and cloud-init are installed on this version of Ubuntu.
            Anything to do with the 99-disable-network-config.cfg script? should I remove this? Not sure what else to try. Is there something that needs to be done in Rancher or VMware that might not have been so obvious here?

          • John Black

            Hi David – I reached out to VMWare to see if I could get a list of the variables associated with guestinfo. So far I haven’t been successful and haven’t been able to find anything except one obscure API document. Do you happen to have this information you could share to help us add additional functionality here? That would be awesome. Thank you so much for this post. It is great!

          • David

            Hi John,

            Thanks for your kind comments :). My only reference has also been this API document you’ve mentioned.

  3. John Black

    Solved by disabling VMware customization in cloud-init

    • John Black

      Sorry, solved by following VMware KB to set disable_vmware_customizations to false in cloud-init. One final question – is there anyway to set a vApp variable to grab the VM name ? doesn’t look like rancher is changing the cloned template to match the vmname – or this has to be figured out – outside of the vApp?

      • David

        Good question. Rancher will change the name of the VM but not via cloud-init – it’s done via docker machine and the name is based on the node pool in Rancher. If you’re quick enough to log into the VM quickly after it’s cloned it will have the template name briefly. Give it a minute or so and you should see the name change to match the VM name.

  4. Anders Johansson

    Hi,
    This is perhaps a newbie question, but my deploy gets stuck in “Waiting for SSH to be available…” and sadly im not able to see the next logical step.
    What can be the cause of this? I have a hunch that it is the ssh key from rancher that is missing in my deploy.
    A nudge in the right direction would be very appriciated.

    • David

      Hi Anders

      Few things I would check:

      – The VM gets an IP address and is reported in vCenter (Rancher will use what’s reported to SSH into)
      – The VM has cloud-init installed
      – The VM has cloud-init in a cleared state where it’s ready to accept new keys
      – The script above needs adjusting if ens160 is not your VM’s interface name

      • Dan Martin

        Hi David, thanks so much for this article. I’m having the same problem and I’m thinking your 3rd point about cloud-init being in a cleared state might be my problem as I’m good on the other points. How do I go about putting “cloud-init in a cleared state where it’s ready to accept new keys”? I’m using the official ubuntu 18.04 cloud image, imported into VSphere and directly converted to a template (it’s never been booted).

        • David

          Hi Dan,

          The cloud image already has cloud-init cleared, so you shouldn’t need to do anything to it. For other templates I run cloud-init clean –logs (clean cloud init and the associated logs)

          Does vCenter report the VM having an IP address Rancher can route to? Feel free to join us in #vSphere on the community slack channel https://slack.rancher.io/

  5. Jim

    Doing this with Rancher 2.4.5 and using Ubuntu cloud image for focal-20.04. It’s working but seems to take a bit for the VM tools to finally show the IP assigned (it shows in the properties immediately, but takes the tools maybe 3 minutes to display it). Rancher unfortunately seems to still timeout waiting for SSH, even though I can SSH successfully using keys written by cloud-init. I want to ensure it isn’t the network conflict issue. I have attempted to use the write_files section of cloud-init to disable cloud-init’s default networking, but that may be happening too late in the boot process. Is there a way to disable without modifying the cloud-image itself?

Leave a Reply

Your email address will not be published. Required fields are marked *

© 2020 Virtual Thoughts

Theme by Anders NorenUp ↑

Social media & sharing icons powered by UltimatelySocial
RSS
Twitter
Visit Us
Follow Me