Ansible: Let me put you on patch game

Greg Heffner August 7, 2024

Bash, powershell, python, oh my. Or should I say RDP, SSH, winrm, oh my? Either way, patching is a necessary evil.

I found ansible when I was looking for a way to patch. I like many others, had a major problem. I have all sorts of devices on my network with no easy way to update them all. Ansible enables automation across all your devices. The community distribution of Ansible contains powerful tools that are supported on most operating systems like RHEL, Ubuntu, MacOS, and Microsoft Windows to name a few.

In this blog I am going to walk you through installing ansible, patching three ubuntu vms, and setting up logging, all while maintaining high availability of a website in a kubernetes cluster.

Installing ansible

Ansible has many ways to be installed. I took advantage of using a Brew Homebrew Formula due to its ease of use. First make sure you have python installed or you can follow these Python Install Instructions. Once python is installed install ansible with "brew install ansible".

Once you have ansible installed the most important step is to setup a host file. A host file is used to identify hosts you want to interact with. You do not need a host file on every device, only the main device you will be running your playbooks from. Check out this video where NetworkChuck explains how to setup a host file.

Ansible allows you to group devices in six different ways:

• IP addresses: 192.168.1.10 or 192.168.1.11
• Hostname: web1.example.com or web2.example.com
• Custom Groups: webservers, databases, all linux
• Variables: web1, web2, web3
• Patterns: 192.168.1.[10:13]
• Ranges: web[1:3].example.com

Once all your hosts have ansible installed on them and you have your host file setup you need to get access taken care of. If you are running ansible on one machine and not doing any type of remote scripting the you dont have to worry about this step. Lucky for us setting up ssh keys on devices isnt that hard. Follow this article from Digital Ocean for a refresher on how to create and share ssh keys.

Once you have everything installed and SSH is setup, an easy way to check if everything works is to run an ansible ping test. I have a group named K8s_cluster I am testing against in my host file.

IT WORKS!!!! As you can see ansible reached out to my devices gave a ping, received a pong, and then went to the next device. If you are not able to ping your devices in this way please check your host file, your network connections to include firewall settings, and you user authentication.

Running your playbook (patching)

Ok the fun part, git clone k8-patchNbounce to follow along. This playbook is broken into two phases. The first phase runs apt update and apt upgrade, and the second phase reboots the devices maintaining high availability. YAML Aint Markup Language, or YAML for short is the language used by Ansible for their playbooks. YAML uses key value pairs when writing just like JSON but unlike JSON, YAML is easy to understand.

Update & Upgrade: As seen in the screenshot this phase of the playbook loops through the host group named k8s_cluster. The first command that is seen updates all apt packages on the device. When you run the command normally from the command line, remember, you have to have sudo privileges to run the update. Ansible allows us to do this in the playbook by passing "become:yes" to use sudo privileges. The packages listed to update with apt is * (or all) but you can be specific here if you like. Next step is to run apt upgrade again using sudo privileges. The upgrade is the dist-upgrade check out this link for info about why I chose this. The final piece in this phase is the logging piece. Once the playbook is run against a device I append the devices hostname and the date it was patched in a log locally on the device.

Reboot: This phase of the playbook loops through the same k8s_cluster group but it is only run one device at a time by using the "serial: 1" command. Since these specific ubuntu nodes in my playbook are in a kubernetes cluster I need to do extra steps to keep a website online. The first command I run is a "kubectl drain" command. This command cordons and drains k8s pods from the node I have selected to a device in the cluster. Once the pods have moved to a new node, then the systemctl reboot command is run on the device again as sudo. You want to move pods to a node that isnt being bounced to keep from having an outage. Check out my blog about Kubernetes here.

High Availability: You can skip this section if you dont care about patching kubernetes ubuntu nodes that live in a cluster. I split this section out due to how important it is. A website cannot afford downtime. Before the playbook moves to the next device in the host file, ansible waits for the SSH connection to become available on the device it just bounced. Why? Well, we told it to in the playbook but the SSH service is started later in the boot sequence which would mean once the service becomes available then successful boot could be validated. You can change this service check for any other service or a better health check if you like. Ansible will pause for 5 seconds after validation and then un-cordon the node which allows for pods to be hosted on it in the K8s cluster. Since this phase of the playbook was run in serial we then jump back up to the reboot steps on the next device in the cluster ultimately bouncing all the nodes and bringing them back online. If any issues occur which would keep SSH from starting or the device form coming online ansible will not move on to the next device and the playbook will error in the logs before all the other nodes go down.

Scheduling

Youre so fancy now patching with ansible. How do I set it up so it happens without having to do a thing? Cronjob. There is all sorts of ways you can schedule script to be run but I find it the easiest to run it in cron. Its nice.

Ansible from Bing AI

Ansible is like a magical helper for computers. Imagine you have a bunch of toy robots, and you want them to do different tasks, like cleaning up toys or dancing. Instead of telling each robot what to do individually, you can use Ansible to give them all the same instructions at once! 🤖✨

Heres how it works:

Instructions (Playbooks): Ansible uses special instructions called “playbooks.” These playbooks tell the robots (servers or computers) what tasks to perform. For example, “Clean up your room!” or “Install a new game.”

Inventory (Guest List): Before the party (or task), you create a guest list (inventory). This list includes the names of all your robots (servers). Ansible knows whos invited to the party!

Magic Spells (Modules): Ansible has magic spells (modules) for different tasks. For cleaning, it uses the “sweep” spell; for installing software, it uses the “install” spell. You just say which spell you want, and Ansible makes it happen!

Waving the Wand (Running Playbooks): When the party starts, you wave your magic wand (run the playbook). Ansible goes to each robot and says, “Hey, clean up!” or “Install this game!” And poof! The robots follow your instructions.

Parallel Magic: The cool part? Ansible can talk to all the robots at once! It doesn’t need to visit each one individually. So, tasks happen quickly, like a synchronized dance.

Remember, Ansible doesnt wear a wizard hat, but it sure feels magical! 🎩✨ If you need something done in your computer world, just call on Ansible!