Tags

, , ,


Overview

When you have 1000’s of hosts in your environment (or even just a few dozen), you need an efficient way to manage package updates. To help me do these each few weeks and when exploits pop up requiring urgent attention, I put a simple process together to perform YUM upgrades on remote hosts using SaltStack. This example is using the single host, ns4.myhost.com.au and shows a real upgrade (and the reboot process if needed) of a host. For multiple hosts see the section at the bottom of the post.

Step 1 – Verify host is online

root@salt:~# salt 'ns4.myhost.com.au' test.ping
ns4.myhost.com.au:
True
root@salt:~#

 

Step 2 – List updates on host

root@salt:~# salt 'ns4.myhost.com.au' pkg.list_upgrades
ns4.myhost.com.au:
----------
e2fsprogs:
1.41.12-18.el6_5.1
e2fsprogs-libs:
1.41.12-18.el6_5.1
grub:
0.97-84.el6_5
initscripts:
9.03.40-2.el6.centos.3
kernel:
2.6.32-431.20.5.el6
kernel-firmware:
2.6.32-431.20.5.el6
libcom_err:
1.41.12-18.el6_5.1
libss:
1.41.12-18.el6_5.1
nspr:
4.10.6-1.el6_5
nss:
3.16.1-4.el6_5
nss-sysinit:
3.16.1-4.el6_5
nss-tools:
3.16.1-4.el6_5
nss-util:
3.16.1-1.el6_5
salt:
2014.1.7-3.el6
salt-minion:
2014.1.7-3.el6
root@salt:~#

Step 3 – Perform upgrades on host

root@salt:~# salt 'ns4.myhost.com.au' pkg.upgrade
root@salt:~#

Step 4 – Verify upgrade

root@salt:~# salt 'ns4.myhost.com.au' pkg.list_upgrades
ns4.myhost.com.au:
----------
root@salt:~#

 

Reboot if needed

If a kernel module is updated and a reboot is required, the following is the simplest method of rebooting the remote host safely. Use a ping running on another host to determine when the host is back online.

root@salt:~# salt 'ns4.myhost.com.au' cmd.run 'shutdown -r now'
ns4.myhost.com.au:
root@salt:~#

While the process is being performed, open a second window and continuously ping the host, especially during the reboot phase.

Update/Upgrade en-mass

To apply this to large groups of servers, a common naming scheme will save you a huge amount of time. In our configuration we have clusters of servers with a common naming scheme. so our shared hosting servers might be shsNNN.myhost.com.au where NNN is the server number. By using an ‘*’ for NNN we can upgrade the lot in one step.

Recently I needed to upgrade a single package, glibc, which had a buffer overrun exploit. The command below did the trick:

salt ‘shs*.myhost.com.au’ cmd.run ‘yum update glibc -y’

1000 servers done! 🙂

 

-oOo-

 

 

Advertisements