, ,

Everyday I use Saltstack to do management of a large Cloud Infrastructure Platform, it has some great features for deploying code to servers and one key feature is deploying scripts that need to run under CRON.

However, a script that runs on 500 servers at the same time and does a lot of disk I/O can cause issues on clustered storage. To lighten the load it is handy to offset the start times of CRON jobs so they don’t all kick in at the same time and create an I/O storm.

By using a random values, for the minute (and even the hour) values in the CRONTAB entry and limiting the range to a set time scale, the script can be pushed to multiple servers safely knowing they will run at different times as needed.

For a random value between 1 and 59 I use: {{ range(1,59) | random }} in the template as shown below. If you want to randomize the hour between 2am and 5am you could use {{ range(2,5) | random }}.

The my-script install.sls file:

 - user: root
 - name: '/usr/local/bin/my-script.sh > /dev/null 2>&1'
 - minute: {{ range(1,59) | random }}
 - hour: 3
 - daymonth: '*'
 - month: '*'
 - dayweek: '*'
 - identifier: 'id-x-cron-root'
 - comment: 'Scan files using my-script as root'