One of my favorite things about SaltStack is how easy it is to extend it to add functionality that will make your life easier, after all the whole point of using Salt is to make things easier. Salt gives you all the tools so that all you have to do is focus on the function.
At my job we deploy a lot of small specialized SBC’s that run really two things on top of a stripped down version of linux: a small API that provides functions to other edge equipment and a salt minion. One issue we sometimes have happen is the service gets stuck or stops working for some reason. This is easily fixable by a service restart but we have almost 10k of these things, who has time to watch them all and do that.
Enter our SaltStack HTTP status beacon. The way this works is that SaltStack will launch the beacon on the minion and execute the code we specify, taking the results of the bacon and pushing it back onto the Salt event bus for us to trigger actions against, which in this case is restarting the service automatically! I won’t cover the event portion here but just how to build the beacon.
So let’s jump right into the code. There’s really only one primary important function we have to define which is the def beacon(config) function. This is our entrypoint that Salt will execute
"""
Will query a set of HTTP endpoints and fire events with the results
Example Config
.. code-block:: yaml
beacons:
http:
- checks:
- name: check1
url: http://someendpoint/
code: 200
- interval: 60
"""
import salt.utils.beacons
import salt.utils.http
import logging
log = logging.getLogger(__name__)
def beacon(config):
# Normally the configuration comes in as a top level array so Salt provides this little utility
# just to make it easier to deal with
config = salt.utils.beacons.list_to_dict(config)
# Setting some tight timeouts helps identify stuck servers since they will accept
# the request but never respond, the defaults wait too long
opts = {
"http_connect_timeout": 5,
"http_request_timeout": 5,
}
checks = []
for check in config["checks"]:
log.debug("processing http status beacon check {}".format(check))
# Query this check, I'm disabling the raise_error flag as we don't want an error we just
# want to know the check failed
result = salt.utils.http.query(
url=check["url"],
status=True,
raise_error=False,
opts=opts,
)
log.debug("http status beacon response: {}".format(result))
# Determine if this is a pass or fail based on the status code provided in the configuration
statusCheckPass = True if (check["code"] == result["status"]) else False
checks.append(
{
"tag": check["name"],
"data": {
"url": check["url"],
"name": check["name"],
"code": result["status"],
"pass": statusCheckPass,
"duration": (end - start),
}
}
)
return checks
The code should be fairly straightforward for anyone with any Python experience, and there’s nothing really Salt specific outside of the config and the documentation at the top (you should always add documentation!).
Important: Something that is not made entirely obvious in the documentation is the beacon must return an array of dict objects that must contain a “tag” key. This tag will be affixed to the end of the end tag on the event bus which I’ll show shortly.
Once you’ve written out the beacon the next thing you need to do is sync it to the minions. To load custom beacons you need to place the beacon in a directory named _beacons in wherever your fileserver root is. This can obviously be gitfs or just the roots directory, whatever you have configured.
Next you’ll need to deploy your custom beacon from the master to the minions. Salt gives you a very easy tool to do this, saltutil.sync_beacons which you execute against your minions.
Last we need to enable our new beacon. This can be done via the Pillar or by editing the minion config files directly, either way you choose the format is the same.
beacons:
http:
- checks:
- name: some-service
url: "http://172.16.10.20/"
code: 200
- interval: 60
Once you are done, simply restart the minion and listen on the event bus to see your new event! You can see the tag I noted earlier appended to the full Salt tag
Look out for a future post where I will show how to build simple reactions to failing checks to restart the service!