Fabric describes itself as a “library and command-line tool for
streamlining the use of SSH for application deployment or systems
administration tasks”. To get this work done, Fabric uses what are called
“fabfiles’, which is just some python stored in a file called fabfile.py. In
terms of execution, it works a lot like Rake from the Ruby world. You have a
base command, in this case fab and each task is read from the fabfile.py
and executed on request. Nothing too earth shattering.
The bit that I’m enjoying about Fabric is that you don’t have to deal with the low level transport stuff, like how to get to the system, or execute a command in sudo with the correct arguments so that STDERR comes back over the channel in a reasonable format, etc. Fabric has all of this built in using python methods to wrap the execution model which does the actual processing of which hosts to perform which actions. Combined with a system to assign “roles”, you can then execute A tasks for the A cluster, while running only B tasks on the B cluster, or others that you might run on all hosts.
Retrieving Hosts from PuppetDB
Prior to executing any of the tasks, you can load a list of hosts however you want. If you have a CMDB, you might use that as your source of truth. However, we have Puppet. All our nodes check into Puppet and dump their facts, status, and their classification information into PuppetDB.
With just a little Python, you can load everything you need form PuppetDB and return the data to the Fabric environment before the tasks are loaded, giving you PuppetDB as the source of data, and Fabric as the execution framework for performing actions.
If I drop the following into the file puppetdb/__init__.py, you can then use
this as a library for retrieving data from PuppetDB
import json
from fabric.api import *
pe_puppetdb = 'puppetdb.lab.example.com'
def query_puppetdb(puppetdb_server,query):
h_list = list()
with hide('everything'):
with settings(host_string=puppetdb_server):
data = run('curl -X GET "http://localhost:8080/v3/nodes"
--data-urlencode \'query=' + query + '\'')
for h in json.loads(data):
host = str(h['name'])
h_list.append(host)
return list(h_list)
def list():
query = env.config['host_query']
h_list = query_puppetdb(pe_puppetdb, query)
return list(h_list)
print("Found:")
for h in h_list:
print(" - " + h)
return list(h_list)
To make the list returned actually useful, you just need to call it and load it up into your runtime environment, and to do this I’ve been using the following pattern.
First I store the query somewhere useful and well formatted.
json_query = [
"and",
["=", ["fact", "domain"], "lab.example.com"],
["=", ["fact", "operatingsystem"], "Debian"],
]
Load the query and some surrounding metadata into a dict blob.
env.config = {
'host_query': json.dumps(json_query)
}
env.config = {
'puppet': {
'pe': {
'ca': 'peca.lab.example.com',
'nodes': puppetdb.list()
}
}
}
Load the roles out of the configuration dict and into the runtime environment.
``python env.roledefs[‘pe_nodes’] = env.config[‘puppet’][‘pe’][’nodes’]
and now we can just do whatever I want with the role. In the simplest of
cases, you might do something like the following to retrieve the uptime from
each system.
```python
@roles('pe_nodes')
@task
def uptime():
run('uptime')
Now fab uptime will execute the ‘uptime’ task on each node returned from
PuppetDB, which in turn just executes the command uptime and returns the
output. Pretty simple.
Why this is Useful
Using this process, I can get around the limitations placed on me by the configuration management tool. That the code is Python can be as awesome or as simple as you want it to be, which is simply awesome. I don’t need to know the semantics of this configuration management, now I only must know a little Python and can be quite effective. With Puppet’s focus on security and centralized control over the infrastructure, someitmes the workflow can get in the way, but sometimes you just need something procedural and out of band.
A few of the things I am using this for are:
- Convert Puppet open source nodes to PE
- Reset a LDAP dev environment from production
- Execute R10k on all the PE and FOSS Puppet masters
- Future HeartBleed-esque SSL re-key
Hooking this up to our CI system has been a nice convenience. I can run the commands locally from my laptop using my credentials, and when I commit to the repo and push it out, the CI system updates the jobs on disk and begins executing the new tools. Once the jobs are setup in the CI system, its just a simple:
#! /bin/bash
cd /opt/fabricrepo
fab run_my_task
Now all of the jobs are actually stored in git and the CI configuration is just for scheduling and notification, centralized access control, etc.
So what?
Lets recap about what each tool gives us.
Using Puppet we get:
All the information about what is being managed on a system us applied through manifests, thus we know what is intended to be managed, and what the intended state is.
The ability to inspect all resource types (packages, services, files, users) on a given system, plus the ability to extend the native types to inspect other aspects of a system. This could lead to things like enumerating the open ports, really anything you could dream up.
Long term manageability of a given system through a robust DSL, while abstracting the user away from needing to know the specifics of how a task is accomplished on the given platform.
Combined with the benefits of Fabric we get:
A full featured language for controlling the flow of execution across the environment.
A simple to use set of DSL functions to handle the transport. Pushing and retrieving files, running commands as a given user, or executing over sudo are just simple methods.
Single use, out of band system for controlling systems where Puppet’s default state of polling the master every so often is not as effective. Think heartbleed SSL re-keying all your Puppet infrastructure.
This gets us pretty close, but I’m thinking of a braver, newer world. One where the tools at hand provide an API to a remote operating system. That it is a Linux box in all its complexity shouldn’t matter. Puppet give you the abstraction across the various platforms so that you don’t need to care how the execution is done.
Currently in fabric, you have to tell it to run this command or that, but this is one of the strengths of Puppet, and always has been; that it no longer requires you to know how something is being done, but only that it has been done.
Yes, and?
To improve upon what we have currently, we might expand on the following areas.
First, Puppet should to have a way to execute that returns a more machine readable output upon completion. This would facilitate the ability to execute remotely in an automated way and capture the resulting output in a useful way. This could be as simple as silencing everything else, while returning the report to STDOUT upon request. With this we would get some JSON, parse it for what we care about etc. Perhaps this is just a report processor.
Second, Puppet should also have a clean way to have the the resource types its
aware of managed and (again) returned in a common parseable format with the
results of the action. Both inspection of a given resource and management of
that resource here would go a long way. Again, this could be quite simple if
we extended the puppet resource command set to have the ability to return
JSON. Again, JSON here would make some sense due to the wide availability of
parsers.
If it were Fabric, getting libraries of methods to reach into systems, call the Puppet resources, and return this data to other Python methods as data objects would be simply tubular.
A combination of those ideas I think would lead us to a world that allows the long term management of the resources on a given system, and also allowing the user to build up a system of tools to manage the infrastructure in an ad-hoc, out of band, procedural way that would gain the benefits of both worlds.
Now that is a system I want to use to manage infrastructure.