On Sunday, I attended to some talks in the Configuration Management devroom. These are the notes and links from the talks.
Arthur Lutz (@arthurlutz) talked about After describing your infrastructure as code, reuse that to monitor it. He presented an architecture in which he ran the Munin monitoring directly from Salt minions. It was interesting to see the variety of metrics they are currently watching.
Marco Ceppi (@marcoceppi) gave a talk entitled Beyond config management: Tackling orchestration and modelling on top of config management. Basically, he described the evolution of Configuration Management and how he considers that modelling is the next step: using abstract primitives to become platform and tool agnostic. It is basically the idea behing Canonical’s Juju, a project in which Marco is involved. He also mentioned a similar tool recently released by Wallmart labs called OneOps. I will give both of them a try!
Walter Heck (@walterheck) explained the learnt lessons of a very big project of Puppet implementation in a manually-managed-infrastructure enterprise in War Story: Puppet in a Traditional Enterprise. Interesting to see that many problems were not tech-related but human-friction related. Changing the way we work is hard but is absolutely necessary for a sysadmin.
Eric Sorenson (@ahpook) discussed about Flexibility and Power in Puppet 4 Language. The most important features of the new version of the tool are: improved network communication protocol (backwards compatible), enhanced packaging and changes in the language (types, support for each Ruby construct, ability to define functions, hierarchy improvements).
Peter Souter (@petersouter) performed a brilliant talk about Hardening your Config Management. On my opinion, the best talk among the ones that I have attended to in this edition of FOSDEM. It was fully packed of wise advice such as:
- Follow the OWASP security principles
- Remove data from code using data abstraction layers. For example: hieras in Puppet, data bags/attributes in Chef, roles in Ansible, grains/pillars in Salt
- Encrypt data with your application tooling: hiera-yaml in Puppet, chef-vault in Chef, ansible vault in Ansible, salt.modules.gpg in Salt, cf-keycript in CFEngine
- Use, if possible, external secret servers, like OpenStack’s Barbican, CloudFlare’s Red October or Hashicorp’s Vault
- Use git-cript for achieving a transparent file encryption in git
- Use automated (spec testing, linting) and manual code reviews
- Compare your security with others when possible
- Establish game days trying to break your security and seeing how much damage is caused and how long would it take to notice the intrusion
- Get a baseline of the status of the infrastructure and monitor it for unexpected changes (using riemann, statsd, ELK, collectd…)
- Search for suspicious activity in your logs, as 4xx, 5xx errors when there is no activity, unexplained increases in temperature of the machines…
- Use your tooling sensitive-protection on (like
no_log: Truein Ansible) - Use automatic hardening frameworks as hardening.io (seems incredibly useful!)
- Follow SSH hardening standars (whitelisted access, bastion hosts, restrict users, increase key strength, rotate keys, use pre-populated knownhosts)
Mark Hoffmann spoke about Literate DevOps. He presented this concept, based upon the Literate Programming principles, as documentation with embedded executable DevOps code. He presented an implementation with Emacs and Org mode. Good food for thought that reminds me the Jupyter notebook approach.
Stéphan Gorget, from Facebook engineering, gave a talk about Managing a complex DNS environment. He explained the really complex architecture that they have deployed in order to attain reliable DNS updates in every machine. They use Unbound and TinyDNS in each host, with a pipeline that merges the manual changes and automated changes, distributes the config files using BitTorrent and restarts the daemons using a framework named sparts. For the external DNS, they use a system that performs TLS termination near the users. They distribute a unique name to look up to a small percentage of users, allowing Facebook to buld a map of the users and resolvers. This map is updated every 2 minutes and sent to the DNS servers, in order to minimize latencies.
The last talk of the devroom was Config Management and Containers by Charles Butler (@lazypower). H explained how containers are disrupting the configuration management ecosystem and the Juju’s approach to modelling them: by using the Charm Layer for Docker. A nice tool, that even supports docker-compose.yml files.
And I finally attended to the closing keynote Putting 8 Million People on the Map, by Blake Girardot (@BlakeGirardot), from the Humanitarian OpenStreetMap Team. It is amazing how the work of volunteers mapping all over the world can help in case of disasters, and the OSM Tasking Manager is a blast. It is very easy to get involved. Happy hacking!