At Mendix we are now managing quite a lot of servers. They are all very similar. Over the last year we've set up about 200 new servers with the same bash script. Whenever we updated the script, we pushed the changes to the existing servers using Fabric. From the start we anticipated and indeed soon ran into an issue: "This will not scale."
Everyone in the devops field, or in a startup that grows too fast, will tell you that you require a tool like Chefpet or if you are very brave CFEngine. So, this week we decided to give Puppet a try, next week we'll try Chef.
We're probably going to write up on this on the Mendix Tech Blog, but I can already tell you this:
When I first read someone comment "So, if you're in the market, good luck making your choice. I'm not making a recommendation here because, quite frankly, I wouldn't recommend either to anyone other than my worst enemy. ;-)" on Hacker News, I thought he was joking. Now I see what he meant.
Our use case is basically this: we create a new Debian server on a IaaS provider using libcloud (Rackspace, Linode, you name it). We run some very very basic initial configuration (put the hostname in /etc/hosts, set up /etc/network/interfaces, we install Chefpet). Then we connect to the Chefpet master and from there EVERYTHING should come up automatically. The server should install the required packages, it should be firewalled, it should have a DNS entry, it should be registered with our Cloud Portal etcetera.
We basically have three common server types: appnode (runs jvm's on sun-java-jre), dbnode (runs postgres), webserver (runs nginx as a reverse proxy).
In my OO mind it makes sense to create a basenode. The other types would be subclasses of this basenode.
In this basenode we would like:
- The resolvers (2+) we've set up in this node's datacenter.
- A local mailserver that forwards to our own mailservers (2+) in this datacenter.
- Some basic nagios-nrpe and munin plugins.
- A vimrc for root, our ssh keys, tools like ack-grep, nmon, mrt, etc..
Before we can install anything we need to run apt-get update/apt-get upgrade.
To be continued.