How is it to be a chef?

How are people deploying applications when chef-client is running as a daemon or via cron?

• It seems like a no-brainer to me to run chef-client as a daemon or via cron in order to keep your machines up-to-date with configuration changes. The idea is to let machines "pull" down configuration changes (at any random interval), with the expectation that everything will eventually become consistent. On the other hand, application deployments tend to be something you'd want to "push" out in some controlled way. Without a lot of very detailed monitoring/alerting in place, it might be too scary to just let deployments happen all at once or randomly across your cluster. You might prefer to deploy to a single application server and watch it for a while before deploying to two more, then four, etc. This might lead you to conclude that: - Maybe chef-client should only be run manually, and configuration changes will just be picked up when a deployment happens. But then you lose the ability to keep your machines consistent. Untouched machines will slowly diverge from what's in Chef, and the longer you wait, the scarier it is to run chef-client on these machines, because you can't always be sure that they can be brought back to a known-good state. In order to prevent this, it could easily become someone's full-time job just to run Chef everywhere manually. - Maybe Chef shouldn't be used for deployments at all. But then what are people using instead? And doesn't this lead to having your configuration management split between two different tools (i.e. system configuration v.s. application configuration)? This just seems wrong to me. I've recently come up with a simple idea that might solve the problem, but I haven't heard of anyone else doing anything like this, so I'd love to get some feedback. My idea is essentially to add something like this to your deploy resources: only_if { ENV['DEPLOY'] } Now you can let Chef run on an interval without worrying that anything will get deployed. In order to trigger a deploy, you'd run chef-client like so: $ sudo DEPLOY=1 chef-client Thoughts?

• Answer:

  Netflix has this "test deployment" methodology down pat.  The secret is AWS EC2. They have a "canary machine" that they deploy to first, and they let it handle a full load of production traffic.  If the canary doesn't shit the cage, they spin up a whole new cluster, and point the load balancer over to it.  If that messes up, it's back to the old cluster.  Either way, poof, one of the clusters is gone at the end of the day. Your gradual strategy would be best without this "doubling on the cloud", because the deployment takes time, and rolling back takes time... You'd want to minimize the screwup if something goes wrong.  However, if you're able to deploy everything "backstage", and just flip a switch when it's all ready, and switch it back if things look bad, then there's much less worry about allowing Chef to provision "everything at once". Reference article: http://highscalability.com/blog/2011/12/12/netflix-developing-deploying-and-supporting-software-accordi.html I can see, if you were using your own machines, implementing a "cloud bursting" strategy, where you temporarily deploy to a new cluster on the cloud, then, if it works after a switch, deploy to the old, locally hosted machines and then switch back, spinning down the cloud cluster.