Tips for Automating Deployment

Here at Cuker, we have been architecting, building, and hosting custom web applications, mostly built in Django, since 2007. One of the real breakthroughs for us was when we started building a fully automated deployment system.

In 2010, just when Heroku was picking up steam, but still did not have any python capabilities, we passed around a link to the 12 factor app. The 12 factor app is a concept that an app by itself should be self contained and thus can easily be managed in different environments.

We built our internal automated deployment toolset around these principles (as well as having deployed a few sites to Heroku). Over the years it has been used to deploy and manage over a hundred websites, through many iterations, redeployments, migrations, and upgrades. It isn’t perfect, but it is (mostly) dependable, explicit, and a huge time saver.

Here are five lessons we learned in the process that we wished we had known early on.

Build Systems, not Scripts
In the beginning, it always seems to start with a script (in our case, fabric). Ultimately, though, you will most likely build out multiple management systems, roll out new versions alongside existing versions, and change out most of your base assumptions. This is called software engineering and is best done without all the horrible global state assumptions of “scripts”. Take the five minutes, make a class. Sure it isn’t final, but keeping things contained in local objects upfront makes it way easier to refactor. When it comes time to have multiple server roles, it is nice to inherit from a base class and build up, rather than copying and pasting code.

Hardcode your Dependencies
Nothing sucks more than redeploying an application only to find that a dependency has since been upgraded and is no longer compatible. In our case, we track dependencies with pip requirements. We hard code every dependency, to the minor, and sometimes subminor level, so that any changes or updates are done locally and staged through the development site to verify full compatibility.

Build a Management Layer
When we started, we had a set of configuration files for each site that specified the git repositories and service permissions (e.g. database, cache, indexer). What it didn’t do was provide a way to manage billing, track deployment history, or notify on completion or issue. Building a separate management application allows you to track and manage the business context of your sites, as well as make life easier for your team (not to mention real-time slack notifications).

Same Base, Different Configurations
We originally worked with a common set of applications, and changed our baseline project for each new site and application. As we grew and iterated we realized that we lost forward progress on the core configuration since we always forked it. Instead, we now focus on having the same base setup, and use the configuration to individually add applications. This lets us iterate our cache configurations, database settings, and so on, without having to update every forked project.

Don’t Get Caught on that Last 5%
When it comes to automation, there is a tendency to get bogged down getting those last little bits automated and running smoothly. We found that it is better to get it 90% automated, and have a couple manual steps. It ultimately will probably occur to someone on your team how to get that last little bit automated, but in the meantime, you can all save time and risk by having computers do the straightforward stuff. For example, you can go down the boto rabbit hole in automating Amazon Web Services, but sometimes it is easier to configure a few things on the AWS console, then run the rest through your toolset.

Conclusion
The intervening years has seen some great tools come out for managing applications like Docker, as well as a broader movement towards “DevOps”, where you treat your applications and servers as “cattle” not “pets”. Even with all these great tools, you need to keep focus on creating as homogeneous an environment as possible, and ensuring that it is empowering and transparent to your team, secure, redundant, and predictable. On top of that, you need to make sure you keep your accounts organized so that you know what the relationship is for every site you manage. But don’t let all this bog you down! Automate 95% and use that time savings to tune up your systems or get a latte.