We work hard to maintain a disciplined, agile, and test driven development practice at Grockit. To this end, our test suite must be green (have no failing tests) before we can push our new code and features to our demo, and ultimately, production sites.

Until recently we ran our test suite locally on a Mac Pro machine running multiple VM ware instances. Our build was split into fast and slow builds so that we could shorten the feedback on tests that fail. This process got us to a build time of about 45 minutes from beginning to end assuming the build didn’t stop short due to a test failing.

We have since moved our test suite into Amazon EC2 so that we could reduce the overall build time by parallelizing our tests from two tests running simultaneously to over 10 tests. We are really excited by the results. We’ve managed to reduce our build time by almost 90%. The build now takes approx five min from start to finish.

Our EC2 distributed build depends in a large part on TeamCity, a continuous integration product from JetBrains. Our build is split into 15 fragments which are each defined as a TeamCity build configuration. When code is checked in, the TeamCity hub, running in EC2, can distribute the running of build configurations to any of nine build agents, which are also running in the cloud. In minutes, all of the build configurations are executed and results are reported back to the central server, informing our team members via a web interface displayed on a large screen in our office. We also made a simple GreaseMonkey script to turn the background of the monitoring page red when any of our build configurations fail, which helps increase visibility.

With our build cut down from 45+ minutes to less than five minutes, there is a palpable sense of increased momentum in our development process. It’s also interesting how removing this bottleneck made other process problems less of an issue. For instance, we were considering adding a feature that allowed our product manager to toggle feature sets as visible or hidden in production dynamically. This was needed because once a feature set was ready, we would need yet another round trip through the build cycle for developers to ‘unhide’ the functionality in the production environment. When our test suite took 45 minutes, this was a big deal. But now that its fast, the pain of this manual step is far less acute. The moral of the story: find the biggest bottleneck and eliminate it before worrying about others.

Most importantly, for our users, this means more frequent deployments of new features, bug fixes and performance improvements.