Why we don’t Continuously Deploy

Continuous Deployment is very much en vogue; its seen as the natural extension of Continuous Integration and is a part of the lean startup philosophy. But we don’t do it and I thought it might be interesting to explain why.

  1. Our users like improvement but they don’t like change. We build a business system for business users and our users each spend 2-8 hours a day using our system. They want to see the system improve but having a consistent, recognisable user experience is more important than having one that is constantly changing, even if that change brings improvement. So we gather all their feedback as they use the system and package that into a release. When that release is ready we do demos and distribute documentation to make sure everyone is aware of the changes coming. Bundling up all improvements into one big change allows us to make sure everyone understands what is coming, is happy about what’s new, and there are no surprises when they log in.
  2. We do QA as well as testing. To me QA is a separate activity to writing and running automated tests. My experience of developer-written tests (unit, functional, integration and so on) is they tend to take a rational and logical view of the functionality under test: what happens when used correctly, what happens when expected parameters aren’t supplied, what happens when parameters are supplied with unexpected values? But users aren’t necessarily ‘rational’ or ‘logical’ as developers see it and having humans bash away at the system, hitting the back button randomly, pasting parameter values from surprising sources, leaving the browser open for an hour while they have lunch, and so on helps us to spot the gaps in the automated testing and the assumptions we made whilst writing them. But manual QA takes time and having continuous pressure to have it done as quickly as possible to support continuous deployment will be counter-productive. So we bundle our changes up into weekly QA releases which means QA can be done at a pace that suits the assurance of quality rather than speed of deployment.
  3. We’re still learning. To me there’s a bit of a paradox in Continuous Deployment as applied to lean startup. On the one hand, multi-variate testing and continuous deployment allow you to gather and act on lots of feedback very quickly. On the other hand, a constantly changing user experience makes it harder to make sense of that feedback. In our system there is a piece of core functionality that users use for ~3 hours each day. If that changed on a daily basis they’d quickly get pissed off with it and wouldn’t use it until it was ‘ready’ or ‘done’. We’re now on our third major re-working of that functionality and the latest version is vastly improved, based on real user feedback. But each new version has been introduced as a whole, with all the change management mentioned above, and so we’ve kept our users onside whilst making quite radical changes to the product.
  4. ‘Continuous’ is a relative, not an absolute, term. Sounds like a stupid statement but its a lesson I learned a while back. In a previous job I was doing some work with a company and their supplier to improve their responsiveness to change. We were pushing for weekly releases but quickly realised that, to a pair of companies used to 18-24 month release cycles, quarterly releases would be a big improvement and much more achievable than weekly. Were we being unambitious? Perhaps from our own point of view as eXtremists, but in terms of the goals our clients had, quarterly would take them along way to achieving what they wanted. At Singletrack we do releases every 6-10 weeks which is so much more continuous than other systems our users use (upgrades every 12-24 months is more the norm) but avoids the disruption that ‘true’ Continuous Deployment would cause.
So we currently do: Continuous Integration in development; Daily Builds with full runs of automated tests (takes about 3 hours otherwise we’d do this more frequently); Weekly QA releases; End-user releases every 6-10 weeks. A lot of this stems from our business: we sell a complex, powerful SaaS product to sophisticated and demanding business users. If we were running a B2C website with a host of irregular and infrequent users (from the point of view of our system, a user that doesn’t log in for hours every day is infrequent) then we would certainly go for a much more continuous process than the one we have now.

2 thoughts on “Why we don’t Continuously Deploy

  1. whatwereallyknow

    Hi Paul, an interesting read. I was wondering whether you feel there is a deterministic relationship between continuous delivery and the first two issues you mentioned? (an unstable user experience, continuous pressure on QA)

    It’s hard for me to comment, you obviously know the constraints you work under a lot better, but I’m not so sure.

    1. pauldyson Post author

      Thanks for your question.

      On the first, we deliver a UI-heavy system to a set of users that spend most of their day in it. The rate at which we currently change the UI (early stage product being developed with lots of user feedback) means they would see old buttons disappearing, new buttons appearing, screen layouts changing, etc. on an almost daily basis. Our users are a pretty demanding lot and they use our system to do their day job. Whilst they do care that the the UI is simple, attractive, easy to use, etc. they care more that they get their job done; i.e. they prefer to work with a sub-optimal bit of design that they are used to (and know we will deliver an improvement – with appropriate comms and training – within 8-10 weeks) than a constantly changing one; even if that change brings improvement sooner.

      On the second, I think the nature of QA in a TDD-centered organisation is very different to that in more traditional dev shop. Our QA people never see basic bugs or fundamentally broken builds as these are caught by the testing and CI/CB mechanisms. What we look to QA for is assurance of quality not just testing – i.e. help us find the areas of the system where we are light in our testing, help us identify the bits of the UX that aren’t as good as they need to be, help us improve the look and feel. These are not things that can be done ‘just in time’ and because of our users we don’t want to put stuff in front of them that our QA people subsequently flag up as not being up to scratch.

      Of course there are techniques we could use to hide changes until we’re ready to show them. But then I’d question the value of CD. Of course there are benefits to CD other than user feedback but I feel we get these from CI/CB and our regular deployments to our QA environment. This is partially due to the platform we work with: force.com comes with very powerful deployment support which means that QA really is just another production environment and deploying to it is no different to deploying to our customer’s production environments; if it works in QA it is guaranteed to work in production. So given the benefit of not doing CD to our users and QA department, and the lack of additional benefits we would gain by doing it, we choose not to.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s