The Golden Heart of Agility

I’ve been asked to review a lot of projects and systems in the last ten or so years. 10 years ago, no-one talked about Agile development (I don’t think the phrase had yet been coined); 5 years ago very few mentioned the ‘A’ word; now every project I see claims to be Agile. What’s depressing is that, despite the very real benefits of an agile approach, very little has changed. Of course, this is a self-selecting group – sadly I’ve yet to be asked to review a totally successful project that delivered a beautifully-realised system – but my impression is that my pool of potential project reviews is not rapidly shrinking due to the success of Agile methods.

One of the common factors of the failing Agile development project is that they are selective in their application of agile development practices. They all have (or have close approximations of) sprints, user stories, prioritisation, planning meetings, daily stand-ups, and visible metrics. People talk knowingly about ‘you aint gonna need it” and “do the simplest thing”. They started out with a sustainable pace. And yet commitment dates are being missed, the customer can’t get a straight answer to when they can go live, changes in requirements are met a collective sucking of teeth and the team is afraid to touch the code in the system in case something breaks unexpectedly (as opposed to all the expected breakages that occur daily in the running system). In short, there is no agility in these Agile projects.

On the flip side, projects that achieve true agility all have a common core of practices:

  • Pair programming
  • Test Driven Development
  • Refactoring
  • Continuous Integration

There can be quite a variation in the approach these projects take to the other stuff: there’s always a daily stand-up and usually user stories, but approaches to planning and prioritisation, the metrics gathered, the lengths and purpose of iterations can vary significantly. Sometimes the customer is on-site and heavily involved, at other times there is a set of people (let’s call them BA’s) acting as a proxy customer. There may or may not be a coach or appointed Scrum Master. But there is always the four practices above.

Of course, there’s nothing magic about these practices. All they do is establish a high degree of agility in the development team which, in turn, helps the wider project to achieve agility. Here’s why:

  • Pair-programming ensures no bit of the system is understood by any one member of the team. The bus-factor for the system is zero. The team develops a shared understanding of the system at every level: code, metaphor, architecture. They learn new skills and techniques from each other. New people join the team or existing members leave without huge impacts on team productivity or ability. Teams that pair-program never say to the customer: “well we can’t estimate or deliver that because Freda’s away and its her part of the code” or “its going to take a while because the guy who used to do that has left”.
  • Test-driven development concentrates the team on really doing the simplest thing possible and making sure they stay focussed on just that. It also produces an executable specification of the system that, combined with pairing, ensures no piece of the system is ill-understood or ‘off limits’. And it provides a safety net that allows the team to rapidly makes changes to the code without fear of unexpected breakages. Teams that practice TDD never say to the customer: “well we think we’re finished but we’re not sure its working properly” or “well we delivered on time but the bug count is too high to allows us to go live”.
  • Refactoring is simply the processes of improving the design and implementation of the system as you go. Refactoring, or rather the clean implementation and designs it produces, combines with TDD to ensure the team can make changes to the system quickly and confidently. Refactoring makes it possible for someone who is less familiar with a bit of the system to pair with someone who is more familiar and find it easy to ‘get up to speed’. Teams that refactor never say to the customer: “Yeah, the bit of the system that’s causing all the problems is a right mess; we need to stop building features and spend some time cleaning it up”.
  • Continuous integration is having one codebase that everyone changes on a continuous basis. That means: no branching (okay, maybe one branch for the live system and the trunk for the next version in development), no keeping changes on your own workstation for a few days, no integration hell. More importantly it means always having a working version of the release in development available for the customer, the users and/or the QA team to review and provide feedback on. Teams that integrate continuously never say to the customer: “Oh, that’s what you meant by XYZ. Well, we’re either going to have to go live with what we have or postpone go-live for a few weeks while we fix it” or “You can review the system in three sprints time” or “Yeah, that bits been broken for ages, I’m sure it will be fixed before we go live”.

I’ve come to think of these four practices as the ‘heart of agility’; they’re certainly not sufficient on their own but they are the foundation on which the other practices build. If that heart is missing, achieving agility is going to be extremely hard, if not impossible. If you’re working on an Agile project that is failing to deliver some or all of the publicised benefits of an agile approach, ask yourself whether you’re really doing all of these four practices all of the time. Of course there may be many factors that affect a team’s ability to deliver but, if you’re not doing these four, you could do worse than working out why not and trying to change the situation.

[Note on the title: Originally the post was called ‘The Heart of Agility” but there’s a somewhat eccentric pub I like in East London called The Golden Heart and the phrase kind of stuck. Pair programming, TDD, Refactoring and CI are not just the ugly muscle that powers agile development, there is something beautiful and valuable about how they combine.

The Golden Heart

The Golden Heart pub in Spitalfields, London

Note on capitalisation: In the late 90’s I gave a session on why XP does away with the need for what I called ‘Big A Architecture’. Software architecture is an extremely important subject but at the time there was a whole industry being built around standards, notations, patterns and processes for software architecture that meant a lot of people could talk about Architecture and could even produce an awful lot of outputs that contributed precisely nothing of benefit to the architecture of a system under development. I see the same thing happening with ‘Big A Agile’ and try and distinguish between Agile methods and achieving agility.]


13 thoughts on “The Golden Heart of Agility

  1. J. B. Rainsberger

    I will buy that these items form the golden heart of agile delivery, but not the golden heart of agility. For that, you need strong, regular business involvement, and you don’t seem to have mentioned any of that here.

    1. pauldyson Post author

      That’s an excellent comment and, you’re right, this is focussing on agile development. The point I was trying to make is that it doesn’t matter how engaged you are with your business, if you’re not doing these four things then you wont be able to achieve true agility. But the opposite is also true: just doing these four things means you’re running a nice agile development project which is probably not going to deliver anything of value.

      I guess an interesting question is “what does the fifth practice look like?”. Pairing, TDD, Refactoring and CI are all fairly well-bounded practices but ‘strong, regular business involvement’ is a wide-ranging concept with many variations and subtleties. XP’s ‘onsite customer’ was always a place-holder for business involvement but my personal experience is that this is rarely achievable and sometimes not desirable; and the product management practices of Scrum can work for or against a team looking for business involvement.

      1. J. B. Rainsberger

        I agree, and also believe that if you don’t regularly earn value from what you deliver, then you haven’t come close to agility.

        I don’t know what specific practices to add that would suffice in forming the golden heart of agility, but I imagine that writing business examples for stories/features ranks high up there. We don’t seem to have a pithy name for the activity where business and technical people sit down together and design examples to clarify the meaning of features or stories.

        After that, we need to add some way to link stories to value streams. I only know how to do this for software that solves my own business problems, and not what to do in general.

      2. pauldyson Post author

        I think a lot of people who have achieved agility can tell you what works for them with regards to business involvement but it is very hard to generalise.

        When I did my first XP project it was on a pre-revenue product so there literally was no customer. The ‘proxy customer’ was the company CTO and that worked pretty well.

        Later I led a project that had two customers (the people that bankrolled the project) but 8 very important stakeholders. We evolved something a bit like a Scrum backlog to deal with all the different viewpoints and priorities and that worked pretty well.

        Later I led a project that fixed a number of technical issues with a system that had been delivered by a third party. It rapidly became clear that trying to get the customer to prioritise, even by explaining the issues in business terms, wasn’t really helping. He wanted us to exercise our ‘expert technical opinion’ in fixing the various problems. On that project we were our own customer and that worked pretty well.

        On the other hand, our approaches to pairing, TDD, refactoring and CI have varied little over the last 12 years bar incremental improvements in our knowledge and skill.

        Perhaps one of the problems with Agile methods is they focus too much on the business interaction and too little on the activities of developers. Reminds me a bit of SSADM where the development was ‘just the work at the end that transformed all the models into code’ – oh so simple!

      3. J. B. Rainsberger

        “Perhaps one of the problems with Agile methods is they focus too much on the business interaction and too little on the activities of developers.”

        That surprises me, because I observe the exact opposite. I observe agile methods focusing too much on delivery and making little headway in helping businesses achieve useful, lasting change above the delivery teams.

        I don’t know how to proceed here. If we whip the delivery teams into shape, then we risk making the marketing department (as just one example) look bad. At that point, the organizational antibodies kick in and push us out. On the other hands, if we try to push the business and technical parts of the organization together before each has started doing a better job, then we risk creating chaos where at least they had useful delineations in work.

        I don’t know how to build trust with such disparate groups that doesn’t involve delivering something, and I certainly don’t intend to invite those groups into the wilderness for team building. 🙂

    2. Rob

      I would not argue with you, but the absolute heart of what we do is deliver and maintain software and whilst there is no doubt business involvement is critical I’d argue it sits in the next ring out. Check out Allan Kelly’s article “Requirements Come Second” in the Agile Journal it explains it much better than me.

      1. J. B. Rainsberger

        I would like to see us expand the meaning of “we” here. If we keep treating “the business” as some nebulous thing that sits out there, then we’ll never realize any of the true promise of agile software development.

      2. Rob

        I don’t treat anyone I work with as nebulous. Obviously the whole company is crucial to it’s effectiveness it’s just that if I was going to introduce “Agile” into an organisation I would start with the engineering practices first before I worried about getting the requirements right or working on a ubiquitous language etc.

        P.S. If you have a better way of referring to the people in my organisation outside of the software delivery team I’d love to know it. Tried customer, stakeholder etc.

      3. J. B. Rainsberger

        I didn’t mean to accuse you of anything, Rob. Rather, in spite of the advances that delivery teams have made in becoming “more agile” (assuming that term has useful meaning), so few organizations have embraces agile ideas from top to bottom than mostly we have teams that deliver better software without improving the bottom line. From the CxO’s point of view, nothing has changed.

        I used to also start agile transitions by streamlining delivery before worrying about marketing, planning, story-writing, and the like. I think the larger “we” does that. Now I hear word that CxOs don’t see any real improvement from that. I interpret that to mean that I need to expand my reach. I encourage others to do so the same.

        I don’t get your “P.S.”. I don’t think I said that you referred your the people in your organization outside the software delivery team in a poor way, perhaps apart from “outer ring”. I think that sets up an us v. them attitude in general, even if you can handle that phrase well.

  2. Rob

    It always comes down to the basics. Coincidentally I’ve recently been thinking about how although software development is very difficult the process is quite simple and the Golder Heart you describe is pretty much it. If you’re not doing these things well there is little value gained spending time on higher level issues such as communication with the customer, gathering requirements and measuring value.

  3. David Bland

    Agile works best when paired with a Customer Development (see Steve Blanks – 4 Steps to the Epiphany).

    So while I advocate Agile, I don’t advocate building software that no one ever uses.. hence pairing this with customer feedback loops.

    Agile is just a crucial piece of the overall successfully tech business model, not the entire puzzle.

  4. Pingback: Agile Business Conference 2009 « Paul Dyson’s Blog

  5. Pingback: Shearing Layers in Software Delivery Part 2: A Reasonable Model « Paul Dyson’s Blog

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s