Over the decades I’ve used many different software development methodologies. So I enjoyed it when the excellent Accidental Tech Podcast had a couple of recent episodes (#55, #56) touching on this topic. My takeaway from their discussion was what works depends on the people, culture and project at hand. So most teams use an ad hoc mix of practices. Totally agree. But when Marco Arment (if I recall) criticized story points and some methodologies as process overkill, I couldn’t help but want to defend what these methodologies have in common. Namely, they all attempt to manage system complexity in light of human psychology. Understanding this commonality makes it easier to switch between approaches. Plus helps you avoid falling into the trap of believing any particular approach is a silver bullet.
Let’s come at this from the four values for Agile software development, which I like:
- Individuals and Interactions over Processes and Tools
- Working Software over Comprehensive Documentation
- Customer Collaboration over Contract Negotiation, and
- Responding to Change over Following a Plan
These values are attempts to deal with complexity. That is, with problems so complicated you can’t generate a top down predictable project plan to get to the answer. So you have to iterate and adjust. This is most clearly shown in values #2 through #4. Working software is valued because software is normally so complex you don’t want to break it. Getting broken software back to a working state is more costly than gradually iterating on a continuously working version. Customer collaboration is about not being too rigid up front, allowing new learning and feedback to drive adjustments to the plan. Responding to change is just that. The point here is despite many differences in details between Agile, Scrum, XP, Test Driven Development, Lean Startup, or what have you, all modern approaches to software development share a trial and error aspect for managing complexity. As long as you don’t lose site of this, you’ll avoid having process for it’s own sake. And remain flexible about the details of any particular approach.
Also note #1 is really about human psychology. Letting people self organize and make their own decisions rather than following more rigid processes. This reminds me of a great piece of coaching I got years ago when I first started working with globally distributed teams: interactions that happen automatically and naturally when everyone is co-located have to be encouraged when people are remote. You don’t appreciate how much informal day to day interaction helps keep team alignment until you watch a geographically separated team slowly drift apart.
Another human aspect to software methodologies is they tend to build up quasi-religious rigidity about doing it right. The stand up meeting must be exactly x minutes long or you’re doing it wrong. I liked David Heinemeier Hansson’s recent screed against this quasi-religious aspect to Test Driven Development entitled “TDD is dead. Long live testing.” The right perspective here is to realize this problem isn’t particular to software. People like having a set of rules on how to get things done. And once you have a checklist, there’s a natural tendency to slowly go overboard enforcing the rules.
This is not a long post. Just a reaction to the ATP podcasts. Perhaps it’s best to finish with this great quote from Steve Jobs:
[T]here’s just a tremendous amount of craftsmanship in between a great idea and a great product. And as you evolve that great idea, it changes and grows. It never comes out like it starts because you learn a lot more as you get into the subtleties of it. And you also find there are tremendous tradeoffs that you have to make. There are just certain things you can’t make electrons do. There are certain things you can’t make plastic do. Or glass do. Or factories do. Or robots do.
Designing a product is keeping five thousand things in your brain and fitting them all together in new and different ways to get what you want. And every day you discover something new that is a new problem or a new opportunity to fit these things together a little differently.
And it’s that process that is the magic.
Exactly. Ideas are cheap. Real learning happens when you struggle to build something, and have to adjust to what you learn along the way. Software processes can be thought of as tools for making these adjustments gracefully. In fact, before I started writing this post I thought scrum used story points to indirectly estimate work because the human mind was better at relative estimation than absolute. In reading up on story points I learned this was a myth, or at least somewhat suspect. The Ancient Greek word for this kind of practical learning is praxis, learning by doing. And if you weren’t aware, that’s where the name of this blog comes from.
Praxtime by Nathan Taylor 
Clicked through your relative versus absolute estimation link, but I think that Tshirt sizing makes a fraught process less frightful. At least for sw eng newbs.
And it is absolutely correct to link Agile to the seminal “No Silver Bullet” essay, as that described software as irreducibly complex. Thus short iterations map to the organic processes of software, rather than the uninformed metaphor of an architecture blueprint.
A decade ago, I used to look forwards to Thursdays since then I would read Linux Weekly News. Now it is Praxtime on Mondays.