Because methodologists need an “enemy” to make their pet process look good, Agilistas use Traditional methods as their whipping boy. In this post, I’m gonna turn the tables by arguing as a Traditionalista (yet again) and using the exalted Agile methodology as my whipping boy. First, let’s start with two legends:
Requirements And User Stories
As you can see, the Agile legend is much simpler than the Traditional legend. On planet Agile, there aren’t any formal requirements artifacts that specify system capabilities, application functions, subsystems, inter-subsystem interfaces/interactions, components, or inter-component interfaces/interactions. There are only lightweight, independent, orthogonal, bite-sized “user stories“. Conformant Agile citizens either pooh-pooh away any inter-feature couplings or they simply ignore them, assuming they will resolve themselves during construction via the magical process of “emergence“.
Infrastructure Code And Feature Code
Unlike in the traditional world, in the Agile world there is no distinction between application-specific Infrastructure Code and Feature Code. Hell, it’s all feature code on planet Agile. Infrastructure Code is assumed as a given. However, since developers (and not external product users) write and use infrastructure code, utilizing “User Stories” to specify infrastructure code doesn’t cut it. Perhaps the Agilencia should rethink how they recommend capturing requirements and define two types of “stories“: “End User Stories” and “Infrastructure User Stories“.
Regarding the process of “Design“, on planet Agile, thanks to TDD, the code is the design and the design is the code. There is no need to conceptually partition the code (which is THE one and only important artifact on planet Agile) beforehand into easily digestible, visually inspect-able, critique-able, levels of abstraction. To do so would be to steal precious coding time and introduce “waste” into the process. With the exception of the short, bookend events in a sprint (the planning & review/retrospective events), all non-coding activities are “valueless” in the mind of citizen Agile.
No Front End
When asked about “What happens before sprint 0?”, one agile expert told me on Twitter that “agile only covers software development activities“.
As the process timeline template below shows, there is no Sprint -1, otherwise known as “the Front End phase“, on planet Agile. Since the Agile leadership doesn’t recognize infrastructure code, or the separation of design from code, and no feature code is produced during its execution, there is no need for any investment in any Front End work. But hey, as you can plainly see, deliveries start popping out of an Agile process way before a Traditional process. In the specific example below, the nimble Agile team has popped out 4 deliveries of working software before the sluggish Traditional team has even hatched its first iteration. It’s just like planet Agile’s supreme leader asserts in his latest book – a 4X productivity improvement (twice the work in half the time).
The flexible, dare I say “agile“, aspect of the Traditional development template is that it scales down. If the software system to be developed is small enough, or it’s an existing system that simply needs new features added to it, the “Front End” phase can indeed be entirely skipped. If so, then voila, the traditional development template reduces to a parallel series of incremental, evolutionary, sprints – just like the planet Agile template – except for the fact that infrastructure code development and integration testing are shown as first class citizens in the Traditional template.
On the other hand, the planet Agile template does not scale up. Since there is no such concept as a “Front End” phase on planet Agile, as a bona fide Agilista, you wouldn’t dare to plan and execute that phase even if you intuited that it would reduce long term development and maintenance costs for: you, your current and future fellow developers, and your company. To hell with the future. Damn it, your place on planet Agile is to get working software out the door as soon as possible. To do otherwise would be to put a target on your back and invite the wrath of the planet Agile politburo.
The Big Distortion
When comparing Agile with Traditional methods, the leadership on planet Agile always simplifies and distorts the Traditional development process. It is presented as a rigid, inflexible monster to be slain:
In the mind of citizen Agile, simply mentioning the word “Traditional” conjures up scary images of Niagara Falls, endless BRUF (Big Requirements Up Front), BDUF (Big Design Up Front), Big Bang Integration (BBI), and late, or non-existent, deliveries of working software. Of course, as the citizenry on planet Agile (and planet Traditional) knows, many Traditional endeavors have indeed resulted in failed outcomes. But for an Agile citizen to acknowledge Agile failures, let alone attribute some of those failures to the lack of performing any Front End due diligence, is to violate the Agile constitution and again place herself under the watchful eye of the Agile certification bureaucracy.
The Most Important Question
You may be wondering, since I’ve taken on the role of an unapologetic Traditionalista in this post, if I am an “Agile-hater” intent on eradicating planet Agile from the universe. No, I’m not. I try my best to not be an Absolutist about anything. Both planet Agile and planet Traditional deserve their places in the universe.
Perhaps the first, and most important, question to ask on each new software development endeavor is: “Do we need any Front End work, and if so, how much do we need?”
Take a minute to look around yourself and imagine how many interdependent chains of specialists and handoffs were required to produce the materials that make your life quite a bit more comfortable than our cave dwelling ancestors.
Total self sufficiency is a great concept, but if you had to do all the work to produce all of the things you take for granted, you wouldn’t be able to. You simply wouldn’t have enough time.
In software development, above a certain (unknown) level of complexity and size, specialists, dependencies, and handoffs are required to have any chance of getting the work done and the product integrated. Cross-functional teams, where every single person on the team knows how to design and code every functional area of a product, is an unrealistic pipe dream for anything but trivial products.
On my first pass through Bertrand Meyer’s “Agile!” book, I interpreted its contents as a well-reasoned diatribe against agilism (yes, “agile” has become so big that it warrants being called an “ism” now). Because I was eager to do so, I ignored virtually all the positive points Mr. Meyer made and fully embraced his negative points. My initial reading was a classic case of the “confirmation bias“; where one absorbs all the evidence aligning with one’s existing beliefs and jettisons any evidence to the contrary. I knew that the confirmation bias would kick in before I started reading the book – and I looked forward to scratching that ego-inflating itch.
On my second pass through the book, I purposely skimmed over the negatives and concentrated on the positives. Here is Mr. Meyer’s list of positive ideas and practices that “agile” has either contributed to, or (mostly) re-prioritized for, the software development industry:
- The central role of production code over all else
- Tests and regression test suites as first class citizens
- Short, time-boxed iterations
- Daily standup meetings
- The importance of accommodating change
- Continuous integration
- Velocity tracking & task boards
I should probably end this post here and thank the agile movement for refocusing the software development industry on what’s really important… but I won’t :)
The best and edgiest writing in the book, which crisply separates it from its toned down (but still very good) peer, Boehm and Turner’s “Balancing Agility And Discipline“, is the way Mr. Meyer gives the agilencia a dose of its own medicine. Much like some of the brightest agile luminaries (e.g. Sutherland, Cohn, Beck, Jeffries, Larman, Cockburn, Poppendieck, Derby, Denning (sic)) relish villainizing any and all traditional technical and management practices in use before the rise of agilism, Mr. Meyer convincingly points out the considerable downsides of some of agile’s most cherished ideas:
- User stories as the sole means of capturing requirements (too fine grained; miss the forest for the trees)
- Pair programming and open offices (ignores individual preferences, needs, personalities)
- Rejection of all upfront requirements and design activities (for complex systems, can lead to brittle, inextensible, dead-end products)
- Feature-based development and ignorance of (inter-feature) dependencies (see previous bullet)
- Test Driven Development (myopic, sequential test-by-test focus can lead to painting oneself into a corner).
- Coach as a separate role (A ploy to accommodate the burgeoning agile consulting industry. Need more doer roles, not talkers.)
- Embedded customer (There is no one, single, customer voice on non-trivial projects. It’s impractical and naive to think there is one.)
- Deprecation of documents (no structured repository of shared understanding to go to seek clarification on system level requirements/architecture/design issues; high maintenance costs for long-lived systems; costly on-boarding of new developers)
I’ve always maintained that there is much to like about the various agile approaches, but the way the agile big-wigs have been morphing the movement into a binary “do-this/don’t-do-that” religious dogma, and trashing anything not considered “agile“, is a huge turnoff to me. How about you?
When creating an architecture or detailed design of a system component, up to three forms of design evaluation can be performed:
- A: Continuous evaluation by the designer during the process
- B: Evaluation by a peer group at one or more review events during the process
- C: Evaluation by an expert outsider group after A and B
These three forms of evaluation are illustrated in the sequence of diagrams below. Note that as each evaluation type is progressively added to the mix, a new feedback loop is introduced into the system.
During A, the conceive/evaluate/repair loop is performed at the speed of thought within the isolated mind of the designer. As the designer applies his skills to his understanding of the problem to be solved, alternative component structures and configurations can pop up instantaneously during each of many high frequency loop traversals. At some point, the designer concludes that his design solution is stable enough for external scrutiny and/or ready for the next step of realization.
During B, the designer walks through his creation in front of a group of peers. He explains the structure and behavior mechanisms of his design and how it solves the problem. If the peer group is qualified, prepared, and objective (QPO), at least a portion of the feedback the group produces will be valuable to the designer and he will use it to improve his design. However, unless all three QPO pre-conditions are satisfied, the peer review process will be a huge waste of time and money.
Jetting in an expert outsider group is the riskiest type of evaluation. Assuming that the QPO criteria is satisfied, the fact that the expert outsider group has no real “skin in the game” after it vacates the premises should be a cause for concern to anyone contemplating the use of the technique.
In general, the more negative feedback loops incorporated into a process, the more likely the process is to produce its intended output. As the final figure above implies, incorporating all three types of evaluation into a design review process can lead to a high quality design. However, blindly ignoring the QPO criteria and/or failing to address the “skin in the game” risk can increase cost, lengthen schedule, and lay waste to your well-intentioned review process – without you ever knowing that it occurred. As ever, the devil is in the details.
Either directly or subliminally, the message I hear from hardcore agilista big-wigs is that an agile process trumps a traditional plan-driven software development process every time and in every context – no exceptions.
On the other hand, the message I hear from traditionalists is… well, uh, I don’t hear much from traditionalists anymore because they’ve been beaten into silence by the hordes of unthinking zombies unleashed upon them by the agilista overlords.
Regarding the “betterness” of #agile over #traditional (or #noestimates over #estimates, or #noprojects over #projects), please leave your handful of personal anecdotes at home. Charismatic “I’ve seen” and “in my (vast) experience” stories don’t comprise science and aren’t sufficient justifications for sweeping generalizations. The science simply doesn’t exist – especially for the construction of large, distributed software systems.
I suppose that if a plausible (and thus, falsifiable) theory of software development processes was to be methodically derived from first principles and rigorously tested via a series of repeatable experiments, the general result would end up looking something like this:
What predictive capabilities do you think a credible theory of software development processes would generate? Show Me Your Curves.
Wanna go on a wildly fun rollercoaster ride? Then watch Erik Meijer’s “One Hacker Way” rant. Right out of the gate, I guesstimate that he alienated at least half of his audience with his opening “if you haven’t checked in code in the last week, what are you doing at a developer’s conference?” question.
I didn’t agree with all of what Erik said (I doubt anyone did), but I give him full credit for sticking his neck out and attacking as many sacred cows as he could: Agile, Scrum, 2 day certification, TDD “waste“, non-hackers, planning poker, the myth of self-organizing teams, etc. Mmmm, sacred cows make the best tasting hamburgers.
Uncle Bob Martin, the self-smug pope who arrogantly proclaimed “If you don’t do TDD, you’re unprofessional“, tried to make light of Erik’s creative rant with this lame blog post: “One Hacker Way!“. Nice try Bob, but we know you’re seething inside because many of your sacredly held beliefs were put on the stand. You seem to enjoy hacking the sacred cows of the reviled “traditional” way of developing software, but it’s a different story when your own cutlets are at steak.
Mr. Meijer pointed out what I, and no doubt many others, have thought for years: agile, particularly Scrum, is a subtle, insidious form of control. At least with explicit, transparent control, one knows the situation and where one stands. With Scrum, the koolaid-guzzling flock is duped into believing they’re in control; all the while being micro-managed with daily standups, burndown charts, velocity tracking, and cutesy terminology. No wonder it’s amassed huge fame and success – managers love Scrum because it makes their job easier and anesthetizes the coders.
My fave laugh-out-loud moment in Erik’s talk was when he presented Jeff Sutherland as the perceived messiah of software development:
I’ve found that the best books and talks are those in which I find some of the ideas enlightening and some revolting. Erik Meijer’s talk is certainly one of those brain-busting gems.
The test of a first-rate intelligence is the ability to hold two opposing ideas in mind at the same time and still retain the ability to function. – F. Scott Fitzgerald
I’ve been developing software in the aerospace and defense industry for 30+ years. Because of the sizes of, and innate System-of-Systems nature of, the solutions in this domain, the processes used to develop them are by necessity, heavyweight. Oh sure, agile/lean techniques can be used effectively in the teeny-tiny-small deep down in the bowels of some of these programs, but to assume “scaling agile” will work in a domain with hundreds of programmers from multiple contractors working on the same distributed system is naive at best, and downright disingenuous at worst.
The most ridiculous examples of inapplicable development practices to the aerospace and defense industry are #noestimates” and “#noprojects“. The “#noprojects” cult is so far off base that they don’t even acknowledge the existence of “programs” – which are a way of organizing a large set of inter-dependent “#noprojects” into, well, a larger grained mission-critical entity worth tens of millions of dollars. It’s like before there were classes and namespaces(packages) added to a programming language to accommodate increasing software system size, there were only “functions” for organizing the structure of a program – and having functions as your only organizing mechanism doesn’t scale well to large software systems.
Hell, the #noprojects people are akin to #noclasses and #nonamespaces people, of which there are thankfully, #nomembers. Really hardcore #noprojects people are perhaps even more loony. They are equivalent to a hypothetical #nofunctionsallowed crowd that demands monolithic, straight-line code only. Line number 1 straight to line number 5,000,000 – and you can’t use loops or “ifs” because they don’t add value and #nofunctionsallowed is all about “maximizing the amount of work not done“.
But hey, if you don’t need any “projects” to build a successful product from the gecko, then by extrapolation you don’t need any “programs” for organizing your projects. Hence, the inimitable BD00 proposes to “scale up” the “#noprojects” movement. We’ll start a revolutionary #noprograms movement, or equivalently, #justcode . We’ll leapfrog both the #noestimates and #noprojects movements at once. W00t!