While watching Neal Ford’s terrific “Agile Engineering Practices” video series, I paid close attention to the segment in which he interactively demonstrated the technique of Test Driven Development (TDD). At the end of his well-orchestrated example, which was to design/write/test code that determines whether an integer is a perfect number, Mr. Ford presented the following side-by-side summary comparison of the resulting “traditional” Code Before Test (CBT) and “agile” TDD designs.
As expected from any good agilista soldier, Mr. Ford extolled the virtues of the TDD derived design on the right without mentioning any downside whatsoever. However, right off the bat, I judged (and still do) that the compact, cohesive, code-all-in-close-proximity CBT design on the left is more readable, understandable, and maintainable than the micro-fragmented TDD design on the right. If the atomic code in the CBT isPerfect() method on the left ended up spanning much more space than shown, I may have ended up agreeing with Neal’s final assessment that the TDD result is better – in this specific case. But I (and hopefully you) don’t subscribe to this, typical-of-agile-zealots, 100% true, assertion:
The downside of TDD (to which there are, amazingly, none according to those who dwell in the TDD cathedral), is eloquently put by Jim Coplien in his classic “Why Most Unit Testing Is Waste” paper:
If you find your testers (or yourself) splitting up functions to support the testing process, you’re destroying your system architecture and code comprehension along with it. Test at a coarser level of granularity. – Jim Coplien
As best I can, I try to avoid being an absolutist. Thus, if you think the TDD generated code structure on the right is “better” than the integrated code on the left, then kudos to you, my friend. The only point I’m trying to make, especially to younger and less experienced software engineers, is this: every decision is a tradeoff. When it comes to your intimate, personal, work habits, don’t blindly accept what any expert says at face value – especially “agile” experts.
There are two common perspectives on the process of architectural design, whether it be for buildings or for software. The first is that a designer starts with nothing—a blank slate, whiteboard, or drawing board—and builds-up an architecture from familiar components until it satisfies the needs of the intended system. The second is that a designer starts with the system needs as a whole, without constraints, and then incrementally identifies and applies constraints to elements of the system in order to differentiate the design space and allow the forces that influence system behavior to flow naturally, in harmony with the system. Where the first emphasizes creativity and unbounded vision, the second emphasizes restraint and understanding of the system context. – “RESTful” Roy Fielding
It might not be a correct interpretation, but BD00 associates Mr. Fielding’s two alternatives with “inside-out” and “outside-in” design.
The figure below illustrates the process of inside-out design. The designer iteratively composes a structure and “hopes” it will integrate smoothly downstream into the context for which it is intended. During the inside-out design process, the parts are king and the system context is secondary.
The figure below depicts an outside-in design process. The designer iteratively composes a structure within the bounded constraints of the context (the “whole“) for which it is intended. During the outside-in design process, system context is king and the parts are secondary.
Because system contexts can vary widely from system to system and they’re usually vaguely defined, messy, and underspecified, designers often opt for the faster inside-out approach. BD00 uses the outside-in design process. What process do you use?
Assume that your team was tasked with developing a large software system in any application domain of your choice. Also, assume that in order to manage the functional complexity of the system, your team iteratively applied the “separation of concerns” heuristic during the design process and settled on a cleanly layered system as such:
So, how are you gonna manifest your elegant paper design as a working system running on real, tangible hardware? Should you build it from the bottom up like you make a cake, one layer at a time?
Or, should you build it like you eat a cake, one slice at a time?
The problem with growing the system layer-by-layer is that you can end up developing functionality in a lower layer that may not ever be needed in the higher layers (an error of commission). You may also miss coding up some lower layer functionality that is indeed required by higher layers because you didn’t know it was needed during the upfront design phase (an error of omission). By employing the incremental slice-by-slice method, you’ll mitigate these commission/omission errors and you’ll have a partially working system at the end of each development step – instead of waiting until layers 1 and 2 are solid enough to start adding layer 3 domain functionality into the mix.
In the context of organizational growth, Russell Ackoff once stated something like: “it is better to grow horizontally than vertically“. Applying Russ’s wisdom to the growth of a large software system:
It’s better to grow a software system horizontally, one slice at a time, than vertically, one layer at a time.
The above quote is not some profound, original, BD00 quote. It’s been stated over and over again by multitudes of smart people over the years. BD00 just put his own spin on it.
On the left we have the process of abstract decomposition, and on the right we have the process of concrete composition:
Note that during the concrete composition from parts to final system on the right, we gracefully transition through two stable, intermediate forms. Some people and communities, especially those obsessed with “velocity” and “time-to-market“, would say “bollocks” to all those value-subtracting, intermediate steps. We no need no stinking intermediate forms:
In an attempt to gain an understanding of the software design he was carrying around in his head, I sat down with a colleague and started talking face to face with him. To facilitate the conversation, I started sketching my emergent understanding of his design in my notebook. As you can see, by the time we finished talking, 20 minutes later, I ran out of ink and I wasn’t much better off than before we started the conversation:
If I had a five year old son, I would proudly magnetize my sketch on the fridge right next to his drawings.
The following C++14 code fragment represents a general message layout along with a specific instantiation of that message:
Side note: Why won’t a C++98/03 compiler accept the above code?
Assume that we are “required” to send thousands of these X-Y position messages per second between two computers over a finite bandwidth communication link:
There are many ways we can convert the representation of the message in memory into a serial stream of bytes for transmittal over the communication link, but let’s compare a simple binary representation against an XML equivalent:
The tradeoff is simple: human readability for performance. Even though the XML version is self-describing and readable to a human being, it is 6.5 times larger than the tight, fixed-size, binary format. In addition, the source code required to serialize/deserialize (i.e. marshal/unmarshal) the XML version is computationally denser than the code to implement the same functionality for the fixed-size, binary representation. In the software industry, this tradeoff is affectionately known as “the angle bracket tax” that must be payed for using XML in the critical paths of your system.
If your system requires high rates of throughput and low end-to-end latency for streaming data over a network, you may have no choice but to use a binary format to send/receive messages. After all, what good is it to have human readable messages if the system doesn’t work due to overflowing queues and lost messages?
In his terrific “Effective architecture sketches” slide deck, Simon Brown rightly states that you don’t need UML to sketch up your software architecture. However, if you don’t, you need to consider documenting the documentation:
The utility of using a standard like UML is that you don’t have to spend any time on all the arcane subtleties of meta-documentation. And if you’re choosing to bypass the UML, you’re probably not going to spend much time, if any, doing meta-documentation to clarify your architecture decisions. After all, doing less documentation, let alone writing documentation about the documentation, is why you eschewed UML in the first place.
So, good luck in unambiguously communicating the software architecture to your stakeholders; especially those poor souls who will be trying to build the beast with you.