Posts Tagged ‘complexity’

A Hoarrific Failure

January 25, 2013 Leave a comment

Work started (on the 503 Mark II software system) with a team of fifteen programmers and the deadline for delivery was set some eighteen months ahead in March 1965.

Although I was still managerially responsible for the 503 Mark II software, I gave it less attention than the company’s new products and almost failed to notice when the deadline for its delivery passed without event.

The programmers revised their implementation schedules and a new delivery date was set some three months ahead in June 1965. Needless to say, that day also passed without event.

I asked the senior programmers once again to draw up revised schedules, which again showed that the software could be delivered within another three months. I desperately wanted to believe it but I just could not. I disregarded the schedules and began to dig more deeply into the project.

The entire Elliott 503 Mark II software project had to be abandoned, and with it, over thirty man-years of programming effort, equivalent to nearly one man’s active working life, and I was responsible, both as designer and as manager, for wasting it.

The above story synopsis was extracted from Tony “QuicksortHoare‘s 1980-ACM Turing award lecture.

Mr. Hoare’s classic speech is the source of a few great quotes that have transcended time:

I conclude that there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult…. No committee will ever do this until it is too late.

A feature which is included before it is fully understood can never be removed later.

At first I hoped that such a technically unsound project would collapse but I soon realized it was doomed to success.

The price of reliability is the pursuit of the utmost simplicity. It is a price which the
very rich find most hard to pay.

The mistakes which have been made in the last twenty years are being repeated today on an even grander scale. (1980)

Dontcha think that last quote can be restated today as:

The mistakes which have been made in the last fifty years are being repeated today on an even grander scale.

Since man’s ability to cope with complexity is relentlessly being dwarfed by his propensity to create ever greater complexity, the same statement might probably be true 50 years hence, no?

Complexity Coping

Three Degrees Of Distribution

December 19, 2012 2 comments

Behold the un-credentialed and un-esteemed BD00’s taxonomy of software-intensive system complexity:

Three Degrees

How many “M”s does the system you’re working on have? If the answer is three, should it really be two? If the answer is two, should it really be one? How do you know what number of “M”s your system design should have? When tacking on another “M” to your system design because you “have to“, what newly emergent property is the largest complexity magnifier?

Now, replace the inorganic legend at the top of the page with the following organic one and contemplate how the complexity and “success” curves are affected:

Social Legend

From Complexity To Simplicity

September 9, 2012 3 comments

As the graphic below shows, when a system evolves, it tends to accrue more complexity – especially man-made systems. Thus, I was surprised to discover that the Scrum product development framework seems to have evolved in the opposite direction over time – from complexity toward simplicity.

The 1995 Ken Schwaber Scrum Development Process paper describes Scrum as:

Scrum is a management, enhancement, and maintenance methodology for an existing system or production prototype.

However, The 2011 Scrum Guide introduces Scrum as:

Scrum is a framework for developing and sustaining complex products.

Thus, according to its founding fathers, Scrum has transformed from a “methodology” into a “framework“.

Even though most people would probably agree that the term “framework” connotes more generality than the term “methodology“, it’s arguable whether a framework is simpler than a methodology. Nevertheless, as the figure below shows, I think that this is indeed the case for Scrum.

In 1995,  Scrum was defined as having two bookend, waterfall-like, events: PSA and Closure. As you can see, the 2011 definition does not include these bookends. For good or bad, Scrum has become simpler by shedding its junk in the trunk, no?

The most reliable part in a system is the one that is not there; because it isn’t needed. (Middle management?)

I think, but am not sure, that the PSA event was truncated from the Scrum definition in order to discourage inefficient BDUF (Big Design Up Front) from dominating a project. I also think, but am not sure, that the Closure event was jettisoned from Scrum to dispel the myth that there is a “100% done” time-point for the class of  product developments Scrum targets. What do you think?

Reasonable Debugging

In Rich Hickey‘s QCon talk, “Simple Made Easy”, he hoisted this slide:

So, what can enhance one’s ability to “reason about” a program, especially a big, multi-threaded, multi-processing beast that maps onto a heterogeneous hodge-podge network of hardware and operating systems? Obviously, a stellar memory helps, but come on, how many human beings can remember enough detail in a >100K line code base to be able to debug field turds effectively and efficiently?

How about simplicity of design structure (whatever that means)? How about the deliberate and intentional use of a small set of nested, recurring patterns of interaction – both of the GoF kind and/or application specific ones? Or, shhhh, don’t say it too loudly, how about a set of layered blueprints that allow you and others to mentally “fly” over the software quickly at different levels of detail and from different aspect angles; without having to slodge through reams of “flat” code?

Do you, your managers, and/or your colleagues value and celebrate: simplicity of design structure; use of a small set of patterns of interaction; use of a set of blueprints? Do you and they walk the talk? If not, then why not? If so, then good for you, your org, your colleagues, your customers, and your shareholders.

Misapplication Of Partially Mastered Ideas

Because the time investment required to become proficient with a new, complex, and powerful technology tool can be quite large, the decision to design C++ as a superset of C was not only a boon to the language’s uptake, but a boon to commercial companies too – most of whom developed their product software in C at the time of C++’s introduction. Bjarne Stroustrup‘s decision killed those two birds with one stone because C++ allowed a gradual transition from the well known C procedural style of programming to three new, up-and-coming mainstream styles at the time: object-oriented, generic, and abstract data types. As Mr. Stroustrup says in D&E:

Companies simply can’t afford to have significant numbers of programmers unproductive while they are learning a new language. Nor can they afford projects that fail because programmers over enthusiastically misapply partially mastered new ideas.

That last sentence in Bjarne’s quote doesn’t just apply to programming languages, but to big and powerful libraries of functionality available for a language too. It’s one challenge to understand and master a language’s technical details and idioms, but another to learn network programming APIs (CORBA, DDS, JMS, etc), XML APIs, SQL APIs, GUI APIs, concurrency APIs, security APIs, etc. Thus, the investment dilemma continues:

I can’t afford to continuously train my programming workforce, but if I don’t, they’ll unwittingly implement features as mini booby traps in half-learned technologies that will cause my maintenance costs to skyrocket.

BD00 maintains that most companies aren’t even aware of this ongoing dilemma – which gets worse as the complexity and diversity of their product portfolio rises. Because of this innocent, but real, ignorance:

  • they don’t design and implement continuous training plans for targeted technologies,
  • they don’t actively control which technologies get introduced “through the back door” and get baked into their products’ infrastructure; receiving in return a cacophony of duplicated ways of implementing the same feature in different code bases.
  • their software maintenance costs keep rising and they have no idea why; or they attribute the rise to insignificant causes and “fix” the wrong problems.

I hate when that happens. Don’t you?

Complexity, Evolution, Growth

March 28, 2012 Leave a comment

Is an increase in complexity required for evolution to occur?

Likewise, is an increase in complexity required for growth to occur?

Frag City

January 17, 2012 3 comments

As the accumulation of knowledge in a disciplinary domain advances, the knowledge base naturally starts fragmenting into sub-disciplines and turf battles break out all over: “my sub-discipline is more reflective of reality than yours, damn it!“.

In a bit of irony, the systems thinking discipline, which is all about holism, has followed the well worn yellow brick road that leads to frag city. For example, compare the disparate structures of two (terrific) books on systems thinking:

Thank Allah there is at least some overlap, but alas, it’s a natural law: with growth, comes a commensurate increase in complexity. Welcome to frag city…

%d bloggers like this: