On the Art of Computer Programming

I have been a software engineer for about 30 years, and it is remarkable how many new observations I still have about the art of programming.  I could write volumes about this, but this morning had an important observation about the need for a good programmer to be able to shift their point of view from an abstract to concrete view with great facility.

There is a memorable quotation from F. Scott Fitzgerald that says:

The test of a first-fate intelligence is the ability to hold two opposed ideas in mind at the same time and still retain the ability to function.

I am not sure whether it's the test of a first-rate intelligence, or even whether Fitzgerald really did.  After all, he wrote it when he was amidst one of the most dark and self-deprecating phases of his life.  But I can say with certainty it is a skill that an extraordinary programmer must possess.

Extraordinary programming involves two fundamentally opposed concepts: abstract conceptualization and mastery of detail.  These two skills are rarely present in the same person.  I often have found great "idea people" who have been able to conceptualize broad and sweeping ideas and yet are frustrated by the need to conquer all of the minutiae required to realize their dreams.  To be successful, they must empower others to do the leg work.  By contrast, there are many detail-oriented people who can master unbeleivably complex systems from top to bottom, but cannot see the larger purpose.

Software systems consist of myriads of components, each of which serves an abstract purpose.  So, extraordinary programmers view software systems as simple things.  It is this simplicity that drives their design.  A computer serves a purpose which can be described simply.  A program serves a simple purpose.  A library serves a simple purpose.  A function serves a simple purpose.   To create the best software, all systems must be reduced to the simplest most abstract notion of what is to be accomplished, and such discrete systems are combined easily to form larger discrete systems.  That is the goal of good design.

To implement such systems, a programmer must start with an abstract notion.  Then, during the debugging and implementation process, only the truly extraordinary programmer is willing to explore virtually every single detail of how the abstract notion behaves, and every detail of how every abstract building block upon which it rests behaves.  They see how functions they call operate, inspect the assembly language generated, and would not hesitate to understand fully perhaps 20 or 30 dependent functions nested deep within software they did not write.

Only the extraordinary programmer resists having the details become the master to the abstract, or the abstract the master of the details.  Any system which is brilliantly conceived and poorly implemented, or brilliantly implemented but built upon an overly baroque or complex abstract notion, is flawed.  To avoid this, it requires being able to simultaneously see a simple, elegant system while also seeing thousands of details which make it operate.  The extraordinary programmer knows that every detail counts.  Every detail supports the abstract concept or it is superfluous and should be removed like a cancer.

There are many essays and philosophies which support this way of thinking.  But, it is harder to do, and more rarely seen than one supposes.  In software, it is the exception rather than the rule, and until it becomes the standard of software design, we will continue to be frustrated by the software we use.

Comments

[David]

When I was younger I coined the expression "zoom perception" to describe being able to see the big picture and the detail all at once. I assumed everyone could do it. Later on I was quite surprised to find just how rare such ability is, as recognition of its existence.

The devil in any successful implementation is always in the detail (be it a complex computer-based system, a jet engine, or an off-road buggy) so an ability to competently deal with detail is imperative; but without the big picture one can easily misdirect effort; elegantly solving the wrong problem (or a peripheral problem).

Humans use abstraction and analogy to form mental models, to help understand and communicate ideas. People with "zoom perception" can 'see' several models of the same thing at once, and can use the model relevant to the detail being considered.

Software is an especially interesting implementation domain in which to build complex things, as most of the code is associated with unusual, unplanned or unexpected events. Getting a piece of code working is always the easiest part; engineering to robustly deal with every manner of exception takes the remaining 90% of the time and effort.

[Gary Wisniewski]

I agree, David, that the ability is rare. I do think most of it can be taught, and one of the big weaknesses of commercial software vendors is that they do not recruit and value individuals who have such training. Rather, it is assumed to be an innate ability.

The common language of abstraction and detail haven't moved into the mainstream, and we're still dealing with lots of self-taught programmers who rely on their instincts. This results in people who argue about whether abstraction is even useful, and companies like whose programming products break abstraction rules left and right in the interests of greater "intimacy with the platform". These are not computer science concepts, these are mostly voodoo.

Consider yourself lucky to have such an innate ability, and even luckier to recognize what it is.