On crafting software
Within living memory, computer programming was handicraft. Individual programmers strained to create works that were both useful and beautiful—the two virtues went together. In 1984, Steven Levy’s book Hackers thrillingly documented this heroic age of computing. The last of the true hackers, in Levy’s term, is Richard Stallman ’74, now better known as the evangelist of free software (“Free as in freedom, not free beer”).
Most software in use today is an industrial product. The resource constraints that once forced a “small is beautiful” discipline on programmers are a thing of the past. Programmers no longer need to worry about saving a byte of memory here or there, because even small computers have huge memories. Programmers don’t need to scrutinize their code for erroneous syntax, any more than today’s computer-assisted writers need to know how to spell. Like word-processing software, today’s programming systems relieve a lot of the tedium of large-scale coding.
Scott Rosenberg ’81,
Dreaming in Code: Two
Three Years, 4,732 Bugs,
and One Quest for
Freed from old constraints, programming teams can now produce software of marvelous complexity. But as any computer user knows, flawlessness is not one of the virtues of modern software. As computers appear inside devices that used to work fine without them, faulty software has also become universal. Now even telephones re-boot themselves mysteriously. More raw materials have not produced better programs, any more than books improved in quality when paper became cheap.
In Dreaming in Code, Scott Rosenberg ’81 chronicles the production of a computer program in the twenty-first century—as the subtitle has it, “one quest for transcendent software.” He tries to recapture the spirit of programming’s heroic age, and the group he tracks is small enough—two dozen programmers, led by Lotus 1-2-3 creator Mitch Kapor—to keep the story personal. He sensitively documents fluctuations in the group’s social structure and its shifts between individualism and teamwork. Rosenberg, a writer and founder of the website salon.com, is not a programmer himself. His perspective is that of an “embedded journalist,” as a fellow writer, James Fallows ’70, described him in the July-August 2006 Atlantic Monthly. To produce the book, Rosenberg lived in the trenches with the programming troops for three years, observing the daily vicissitudes of their spirits.
But being an embedded journalist is dangerous work. A journalist of engineering runs no risk of personal injury, but the project he is documenting may not have an inspiring denouement. The classic of the genre is The Soul of a New Machine, by Tracy Kidder ’67, a brilliant narrative of computer engineering. But Rosenberg’s story has no conclusion—at the end of the book, the software is not ready.
And that is fitting. The real subject of the book is not “Chandler,” the particular piece of software that Rosenberg watched under development, but the question of why producing software is hard. We are about 50 years into the computer age now, and even the most professional programming organizations still miss deadlines, abandon huge projects after tens of millions of dollars have been invested in them, and ship code that fails straight out of the box. Why does software engineering seem to be so much harder than building bridges? Why is it, as the late Tom Cheatham [former Gordon McKay professor of computer science] explained so well in the many software courses he taught at Harvard, that you cannot look at a software project, as you can look at a bridge under construction, and be confident that it is half-finished?
Rosenberg sprinkles possible explanations throughout his book; the narrative glides back and forth between events in the lifecycle of Chandler and relevant observations by past gurus of software engineering. The grand old man of the field is Fred Brooks, Ph.D. ’56, who wrote a classic essay, “The Mythical Man-Month,” in 1975, at the close of the first years of programming’s industrial era. Brooks’s law is that adding programmers to a late software project makes it later. The new troops require training by the old to be brought up to speed, and unless communication is managed carefully, twice as many programmers require almost four times as many interactions among themselves. This is at core an observation about knowledge-based teamwork, but other maxims are more specific to software engineering. Brooks also observed that good programmers are up to 10 times as productive as their average colleagues, a phenomenon that helps explain the discipline’s high tolerance for malodorous but talented eccentrics.
As a tour of homely wisdom about the challenges of software engineering, interpolated with the dramatic narrative of a real-life case study, Dreaming in Code is an entertaining success. But the two parts of the story don’t hang together very well, because the Chandler project is in multiple ways not a normal programming project. Unwisely, it was begun without much in the way of a design specification. It is an “open source” project, developed in an atmosphere of public transparency owing something to Stallman’s ideals, but without his doctrinaire libertarianism. Open source is an important methodological development in software engineering, but is still a work in progress. Atypically, Chandler has no discernable budget; Rosenberg’s narrative makes no reference to the possibility of running out of money. Even quests for transcendent software can profit from budgetary discipline. In the middle of the project, Kapor, who is the project’s chief bankroller and also chief visionary, decides to become the project’s development manager as well—a move as foolish as lawyers representing themselves in court.
Rosenberg recognizes how crazy this all seems. In the middle of the story, he says,
By now, I know, any software developer reading this volume has likely thrown it across the room in despair, thinking “Stop the madness! They’re making every mistake in the book!”
Just so. In the end, it is hard to know whether writing Chandler is hard because of difficulties inherent in software (which led Brooks to entitle a later essay “No silver bullet”), or whether Chandler’s problems could have been overcome by scrupulously adhering to management methods now conventional in the industry, inspired by all that homely wisdom of the past. Because the Chandler project is idiosyncratic, the book fails to be the epic of software engineering Rosenberg wished to write. It is not the software analog of Tracy Kidder’s still-vibrant book.
Though Rosenberg is a good writer, he occasionally gets too immersed in his subject and loses the reader in jargon. Even geeks will grow impatient with sentences such as “CPIA would be built on top of wxWidgets, but its goal was to wrap Chandler’s Python code into packages called ‘blocks’ that would be connected to each wxWidgets element.”
And we still don’t know how Chandler will turn out. I checked the product’s website for the latest news, and found this look-on-the-bright-side internal communication of the engineering team, confirming Rosenberg’s observation that all programmers are optimists: “Book will be available sometime between October and January. This is good for us: gives us a chance to develop further before book out.” Rosenberg is as sympathetic to the programming project as the engineers are to his book project, and the result seems to lack journalistic objectivity about the source of the woes it depicts.