Danger! Software Craftsmen at Work.

On October 12, 2010 at QCon David Harvey gave a talk entitled Danger! Software Craftsmen at Work.  This talk was nicely summarized and expanded upon by Gael Fraiteur in his blog: Why Should We Care About Software Craftsmanship?  Part 2.  But I have a few things to add.  


Harvey makes the following points (as described by Fraiteur):

1. The Manifesto for Software Craftsmanship is empty of content because it is not refutable, i.e. it is not possible for a reasonable person to disagree.
   
2. The opposition of software craftsmanship to software engineering is pointless and may even give permission to software developers to ignore the lessons of software engineering.
   
3. Metaphors, and the language we are using to describe ourselves and our activity, do matter. The people around us think of a craftsman as someone producing leather bags, not items you can rely on. Although software developers have their own definition of craftsmanship, what eventually matters is the perception of our customers. By choosing inappropriate metaphors, we are increasing the gap between those who build software, and those who use it. 

The Empty Manifesto
Is the Manifesto for Software Craftsmanship empty because it is irrefutable?  I think the notion is absurd.  That's like saying that the Hippocratic Oath, or the Golden Rule are empty because they are irrefutable.  The Manifesto is not a scientific hypothesis that requires experimental verification.  Rather the Manifesto is a statement of beliefs and values that the signatories adhere to and promote.  The Manifesto contains articles of belief, not statements of measurable fact, and is therefore not required to be falsifiable.

Is the Manifesto irrefutable?  Would that it were so!  Unfortunately the Manifesto is regularly refuted in both word and deed.  For example, the first article of the Manifesto states that we value well-crafted software over working software; yet there continues a significant debate on the topic of "good-enough software".  There is a large cohort of software developers who contend that well-crafted code is antithetical to time-to-market. 

The second article of the Manifesto is more interesting still.  It promotes the steady addition of value over simply responding to change.  What value does this refer to?  It refers both to the value of the software's function, and the value of it's structure.  That is, we as craftsman, will continue to steadily improve both the structure and function of the software, rather than simply responding to change.  This kind of responsible behavior is refuted daily by the actions of developers who release changes that damage both structure and function.  If you doubt that this happens, consider this video of bad code that I posted a few months ago.  Believe it or not, this is deployed production code.

Engineering vs. Craftsmanship.
Is craftsmanship antithetical to engineering?  Harvey suggests this possibility based on some of the statements in Pete McBreen's classic book Software Craftsmanship in which he derides some of the more egregious, high-ceremony and high-documentation practices associated with Software Engineering.  Harvey suggests that this may give "permission" to budding software craftsmen to ignore the good software engineering work that has been done over the decades. 

I agree that this would be a bad thing.  We don't want anyone in the Software Craftsmanship community to dismiss Software Engineering out of hand.  The problem I have with Harvey's  suggestion, however,  is that none of the leaders in the Software Craftsmanship movement espouse the view that the history of Software Engineering is worthless.  Indeed, quite the opposite is true.  Software Craftsmen see themselves as Software Engineers.  That does not mean we accept all of the Software Engineering dogma that has accumulated over the decades.  It also doesn't mean that we reject it.  It does mean that we learn it.

The Software Craftsmanship community is deeply committed to learning the lessons of the past.  That means studying the things we did right, and the things we did wrong.  Software Craftsmen immerse themselves in their craft.  We continue to read the old books by DeMarco, Yourdon, Parnas, Dijkstra, Hoare, Weinberg, and their brethren.  We study at the feet of the old masters so that we can learn how to be the next masters. 

It is true that we have currently tabled some of the older practices that have been associated with Software Engineering; but we do not disrespect those practices, nor the people who at one time proposed and adopted them.  They were pioneers who led the way and who, in some cases, showed us the paths to avoid.

The Craftsman Connotation
Harvey advises us to take care with the metaphors we choose.  He makes the point that terms like craft, dojo, kata, apprentice, master etc., can have negative connotations.  The word "craft" for example, may bring to mind the kind of quality one experiences at a flea-market or a craft fair.  The martial-arts terms that are sometimes common in craftsmanship circles may bring to mind the notion of the "omnipotent loner" like Bruce Lee, or Neo.  The terms "Master", "Journeyman", and "Apprentice" may bring to mind the secretive guilds of the middle ages with all their ritual, mysticism, and intrigue. 

I think this is a legitimate concern.  I also think it's easily dealt with.  The strategy I've been using is "guilt by association".  When I talk about Software Craftsmanship, I also talk about Software Professionalism.  I use the terms interchangeably in order to enforce the association in my listeners' (and readers') minds.  When I talk about dojo's and kata's, it is always in the connotation of "practice'.  I use the terms together so that there is no doubt about what the terms actually mean.

Harvey is right in that we don't want to create a "secret language".  There is nothing wrong with the memes we've chosen to communicate amongst ourselves; but we have to be careful to make sure we associate those memes with concepts like professionalism, practice, and seniority that our customers and employers understand and appreciate.   We want these people's support.  We want them to believe and trust in the values that we espouse.  We will not accomplish that by disrespecting the metaphors that they depend upon.

The Cost of Code?

In a panel at #scna yesterday, @chadfowler asked the question: "How many projects fail because of the code?"  I think the point he was trying to make was that the primary causes of project failure are business issues, not technical issues.

I posed this question on twitter earlier today.  The responses came quickly, and virtually all of them agreed that business issues are the more significant causes of project failure.

It's certainly true that projects fail because of cost, requirements, schedule, management. etc.  It's also true that we seldom trace the cause of failure back to something as mundane as code.  So Chad's point, if that's what it was, certainly has merit.

The conclusion that we might make from this is that code just isn't very important in the long run, and the the craftsmanship movement might just be a load of Yak Shaving.  Indeed, Chad asked us to consider just that in his talk at #scna.  If we follow that reasoning, then we should decrease our emphasis on technological prowess and skill, and increase our emphasis on business, requirements, budgets, schedules, and management. 

Before I counter this argument, let me say that I do know of projects that have failed because of the code.  Indeed, I know of companies that have failed because of the code. 

This isn't actually very difficult to believe or understand.  We all know that when the code is a mess, it becomes more and more costly to maintain and improve.  If that cost exceeds what the project can afford, the project fails.  If that cost exceeds what the company can afford, the company fails.  In the cases that I am aware of, this is precisely what happened.  The code was simply too costly for the business model to support. 

So let's try a simple thought experiment.  What fraction of projects would fail if the code was infinitely expensive to produce and maintain? Clearly all projects would fail because the code would be too expensive for any finite business model to support. 

OK, so what if the code cost nothing to produce and maintain?  What fraction of those projects would fail because of the code?  Again, the answer is clear.  No project would fail because of the code, if the code costs nothing to make. 

What does it mean to cost nothing to make?  It means that you would have the code you needed the instant you needed it.  The code would simply be there, instantly, fully functional,  free of defects.  Any time you needed a change, the change would instantly be in effect, fully deployed, fully operational.

So let's say you are thrown into a cave  by some thieves.  In that cave you find a old beat-up PC jr, complete with the IR chicklet keyboard.  You pick up that keyboard and rub a smudge off the enter key.  Wooosh!  a genie appears on the screen and grants you the ability to have zero cost code for the rest of your life!  Would any of your projects ever fail from that point on?

Remember, nobody else has your ability.  Nobody else can produce the code they want, instantly, and without defect.  Nobody else can make and deploy changes in zero time.  So you have a tremendous competitive advantage.  Is there any way you could fail?  I think my dog Petunia might fail, but anyone smarter than that should become a multi-trillionaire.

If we had that magic PC jr, there wouldn't be any schedule or budget issues.  The cost of mismanagement and/or bad requirements would be close to zero.  So all those things that cause projects to fail would become irrelevant.  

But we don't have that magic PC jr.  Code does cost money to produce and maintain.  But if I, as a craftsman, can  invoke a fraction of the power of that Genie to reduce the cost of producing and maintaining code, then I simultaneously reduce the cost and risk of mismanagement, of bad requirements, of tight schedules, and of tight budgets.  By reducing the cost of the thing that's being managed, we reduce the cost of error and increase the chances of success!

Why is it that projects fail due to bad requirements, bad management, bad schedules, and bad budgets?  They fail because the cost of error is huge.  Why is the cost of error huge?  Because the cost of the code is so horribly large.  If code cost nothing to produce, the cost of error would be close to zero.

This realization has not been lost on the business community.  They tried to solve it by reducing the hourly rate of programmers.  They set up horrifically expensive and risky mechanisms in order to hire programmers who lived half a world away in a wildly different culture.  They faced the issues of time zones and languages, and cultural mismatch in order to reduce the cost of code.  They did this because they understood that the it is that cost that drives the cost of management error.  They did this because it is that cost that makes projects fail.

Unfortunately this strategy didn't work as well had been hoped.  Some folks have made it work; more or less.  But the majority of the off-shoring efforts have been disappointing.  And so the cost of code remains high, and therefore the risk of error is also high.

And that brings us back to the question at hand.  How many projects fail because of the code?  The argument above suggests that all failures are a direct result of the cost of code.  How many projects fail because of code?  All of them!

More importantly, what is the single most effective way to increase the chances of project success?  Is it improving requirements?  Management?  Schedules and budgets?  All those things would help, but they are all secondary to the thing that truly drives project failure:  The cost of the code.

Be a good one. #scna 2010

I didn't expect it, but something profound happened at #scna this week.  I expected the conference to be good.  I expected it to be fun.  I expected to see many old and new faces and have stimulating conversations.  And in all these things my expectations were met.  What I didn't expect was the gelling.

There was a meme at this conference that pervaded every talk and every session.  Doug Bradbury (@dougbradbury) coined it in the title of his talk: Made to Make.  His point was that we are makers.  We love to make.  There is something within us drives us to create.   Doug opened his talk with a story from his childhood.  He was eight, hanging out with his grandfather in the workshop.  His grandfather saw him leaning over his child-sized workbench and asked him what he was doing.  "I'm making stuff." was his reply. 

Keavy McMinn (@keavy), in her talk Artist to Programmer, reiterated this meme more directly when she quoted one of her friend's tweets: "I just want to make stuff, I don't really care if its Flash or objective-C or fuzzy felt" and again later: "The future belongs to the few of us still willing to get our hands dirty".  One of the most moving moments in Keayv's presentation was her description of daily refactoring a one-ton tower or bricks during an art show.  She showed pictures of this tower from day to day.  Each different.  Each telling a different story.  Each lovingly built.

Michael Norton (@DocOnDev) talked about the history of medicine, medical education, and medical certification.  He placed it all on a time line and showed how medicine transitioned through phases of empirical observation, to the canonization of an initial body of knowledge, to rapid theoretical and technological development, to the intensely supervised and collaborative learning model of today.  And throughout this long history and transition, medicine began as, and remains, a craft developed by people who love what they do. 

I (@unclebobmartin) and Michael Feathers (@mfeathers) both gave talks about functional programming, showing us new (old) ways to make our stuff.   Enrique Comba Riepenhausen (@ecomba) gave an impassioned talk about fostering partnerships with our customers, reiterating the pleas and advice from both Ken Auer and Chad Fowler (@chadfowler) reminding us that: We make things for others.

There were lots of open-space sessions about all kinds of things.  Laptops were always open, Code was never far away.  There were "randori" coding sessions on stage.

  Me exhorting Adewale Oshineye (@ade_oshineye) to "Write a Test".

There were impromptu coding sessions and lessons and discussions. 

Improptu Clojure Coding Session

Corey Haines (@coreyhaines) gave the closing talk, and it summarized the tone perfectly.  The message, set amongst stories of cats, and cows, and redwoods, was simple: We are makers.  We love what we do.  We are happiest doing what we do.  So we need to do the things that make us happiest.

Yes, there was a gelling at #scna 2010.  It was a gelling around a meme.  It was the consolidation of an idea.  It was a group of people who found themselves to be in violent agreement over a central organizing notion. 

Abraham Lincoln said it over 100 years ago.  Had he been at #scna 2010 he might have gotten up on stage after Corey's talk and sent us home with the following exhortation:

"Whatever you are, be a good one."

(Thanks to Monty Ksycki for taking all those great pictures!)

The Craftsman 62, The Dark Path.

Fri, 17 Mar 2002, 14:00

"Hey Alphonse," Jerry called as I walked by, "let's do a bit of practice.  I've got a kata I'd like to show you."

I felt I could use the break so I walked over and sat next to Jerry.

"Sure, Jerry, what's a Kata?"

Jerry rolled his eyes.  "You've never done a kata?"

I could feel my guard going up, but I tried to relax.  "No, can't say I have."

Jerry smirked and then called over to Jasmine: "Hay Jaz, do you want to tell Alphonse what a kata is?"

Jasmine's long dark hair swished playfully as she turned her head to face me.  She nailed me with those sparkling green eyes as she answered: "What, the hotshot's never done a kata?"

"He says not.  Can you believe it?"

"Jeez, what do they teach these kids nowadays?"

"Oh come on!" I said, starting to get annoyed.  "You guys are only a couple of years older than me, school hasn't changed that much."

Jasmine smiled at me, and I felt my annoyance evaporate.  That smile...  "Relax Alphonse, we're just poking fun at you.  A kata is just a simple program that you write over and over again as a way to practice.  We do them all the time.  It's part of our normal discipline."

"You write the same code over and over?"  This was new to me, and it didn't make a lot of sense.

Jerry nodded and explained: "Yeah.  Sometimes we'll do a kata two are three times in a row, exactly the same each time.  It's a good way to practice your hot-keys."

"And sometimes," Jasmine added, "we solve them in different ways, using different techniques as a way to learn different approaches and reinforce our disciplines."

"And sometimes we just do them for fun." Jerry concluded.

"Which one are you going to show him?" Jasmine asked.

"I was thinking about doing 'Word Wrap'."

"Oh, that's a good one.  You're going to like this Alphonse.  You guys have fun." And with that she turned back to her work. 

I turned to Jerry and asked: "Word Wrap?"

"Yeah, it's a simple problem to understand, but it's oddly difficult to solve.  The basic premise is really simple.  You write a class called Wrapper, that has a single static function named wrap that takes two arguments, a string, and a column number.  The function returns the string, but with line breaks inserted at just the right places to make sure that no line is longer than the column number.  You try to break lines at word boundaries."

I thought about this for a moment and then said: "You mean like a word processor, right?  You break the line by replacing the last space in a line with a newline."

Jerry nodded.  "Yeah that's the idea.  Pretty simple huh?"

I shrugged.  "Sounds simple, yes."

Jerry pushed the keyboard in my direction.  "OK, then why don't you start."

I knew this was a trap of some kind, but I couldn't see how.  So I said: "I suppose we should begin with simple degenerate tests."  I took the keyboard and began to type. I got the first few tests passing as Jerry watched.


Jerry got real interested as I wrote this.  When I got the second test working he said: "What's all that @RunWith and @Suite stuff you are typing?" 

I smiled.  Apparently I was about to teach Jerry something.  "Oh, yeah." I said nonchalantly.  "That's the TestNest pattern.  I learned it from Justin a few days ago.  It lets you put more than one test class in a file.  Each test class can have it's own setups and teardowns."

"Yeah, that's kind of slick.  But who's this Justin dude?"

I pointed and counted ceiling lights.  "He works just down the hall, beneath the 8th light."?

"You mean by those guys who are always walking on treadmills while they code?"

I nodded and kept on coding while Jerry stared back down the hall and recounted the lights. 


Jerry looked back just in time to see that last test pass.   He looked over the code and nodded.  "Yes, that's just about exactly how I solved it the first time.  That replaceAll is a bit of a hack isn't  it."

"Yes, but it gets the test to pass.  'First make it work, then make it right.'"

Jerry nodded.

"Anyway, it's pretty straightforward so far." I said.  And so I went on to write the next test.


Jerry nodded sagely.  "Yes, that's the obvious next test."

"Yes, and with the obvious failure." I agreed.  So then I looked back at the code. 

I stared at it for a long time.  But there did not seem to be any simple thing that I could do to make the test pass.

After a few minutes, Jerry said: "What's the matter Alphonse?  Stuck?"

"No, this should be simple.  I just..."  In frustration I took the keyboard and began to type.  I typed for quite a while, adding and erasing code.  Jerry nodded knowingly, and sometimes grunted.  After about five minutes Jerry stopped me.  The code looked like this:


"Are you sure you're on the right track, Alphonse?"

I looked at the code and realized that I had been coding blindly.  I ran the tests in desperation, but they just hung in an infinite loop.  I could kind of feel what needed to be done, but it wasn't at all clear how I should proceed. 

"Give me another shot at this." I said, as I erased all the code and started over.  Jerry just smiled and watched as I flailed around for another five minutes or so.  Finally, with lots of tests failing he stopped me again. 


"What are you doing wrong, Alphonse?"

I stared at the screen for a minute. Then I said: "I know I can get this working,  give me another shot."

"Alphonse, I know you can get it working too; but that's not the point. Stop for a minute and tell me what you are doing wrong."

I could hear Jasmine stifling a giggle.  I looked over at her, but she didn't meet my eye.  Then I took my fingers off the keyboard and hung my head.  "I can't seem to get this test to pass without writing a lot of untested code." I said.

"That's true." Said Jerry, but it's not quite the answer I was looking for.  You were doing something wrong.  Something really wrong.  Do you know what it was?

I thought about it for awhile.  I had been trying to figure out the algorithm.  I had tried lots of different approaches.  But all my guesses turned out wrong.  --   Oh!

I looked Jerry square in the eye and said:  "I was guessing."

"Right!" Jerry beamed.  "And why were you guessing?"

"Because the test was failing and I couldn't figure out how to get it to pass."

Now Jerry narrowed his gaze, almost like he was looking through me.  "And what does that tell you?"

"That the problem is hard?"  I guessed.

"No, Alphonse, the problem is not hard.  When you see the solution, you're going to be very angry at yourself.  The reason you could not figure out how to pass that test, Alphonse, is that you were trying to pass the wrong test."

I looked at the tests again.  They seemed perfectly logical.  So I asked Jerry: "How could these be the wrong tests?"

Jerry smiled with a grin that rivaled Jasper's.  "They are the wrong tests, Alphonse, because you could not figure out how to pass them."

I gave Jerry a stern look.  "You're being circular, Jerry."

"Perhaps I am.  Look at it this way.  The test you are trying to pass is forcing you to solve a very large part of the problem.  Indeed, it might just be the whole problem.  In any case, the bite you are taking is too big."

"Yeah, but..."

Jerry stopped me and said: "Did you ever read The Moon is a Harsh Mistress Alphonse?"

"Uh... Heinlein, wasn't it?  Yes, I read it a few years back.  It was a great story."

"Indeed it was.  Do you remember this quotation?"

"[W]hen faced with a problem you do not understand, do any part of it you do understand, then look at it again."

"As a matter of fact, I do.  I thought it was very profound."

"OK then Alphonse, apply that here.  Find some part of this problem that you do understand."

"I understand the problem..."

"No,  you think you understand the problem, but clearly you don't.  If you understood it, you'd be able to solve it.  Find some simpler tests to pass."

I thought about this for a few seconds.  What was so hard about this problem?  The thing I'd been struggling with was how to deal with breaking the lines at spaces?  Each of my "solutions" was tangled up with hunting for just the right space to replace with a line end.

I looked at Jerry and said: "What if I solved the part of this problem that did not deal with spaces?  Lines that have no spaces only need to be broken once they've hit the column limit."

Jerry pointed at the keyboard, and I started again.  I wrote the same degenerate tests.


But then I changed tack and wrote a test that wrapped a line without spaces. That test was trivially easy to pass.


The next test was pretty obvious. It should continue to wrap a string without spaces, creating lines that are no longer than the column limit.


Jerry looked at the test and nodded. "How will you solve that, Alphonse?"

"I just need to put a loop into the wrap function." I said.

"I think there's an easier way." He said.

I looked at the code for a bit, and then said: "Oh! Sure, I could recurse."


The tests passed, and Jerry nodded approvingly. "That looks like a framework you could build upon. What's next?"

"Now that I can wrap lines without spaces, it ought to be easier to wrap lines with spaces!"

"Give it a shot." He said.  So I wrote the simplest test I could.  A space right at the column limit.


"Do you remember how you made that test pass last time?" Jerry asked.

"Yeah." I grimaced. "I use the replaceAll hack."

"Is that how you're going to solve it now?"

I looked at the code, and the answer was obvious.  "Of course not!" I exclaimed. "All I need to do is check to see if the character at the column limit is a space!"  and I wrote the following code.


"Why'd you put that wrap call in there?"  Jerry asked. "You're getting a little ahead of yourself, aren't you?"

"I guess, but it's kind of obvious that it belongs there.  Just look at the symmetry!"

"I agree." Jerry said smiling.  "Continue on."

The next test was just as obvious.  The space should be before the column limit.  So I typed the following:


"Passing this one is going to be tricky." I said under my breath.

"Is it?" Jerry queried.

I looked again, and it hit me.  "Oh, no, it's just a small change!"  And I typed the following.


The tests passed, and I was getting excited.  "This is so strange, the whole algorithm is just falling into place!"

"When you choose the right tests, Alphonse, they usually do."

"OK, so now let's make the column boundary really small so that it has to chop the string up into lots of little lines."


"That one passes right away!" I said.  Wow, I think we're done.

"Not quite." Jerry said.  "There's another case."

I studied the tests.  "Oh, there's the case where the character after the column limit is a space." I wrote the tests, and it was trivial to pass.


Jerry smiled as the tests passed. "That's the algorithm all right. But I bet you could clean this up a bit."

"Yeah, there is a lot of duplication in there." So I cleaned up my work with the following result.


I looked at the code in some astonishment. This really was a very simple algorithm! Why couldn't I see it before?

"You were right." I said to Jerry. "Now that I see this algorithm for what it is, it's kind of obvious. I guess choosing the right tests is pretty important."

"It's not so much choosing the right tests, Alphonse; it's about realizing that you are trying to solve the wrong test."

"Yeah, the next time I get stuck like that, and start guessing and flailing, I'm going to re-evaluate the tests.  Perhaps there'll be simpler tests that will give me a clue about the real solution."

And then I stopped myself and asked: "Is that true, Jerry? Is there always a simpler test that'll get me unstuck?"

Jerry was about to answer when a spitwad hit him in the side of the face.  Jasmine was laughing and running down the hall.  Jerry lept out of his seat to chase after her.

I just shook my head and wondered.