QA or When do you flip a pancake?

When do you flip a pancake?  We know what a good pancake looks like.  It's nicely brown on both sides, with a cheery ring of white along the rim.  But when you start with a pitcher of pancake batter, and a restaurant full of hungry lumberjacks,  how do you get your pancakes to meet those requirements?

The requirements are completely unambiguous.  Brown sides.  White ring.  What could be simpler?  So let's design a development process that will make our pancakes come out right:

1. Developer pours pancake batter.
2. Developer flips pancakes at a rate based on the schedule.
3. Developer throws "done" pancake over the wall to QA.
4. QA inspects pancake.  Rejects are thrown back over the wall to the developer to "fix". 

Clearly, unless the developer is really good at estimating, the result of this process will be a lot of wasted pancakes, delayed breakfasts, angry customers, and accountants who, concerned about costs, suggest that pancake developers should be outsourced.

OK, so let's design a different process.  Based on our observations of pancake  cooking, we notice that the perfect time to flip the pancake is when the bubbles on the top have just popped, and the surface starts to look slightly dry.

1. Developer pours pancake batter.
2. Developer applies the test to determine when to flip the pancake.
3. Developer sees pancakes are perfect and ships them.  
4. QA simply observes the pancakes on a statistical basis.

When we apply this process, we find that QA hardly ever rejects a pancake.  Breakfast cooking becomes efficient.  Customers are happy.  Accountants, concerned about costs, suggest downsizing the QA group.

Moral of the story:  Developers are not done until the acceptance tests pass!

Who designs the acceptance tests?  QA of course.

I recently read a blog by Dennis Stevens entitled We Are Doing QA All Wrong. He is, of course, quite right.  We are, and have been, doing QA all wrong for years.  Indeed, the current role of QA only exists because developers have been so bad at doing their jobs.

QA is at the end of the process because development never learned when to flip the pancakes. Decades ago, frustrated by the terrible quality coming out of development, managers created an inspection step at the end of the process.  QA (or QC as Stevens would have it) was born.  This role for QA reinforced the bad behavior of development that spawned it.  Because QA was at the end, developers didn't need to care about getting things right.  Developers no longer had to worry about bugs; that was now QAs role.  All developers needed to do was to "say" that the code worked as far as they were concerned, and then throw it over the wall.  Deadlines are a lot easier to make when you don't have to make the code actually work.

Management, in order to justify the existence of QA to the accountants, who were very concerned about the cost, began to measure QAs efficiency.  The more bugs that QA found, the better the job they were doing.  Notice how insane that is!  The only way for QA to be measured well is for development to screw up royally.  The more bugs that developers create the better QA looks. 

And so an unholy alliance of blame avoidance was born.  Developers can appear to meet deadlines by delivering crap.  QA is measured highly because they find lots of bugs.  Everybody is happy -- except the end customer, and the accountants who are very concerned about costs.

Look, this isn't rocket science.  If QA's input is primarily at the back end of the process, you are going to have lots of waste, rework, delay, and angry customers.  I mean:  Duh!  (Pronounced "DUUU-uuuh")

So where do we put QA?  How do we break the unholy alliance, and stop avoiding the blame?  Simple!  Move QA to the front.

What if QA's job was not to test the product, but to design the tests that developers use to know when to flip the pancake?  If QA created suites of automated acceptance tests using tools like FitNesse, then developers would know when they were done.  Developers would continue working until the acceptance tests all passed.  Indeed, it would be the developers who executed those tests.  

This is how good agile teams are organized.  QA (and development) works with the business to define the requirements as a suite of automated tests that developers execute to know when they are done.  When all the tests pass, QA makes a final exploratory pass over the product, and sends it on to production.

That last step is a little more complicated than that, but is beyond the scope of this article.  Suffice it to say that exploratory testing is a craft in it's own right that needs to be part of the process on an on-going basis.

So, in the end, when do you flip a pancake?  What is the definition of "done"?  Developers are done when the automated acceptance tests design by QA all execute and all pass.

Why Clojure?

I have recently become quite an enthusiast for the language Clojure.  But why?  Why would someone who has spent the last 30 years programming in C, C++, Java, C#, and Ruby suddenly become enamored with a language that has roots that go back to 1957, i.e. Lisp?

During my first few decades as a professional programmer, I never learned Lisp.  I had heard of it, of course; though mostly in derisive terms.  People sneered about it with names like "Lots of InSignificant Parentheses".  So my view was not particularly favorable.

A few years ago, someone suggested that I learn Lisp by reading a book entitled: "The Structure and Interpretation of Computer Programs".  So I went to Amazon and ordered a copy from the used books section.  It arrived a week or so later, and then sat on my "to read" stack for a couple of years.

I started reading it about two years ago; and it changed everything I had previously felt and believed about Lisp.  It also changed a great deal of what I felt and believed about programming in general.  In short, the book was startling.

SICP is a literary masterpiece.  It's not often that you can say that a technical book is a page-turner, but that's just what I found SICP to be.  The book moves from topic to topic with rare ease and clarity, but more importantly it moves with purpose and mission.  As you read it, you can feel the authors slowly building a tension towards a climax.  The chapters fly by as you read about data structures, algorithms, message passing, first-class procedures, and so much else.  Each concept leads inevitably to the next.  Each chapter adds to the ever building tension.  By time you are half-way through the book, the sense that something important is about to change becomes palpable. 

And then something important changes!  Something you had not anticipated.  Something you should have guessed, but did not.  On page 216 they introduce a concept so familiar that most programming books start with it.  On page 216 they prove to you that you've had some wrong ideas about programming all along.  On page two hundred and sixteen, after talking about algorithms, data structures, recursion, iteration, trees, high-order procedures, scoping, local variables, data abstraction, closures, message-passing, and a plethora of other topics -- after all that, they introduce assignment!

And with that elegant coup-de-grace (which is not the last in this book!), they vanquish the concept that programming is about manipulating state.  With that one stroke, they force you to look back on all you had done in the previous pages in a new and enlightened way -- a functional way.

Moore's Law
Why is functional programming important?  Because Moore's law has started to falter.  Not the part of the law that predicts that the number of transistors on a chip doubles every two years.  Thankfully, that part of the law seems to still be in effect.  The part that faltered is the part that says the speed of computers doubles every two years. 

What this means is that our computers can still get faster, but only if we put multiple CPUs on a chip.  This is why we've seen all these multi-core processors showing up.  And that means that programs that need greater speed will have to be able to take advantage of the multiple cores.

If you've ever written multi-threaded code, the thought of eight, sixteen, thirty-two, or even more processors running your program should fill you with dread.  Writing multi-threaded code correctly is hard!  But why is it so hard?  Because it is hard to manage the state of variables when more than one CPU has access to them.

And this is where functional programming comes in.  Functional programming, of the kind shown in SICP, is a way to write code that does not manage the state of variables, and could therefore be partitioned to run in parallel on as many processors as you like -- at least in theory.  In practice it might not be quite that trivial; but one thing is certain.  Moving functional programs to massively parallel system will be easier than moving non-functional programs.

Why Clojure?
So why is Clojure the best option for a functional language?  After all, there are lots of functional languages out there.  Some are old, like Haskell, and Erlang.  Some are new like Scala and F#.  Why is Clojure the language that has everybody so fired up?  Here are just a few reasons.

• Clojure is Lisp.  And Lisp is a functional, simple, well-known, elegant language. The syntax is almost laughably terse. This is in contrast to languages like F# and Scala which have a complexity and "quirkiness" reminiscent of C++.
  
• Clojure is Java.  Clojure sits on top of the Java stack, and has the ability to inter-operate with Java with extreme ease.  Java programs can call Clojure, and Clojure can call Java.  You can write Clojure code that derives from Java classes and overrides Java methods.  In short, if you can do it in Java, you can do it in Clojure.  What's more there is a Clojure port for the CLR!  So Clojure may be the only functional language that inter-operates well with both major VMs.
  
• Clojure implements Software Transactional Memory which means that any time a Clojure programmer want's to change the state of a variable, they must do so using the same kind of transaction management as they would use for a database. This enforces the functional paradigm do a degree that few other functional languages do.  The STM facilities of Clojure are elegant and simple, just like the rest of the language.  They do not intrude where they aren't needed, and they are simple to employ where state must be changed.
   
• Clojure is fast.  Data structures in functional languages are immutable.  For example, you can't add an item to a list, instead you create a copy of the list with the new item added.  This copying could obviously slow things down a lot.  Clojure manages complex immutable data structures using a sharing technique that eliminates the need to make deep copies of those structures. This means that Clojure runs very fast. 
  
• Clojure is supported.  There are tutorials and blogs.  There are IDE plugins.  And there are mailing lists and user groups.  If you program in Clojure, you won't be alone. 

Conclusion
The last few decades have seen us migrate from procedures to objects.  Now the physical constraints of our hardware is  driving us to make a similar kind of paradigm shift towards functional languages.  The next few years will see us experiment with many different projects using those languages as we try to figure out which functional languages are best.  I fully expect Clojure to be ranked very highly when the results of those experiments come in.