Wednesday, August 18, 2010

Why Clojure?

I have recently become quite an enthusiast for the language Clojure.  But why?  Why would someone who has spent the last 30 years programming in C, C++, Java, C#, and Ruby suddenly become enamored with a language that has roots that go back to 1957, i.e. Lisp?

During my first few decades as a professional programmer, I never learned Lisp.  I had heard of it, of course; though mostly in derisive terms.  People sneered about it with names like "Lots of InSignificant Parentheses".  So my view was not particularly favorable.

A few years ago, someone suggested that I learn Lisp by reading a book entitled: "The Structure and Interpretation of Computer Programs".  So I went to Amazon and ordered a copy from the used books section.  It arrived a week or so later, and then sat on my "to read" stack for a couple of years.

I started reading it about two years ago; and it changed everything I had previously felt and believed about Lisp.  It also changed a great deal of what I felt and believed about programming in general.  In short, the book was startling.

SICP is a literary masterpiece.  It's not often that you can say that a technical book is a page-turner, but that's just what I found SICP to be.  The book moves from topic to topic with rare ease and clarity, but more importantly it moves with purpose and mission.  As you read it, you can feel the authors slowly building a tension towards a climax.  The chapters fly by as you read about data structures, algorithms, message passing, first-class procedures, and so much else.  Each concept leads inevitably to the next.  Each chapter adds to the ever building tension.  By time you are half-way through the book, the sense that something important is about to change becomes palpable. 

And then something important changes!  Something you had not anticipated.  Something you should have guessed, but did not.  On page 216 they introduce a concept so familiar that most programming books start with it.  On page 216 they prove to you that you've had some wrong ideas about programming all along.  On page two hundred and sixteen, after talking about algorithms, data structures, recursion, iteration, trees, high-order procedures, scoping, local variables, data abstraction, closures, message-passing, and a plethora of other topics -- after all that, they introduce assignment!

And with that elegant coup-de-grace (which is not the last in this book!), they vanquish the concept that programming is about manipulating state.  With that one stroke, they force you to look back on all you had done in the previous pages in a new and enlightened way -- a functional way.

Moore's Law
Why is functional programming important?  Because Moore's law has started to falter.  Not the part of the law that predicts that the number of transistors on a chip doubles every two years.  Thankfully, that part of the law seems to still be in effect.  The part that faltered is the part that says the speed of computers doubles every two years. 


What this means is that our computers can still get faster, but only if we put multiple CPUs on a chip.  This is why we've seen all these multi-core processors showing up.  And that means that programs that need greater speed will have to be able to take advantage of the multiple cores.

If you've ever written multi-threaded code, the thought of eight, sixteen, thirty-two, or even more processors running your program should fill you with dread.  Writing multi-threaded code correctly is hard!  But why is it so hard?  Because it is hard to manage the state of variables when more than one CPU has access to them.

And this is where functional programming comes in.  Functional programming, of the kind shown in SICP, is a way to write code that does not manage the state of variables, and could therefore be partitioned to run in parallel on as many processors as you like -- at least in theory.  In practice it might not be quite that trivial; but one thing is certain.  Moving functional programs to massively parallel system will be easier than moving non-functional programs.

Why Clojure?
So why is Clojure the best option for a functional language?  After all, there are lots of functional languages out there.  Some are old, like Haskell, and Erlang.  Some are new like Scala and F#.  Why is Clojure the language that has everybody so fired up?  Here are just a few reasons.
  • Clojure is Lisp.  And Lisp is a functional, simple, well-known, elegant language. The syntax is almost laughably terse. This is in contrast to languages like F# and Scala which have a complexity and "quirkiness" reminiscent of C++.
  • Clojure is Java.  Clojure sits on top of the Java stack, and has the ability to inter-operate with Java with extreme ease.  Java programs can call Clojure, and Clojure can call Java.  You can write Clojure code that derives from Java classes and overrides Java methods.  In short, if you can do it in Java, you can do it in Clojure.  What's more there is a Clojure port for the CLR!  So Clojure may be the only functional language that inter-operates well with both major VMs.
  • Clojure implements Software Transactional Memory which means that any time a Clojure programmer want's to change the state of a variable, they must do so using the same kind of transaction management as they would use for a database. This enforces the functional paradigm do a degree that few other functional languages do.  The STM facilities of Clojure are elegant and simple, just like the rest of the language.  They do not intrude where they aren't needed, and they are simple to employ where state must be changed.
     
  • Clojure is fast.  Data structures in functional languages are immutable.  For example, you can't add an item to a list, instead you create a copy of the list with the new item added.  This copying could obviously slow things down a lot.  Clojure manages complex immutable data structures using a sharing technique that eliminates the need to make deep copies of those structures. This means that Clojure runs very fast. 
  • Clojure is supported.  There are tutorials and blogs.  There are IDE plugins.  And there are mailing lists and user groups.  If you program in Clojure, you won't be alone. 
Conclusion
The last few decades have seen us migrate from procedures to objects.  Now the physical constraints of our hardware is  driving us to make a similar kind of paradigm shift towards functional languages.  The next few years will see us experiment with many different projects using those languages as we try to figure out which functional languages are best.  I fully expect Clojure to be ranked very highly when the results of those experiments come in.

35 comments:

  1. I think the real power of Clojure is the JVM/CLR thing. Immutability, purity, and being functional in nature is shared across almost all other functional languages.

    ReplyDelete
  2. I really wanted to like Clojure, but once I pulled the covers open I hit the everything ad-hoc/by-convention wall that has kept me from Lisp all these years. That said, I think Clojure's approach to things, especially STM, really rock.

    ReplyDelete
  3. Very interesting read - Especially fun because SICP was also for me a gate opener to the land of Lisp.

    Best regards,
    Lau B. Jensen

    ReplyDelete
  4. @galilyou

    With the little I've played with Clojure, I think it's more than just the JVM/CLR. As Uncle Bob points out, it's the elegance and simplicity too.

    ReplyDelete
    Replies
    1. Elegance and simplicity?! That's all that need be said for me. Sold.

      Delete
  5. After 30 years of programming in C, C++ and Python I started reading SICP, The Little Schemer and some LISP books (as well as "Programming Clojure") and though I am excited about Clojure I am having a hard time getting motivated having to learn Java first. I won't go near a CLR language and recent patent actions on Java have got me wondering if its days are numbered. Wake me up from my cryogenic sleep when there's a native version!

    ReplyDelete
    Replies
    1. JVM is an attraction of Clojure. If Clojure goes the way of Haskell or Python, that is, it's own way, their will be no advantage for me. Java and JVM are two different things. JVM is wonderful, Java is not. If Java goes away, I will not miss it. I was considering Clojure also because of CLR but am finding out that programs for Windows are better and easier with JVM.

      Delete
  6. Clojure goes in the right direction, but unfortunately, the trend and research is in advanced type systems and some syntax.

    An intermediate solution is for someone to fork Clojure and give it some nice Python-like syntax

    ReplyDelete
  7. A really nice writeup on Clojure, Bob. I've been learning Scala lately and now you've tweaked my interest in Clojure to do a little compare and contrast. Thanks!

    ReplyDelete
  8. Like Jim & Roy, I really wanted to love Clojure, and there are some brilliant things there, but the looseness of the data model and the gnawing feeling that macros are no longer decisive enough an advantage to justify the cryptic syntax are driving me away.

    I'm doing a dual implementation of a personal project in Scala and Clojure and the Scala project has developed a commanding lead. After working almost exclusively in dynamic languages for ten years I'm amazed at how powerful and helpful a *good* type system can be.

    ReplyDelete
  9. This comment has been removed by the author.

    ReplyDelete
  10. Being LISP for JVM, I do not know how they can account for the JVM's lack for TCO? How do they do it? Do they do it? Does it force the compiler to change to a special goto in source before it compiles it?

    ReplyDelete
  11. Jaime, The JVM cannot do TCO, but clojure provide an special pattern

    (loop [...]
    ...
    (recur ....))

    this pattern indicate to the language that the recursion can be optimize. This will be like that til Java 7 (whatever that means)

    ReplyDelete
  12. Great post!

    I had already heard of SICP, but now I'm really compelled to read it and start delving deeper into the functional languages realm.

    ReplyDelete
  13. This comment has been removed by the author.

    ReplyDelete
  14. Thanks for the post and it's basically the same thing that you talked about in last Rails Conf (http://www.youtube.com/watch?v=mslMLp5bQD0).

    And thanks to put another book (and language) to my too learn stack, rsrs

    ReplyDelete
  15. Clojure forces you to write clean code. Even a couple of lines of Clojure can become a mess if you're not breaking out your abstractions.

    Maybe that makes it "harder" to use- because you must be disciplined.

    For the disciplined programmer, there is a wealth of power and flexibility unknown in the myriad popular ALGOL-esque languages.

    ReplyDelete
  16. You said "laughably terse". I think you meant "laughably simple".


    Your wording suggest that the use of sharing is specific to Clojure. It's not. Any functional language worth it's salt use sharing almost everywhere. Maybe you wanted to say that Closure's persistent arrays are faster than the equivalent in other languages? (I'm not sure that's even true, but at least it's testable).

    ReplyDelete
  17. *sigh* ... why can't we have an educated debate about programming language without bashing the ones we don't like?

    ReplyDelete
  18. @steve - Hi, welcome to the internet.

    ReplyDelete
  19. @hadi hariri the elegance and simplicity inherited from Lisp, so it should be JVM/CLR/Lisp :)

    ReplyDelete
  20. "Clojure is Lisp. And Lisp is a functional, simple, well-known, elegant language. The syntax is almost laughably terse. This is in contrast to languages like F# and Scala which have a complexity and "quirkiness" reminiscent of C++."

    I think what you're trying to get across here is that Clojure is simple, but I don't think this is the case:

    1. Clojure isn't actually a small or simple language. It's actually pretty complex and quirky. Pop quiz, what's the difference between pr-str, prn-str, print-str, and println-str? The different concurrency models (atoms, agents, refs) are all nice to have, but there should be one default (probably refs) that works for most things, and then have the others as libraries. Having them all in the core language means there's more to remember. The STM is great, but without a system to track effects, you can definitely blow off your leg by sticking something non-transactional within a transaction. Even the imports are non-obvious, what's the difference between require and use again? When I've been doing clojure for a few days, I remember the difference between them of course. I always have to look it up when I come back to clojure, though. There are a bunch more examples--the main issue is that Rich Hickey is a really smart guy and he made a language for smart people who can keep it all in their head. What about the 90% of programmers like me who can't keep a large and complex language in their head?

    2. One thing lisps teach you after you see a couple of them is that syntax complexity doesn't say anything about language complexity. The scheme of SICP and clojure are worlds apart in terms of complexity--scheme is much more simple. What about Liskell (Haskell + lisp syntax)? Sure it's a lisp, but that doesn't make it simple. Unsurprisingly, it's about as complex as using normal haskell--just with lisp syntax. What about languages like BitC (a functional language for low level programming) that happen to use lisp syntax? Again, syntax has almost nothing to do with language simplicity.

    ReplyDelete
  21. I learned clojure last year and ported my game system to it from haXe, i didn't know functional, or lisp before i started - well only what javascript/haXe provide. It was an extremely interesting experience coming from an oop only background and I have come away with my code style, imo, considerably improved, and also the way i structure code better much flatter.

    Rich's videos are fantastic and well worth everyone watching them, he lays out the case for functional, what's wrong with oop, and basically leaves you realising there are better ways to get stuff done.

    Finally, however, I went back to haXe because I really can't live without strict typing and i preferred V8/Node.js as a runtime. I think the much debated static/dynamic dilemma is the main deciding factor for Clojure.

    ReplyDelete
  22. I have been learning both Clojure and Scala lately and both are great languages worth learning. Clojure is a very cute and elegant language. It forces you to rethink what you have learned about (object oriented) programming. It has been an eye opener for me.

    Despite Clojure's elegance, I still prefer Scala for most day to day programming tasks due to its (powerful!) static type system. I do believe static typing can prevent many bugs and make code easier to read.

    ReplyDelete
  23. F# isn't complex. It is MS based, which you surely don't like, but it sure as hell isn't complex. Most everything can be done with let, match, and fun.

    ReplyDelete
  24. what i find most interesting about SICP that after chapter 5, when assignment and mutability are introduced, it makes it plain how much more complex things become, even in single-threaded programs. the book doesn't fall apart, but my comprehension does.

    i used to think that this was a limmiting factor for Scheme and LISP in general, but comming back to SICP after a few years (and having lost my youthful ego quite a bit) I now see that this is wrong thinking.

    OOP with mutable objects is definately a great way to abstract business logic, but I think we've been going down the wrong road for the past 20-odd years by trying to apply this hammer everywhere. now that we are forced to face up to multithreaded mutability headaches, we need to take another look at the functional side.

    the issue has been: we have all these neat Java (and .Net) libraries that we don't want to give up. LISP is fine and all, but there aren't many GUI libraries or web frameworks for it, are there? how am i going to make my living in it? Practical Common Lisp addresses only some of our needs.

    well, that's the other beauty of Clojure. as well as addressing the need for writing functional, immutable, multithreaded code elegantly and comfortably, it has embraced the most widely available platform on the planet. consuming and extending Java objects from within Clojure is actually easier than in Java itself, is much less verbose, and the syntax is far simpler(and the number of paretheses involved is actually less, with less of a soupy mix too).

    that's why it excites me

    the supporting tools are still maturing and require a steep learning curve (and seem to be emacs-centric which turns some people off... i can't imagine why?), but they are coming along, and tools like lieningen already are far better than ant. Clojure may not be the Omega language but it's definately one to watch and play with, because what ever comes to dominate this space is going to feel a lot like it.

    ReplyDelete
  25. @Kobold
    You got somethings really wronge. Clojure is simple because its build with some simple primitives all the rest of it is in librarys and we can change it without changing the compiler.

    Why there are so many print methodes I don't know I don't know. I agree that is is a bit wired. Just use something and when you run into problems look for something diffrent. Thats maybe bad library design but it does not make the language complex.

    I think you are totaly wronge with the default concurrency model. Concurrency is hard and you NEED to think about it when you programm. There is no silber bullet. refs do add quite a bit of overhead and if you just wont a simple counter you don't need a ref.

    I think for that reason the functions to update the concurrency primitivs are the same.

    I found the namespace thing (use, require ...) difficult too but once you read understand how the namespace system workes this falls out naturly its accully really easy and elegant.

    ReplyDelete
  26. @Kobold, Clojure is not complex. There really is almost no syntax. It sounds like your problem really is about learning a new library of functions. That would be true with any language. Also, Clojure gives you simple meta programming because code is data, which simplifies tasks even more. I work mostly with business level programming and Clojure is dead simple. Things you do over and over again are abstracted away making your code easily readable and smaller: map, reduce, filter, etc. I looked at Scala and frankly I found it more of an academic exercise in 'typing' then a productive language, case classes not withstanding.

    ReplyDelete
  27. Uncle Bob,

    I have a small brain and am working in bioinformatics domain currently. Day-by-day my frustration is increasing as to which direction I shud steer my career. Things bothering me are:

    1) Java's future ? (at oracle's hands)
    2) Perl, Python, PHP and Ruby failure with MP.
    3) Scala huge syntax to learn??

    Finally, after repeated search in net, I got that Clojure and Haskell are good.

    But........., Clojure has better library support due to Java. It needs JVM :-( Haskell has space leak issue. Oh my god ??? my head is spinning, where shud I go ?? What shud I learn ?? AAAHHHHH :-((

    Siva

    ReplyDelete
  28. I am not sure about book (haven't read it yet) but surely your post is startling :)

    Javin
    Why multiple inheritance is not supported in Java

    ReplyDelete
  29. Clojure is also a declarative language. A declarative language, like SQL, is inherently more expressive. I've wanted to use a declarative language in production programming for way over 20 years. Clojure now makes it possible. I am so very happy. Lisps, Hope, Scheme, Haskell. None were ever ready for the type programming I do at work. I would die without SQL. I will feel the same about Clojure, I know.

    ReplyDelete
  30. It’s actually a great and helpful piece of information. I am happy that you shared this helpful information with us. Please stay us informed like this. Thank you for sharing.


    Printhead911

    ReplyDelete
  31. It's too complicated. Maybe Java code is longer but it is much easier to read because it is talking to You and has a clean structure, whereas this LISP is nesting and neting and nesting. I compared two code versions. One Java one Clojure. It took me 20 secs to understand Java and Clojure I did not get at all. And that concurrency is no prob at all. Once You understood threads You take a piece of paper and build cool stuff.

    ReplyDelete
  32. I've been playing with the idea of clojure for a while. This blog post helped seal it for me :) Thanks. Great post.

    ReplyDelete