As a person who learned the hard way about how tough it is creating web applications that can handle state and concurrency issues (if you’ve ever seen MySQL replication lag times as high as 30 *minutes* you can sympathize), the rise of languages designed from the ground up to support concurrency has caught my eye. If you’ve heard of Erlang or Haskell you are somewhat familiar with the topic.

Then one day I ran across Clojure, a language that is a dialect of Lisp but runs on the JVM. My, my, what in interesting combination. I’ve talked about Clojure before, but I have not talked about the amazing person behind it, Rich Hickey.

The idea of creating your own programming language is one that I find extremely intimidating, not having a formal background in computer science. I have a vague idea of how you would do it: write parsers for your code and then compile them into byte code that can be run by something. Very simplistic, I know.

So Rich Hickey is a one of these programming gurus who I am sure is a super-nice guy but I would be totally intimidated to have any sort of meaningful programming-related discussion with. First of all, he’s a Lisp hacker. Some of my friends have described Lisp as “the cockroach of the programming world, which will be around long after the sun of other languages have dimmed” and I can’t say that I disagree much with that statement. For someone who learned to program by teaching himself PHP, Lisp’s syntax takes a lot of getting used to.

But then I think the best trick was making Clojure run on the JVM. Why do I think this is significant? It means that if there is a feature or a library that doesn’t exist in Clojure, you can always use a Java equivalent (if such a one exists). In Tim Bray’s “Concur Next” series he showed an example where he used a Java library in Clojure to help him speed things up.

So not only does he have a well-designed and well-thought-out programming language, he is also great at explaining programming concepts. Go and watch this video on the topic of basic principles like state, identity, value, time, and many others in order to understand how to build concurrent and parallel systems. Fascinating talk, and I highly recommend seeking out other talks that he’s done as well.

All this stuff is absolutely fascinating to me, because my entire programming career has been spent in the stateless environment of the web. Oh sure, we can fake state and concurrency using sessions and cookies, but they are really just hacks. Does a web application need to be stateful and/or concurrent? The answer is, of course, it depends. Online games with multiple participants definitely need to worry about sharing state across multiple clients, and any web application that is taking large amounts of data and processing it (like munging it together based on aggregate conditions) could benefit from being able to crunch the data concurrently, with all jobs reporting stuff as they go. Any task that would benefit from being broken down into smaller tasks that can be run at once is a candidate for concurrency.

So, why does Rich Hickey make me feel stupid? Not only is he also capable of explaining the hows and whys of a concurrent programming language, he is encouraging people to think ahead to the next generation of programming problems and potential solutions to those problems.. It’s highly unlikely I will ever create a full-blown, general-purpose programming language. But Rich Hickey’s work has gotten me thinking about programming in event-driven environments, and wondering what tasks that my current applications perform could benefit from concurrency and parallelism.

So as we start 2010, and you’re a web developer, what is out there for you to play with? Let’s see…

• Mature web application frameworks for the Big 3 open source scripting languages (PHP, Python and Ruby) that can help you, when used properly, rapidly build applications, if you are willing to spend the time understanding the limitations of frameworks. You might hate frameworks, but you certainly cannot ignore them. Yes, you could consider WordPress itself to be a framework
• Multiple languages that will run on the JVM that can be used for web development, if you are willing to spend the time understanding the JVM and how to get these things talking to Java as well. Ignore the Java haters, I think this is a good trend.
• Multiple actually-usuable-outside-of-small-projects alternatives to RDBMs, if you are willing to spend the time to figure out when you should use them. The trend is towards having ridiculous amounts of data, and it appears that in the open source world the RDBMs is not up to the task. Think Cassandra, Redis and Hadoop along with Pig.
• multiple mature Javascript libraries that make creating all that awesome Ajax goodness easy, if you are willing to spend the time to pick one with the features you like
• multiple mature (or soon-to-be mature) functional programming languages that can be used to build web applications, if you are willing to spend the time to learn functional programming. I mean, really, how can you not look at Clojure and Erlang and not be impressed?

But again, who cares what I think? If you do care, then you are at the same place as me: paralysis by analyses. What framework should I pick? Do I go document database or key-value store? If I don’t want to learn Java, can I do some stuff with Scala or see if Jython is up to the task? Can jQuery do long-polling / comet stuff? Is functional programming too weird for a web monkey like me to figure out?

Crap, that’s a lot of stuff to worry about, isn’t it? Is it any wonder I get stressed out trying to figure out how to build stuff at work? Or what stuff to dabble with in my spare time? We are suffering from an embarrassment of riches, which is being hidden by the constant religious wars that developers seem to get embroiled in. If it’s not “my language is better than your language”, it’s “my editor is better than your editor”. “My programming paradigm is better than your programming paradigm.” “If you did interesting work, you’d get accepted to speak at conferences more”. (Sorry, that was me arguing with myself).

If anything, I have found that my focus is starting to change. I’ve started realizing that what is really happening is that despite there being ridiculous amounts of choices of tools to solve problems, it has become EASIER to solve these problems. In my planning for a work project I stumbled across the now-infamous Rasmus Lerdorf “The no-framework PHP MVC framework” blog post. While Rasmus is a programming god, he is also a very smart guy. He understands that programming is about solving problems, and he (at least to me) seems to not have the ego that says you must build up your own complicated solutions to problems.

This has gotten me to consider how I had been approaching problems. Too often, I have been approaching them from the code out to the display. Worrying about the technology, instead of how people will use it. Rasmus proposes an easy problem – a sample Ajax app that uses exactly 4 things: a scripting language, a Javascript library, a data storage source, and a data exchange format. In his specific example he used (in order) PHP 5, Yahoo YUI, sqlite and JSON. You could’ve done this with Python, JQuery, MySQL and XML. Or Ruby, Dojo, CouchDB, and JSON. My point is that we often find ourselves caught up in what we are USING to build something rather than solving the problem.

I have really dislike the idea of New Year’s resolutions, because they are almost always broken. But in this case, I think a few are warranted: this year I resolve to do a better job of figuring out the problem I need to solve first and worry about the technology needed second. Sometimes my desire to feed the part of me that wants the new and shiny takes control and creates a potentially bad solution. Sure, it would be awesome to get paid to learn a new programming language and a new web application framework to go with it, but if it gets in the way of solving the problem, well, you’ve suddenly become the problem instead of the solution. Sounds so cliched, doesn’t it?

So, in the spirit of that, let’s look at 3 problems I need to solve in the next little while that I perhaps can also use to feed the Imp of the New and Shiny:

• Analysis of some MySQL slow-query logs are needed at work. First solution was to play with a tools specifically for analyzing the slow query log. They provided summaries that were simply not useful to me. Perhaps Pig + Hadoop will give me the high-level solution to collect the information we seek.
• I have a very simple idea for collecting absurd stories you’ve told to your kids. My first instinct was to create the project using a web application framework. Upon reflection, perhaps the combination of WordPress and then a custom-written story submission and approval system is the better approach.
• Plans are being made to create the next platform our company will use for online fantasy games. Since we cover multiple sports, I have no desire to create a separate version of the platform for every sport. Instead, inspired by the realization that Rails is nothing more than a DSL (or Domain Specific Language) on top of Ruby, I have started work on a very simple DSL for defining scoring rules. Wouldn’t it be awesome for the games admin to be able to select a stat category for a sport (say, rushing yards in football) and then use “(1 point each 10) + (5 at 100 total)” as the scoring rule? Parsers are actually pretty easy as long as your DSL is consistent.

I hope I’ve made sense here, as I’m trying to show how I wish to change my approach to solving programming problems. There is no doubt that I will pick up the new and shiny as I find a way to solve these problems. Some might be written in PHP. Some will be written in Python. Some will be written in ScoreRule (to pull a name for my sports DSL out of my ass). But the real winner is the people who will use these things. Shhh, don’t tell anyone: I’m thinking about the end user for once instead of the programmer.

Concurrency is coming, better be ready

The Continuation Mondad in Clojure. Even if you hate Lisp (I’m looking at you, Kevin) or don’t give a damn about Clojure it is worth reading about to understand how you can write concurrent applications.

Real-time web applications are coming, better be ready

I have a new project I am going to be starting at work, one that can potentially involve real-time information. Check out this series on how to build real-time web applications using Django, Python and a javascript library called Orbited. When you combine this with the you-can-see-it-on-the-horizon-but-not-quite-here-yet concept of web sockets, I can see some of the concepts of what I called Web Pi (3.14159, get it? Get it? Um okay, any way) starting to take shape.

TDD With PHP is already here, where have you been?

Giorgio Sironi has gone and written the book I was thiking of writing this winter. If you want to learn more about doing TDD with PHP, go and grab a copy of his e-book. It’s 100% free, but I liked it so much (and admired him for tackling a topic I have interest in) I donated $10 to him. You should do the same.

Chris should not post links on Twitter that cannot be discussed in 140 characters or less

Finally, on Twitter I posted about this blog post which talked about how it’s only people who are financially secure who pound away at the “only do what you love” message. When you don’t have to work, that’s easy to do. I got into a long conversation with Amy Hoy about this topic, although Twitter is a crappy medium for doing this sort of thing.

I respect Amy a great deal, so I was kind of bummed when she and I appeared to disagree on some the key points of the blog posting. Basically, “do what you love” is a lie because it is not always possible. “Do what you do, with love” is a better approach to thing. Not everyone gets to work on exactly what they want to, so perhaps I am showing some signs of neo-puritanism or something when I said “well, even if you have a crummy job you should be doing it as best as you can.” I proceeded to be smacked around by several people who do not share that philosophy. Amy seemed to feel that the blog article was (to use her words) bullshit and was advocating “eating a shit sandwich and liking it” by giving people justification for “hiding behind the mortgage” as an excuse for not taking chances. It will no doubt surprise Amy that I happen to agree with her, but utterly failed to get that across on twitter. My apologies to Amy for that. We’ve all worked crummy jobs, and it has been my personal experience that the best way to get out of a crummy job is to work as hard as you can to build up your skills so you are ready to leap on better opportunities when they come around.