@TheKeyboard

How Not To Suck At PHP

2012-02-03T00:00:00-05:00

Since I hadn’t blogged for a bit I thought I would ask my Twitter peeps what they might want to hear my rant about. One of the more interesting suggestions I got was from Travis Northcutt who said the following:

“Got it. I have no formal CS background, but I’m trying to learn enough to build (something). I know enough PHP to kind of, sort of get by but am trying to solidify my foundation. So if I’m being selfish, write a post to someone in my position who wants to not suck at building stuff.”

This is actually a great question. Although the self-centered answer is to start off by buying my book, that isn’t really the question that is being asked.

I thought about this question for a while and have some thoughts on what it really means to know how to not suck at building things using PHP. In my never even remotely humble opinion I think the key is to understand what PHP is really good at.

I have spoken at length about my opinion that PHP won the web war because it was so tightly coupled with HTML and very early on had support to run with Apache via mod_php. However the world has changed, and the web war itself has changed. I think I can say with a smirk that the web war is now an asymmetrical one, where the expectations of behaviour by users is totally different.

But PHP is still in a really good position because of its true nature: PHP is glue. It’s glue that binds things together in a way to quickly produce HTML output. There is a reason the LAMP stack (with PHP as the “P”) was so successful: if you had to get data out of a database and onto an HTML page there was no easier way.

One of the trends that I see in PHP is that the frameworks that sit on top of PHP are starting to ask for influence over the development and direction of PHP itself. I have mixed feelings about this. On the one hand, it is good that people push for newer features in their chosen programming language. I think the PHP core is not necessarily against these suggestions but they move at different pace than the framework people. But my other concern is that we don’t end up with a framework driving the language.

When you think of Ruby, unless you are in denial or a really old Ruby hacker, you think of Rails. Without Rails Ruby does not get the push that it did and continues to get. It’s a good language but it would’ve languished in obscurity if not for DHH.

So what does this have to do with not sucking at building stuff in PHP? I think it’s important to understand that you can build cool stuff without relying on a framework. Heresy, I know. But if you keep thinking of PHP as glue it totally makes sense. How many applications are REALLY just PHP? They are really web server + data source + Javascript + CSS + HTML. Think about that for a minute. PHP is the glue that binds all that together.

The danger for people like Travis who are starting out with PHP is that they will get caught up with a mindset that a framework is the solution to their problems. It is one solution to the problem of building a web application, but failing to understand how to write PHP that behaves like glue will lead to a lot of problems later. Because when the time comes for you to do something that the framework did not anticipate you will be fucked. And you will be incorrect when you blame the framework.

So my advice to Travis is that he should worry about learning to use PHP like glue and correctly identify the problems he is trying to solve NOW instead of worrying about the problems he might have to solve later. There will be time to fix your problems. Some of those will be solved by using tools that are not written in PHP, but PHP can still glue them together. NoSQL data sources. Distributed workers. Queuing systems. Real-time document searching capabilities. These are all things that are NOT written in PHP but it is extremely likely that someone has written a library in PHP that can talk to them.

Hope that helps you Travis!

PHPUnit aborted fix

2012-01-17T00:00:00-05:00

A while back I ran into a problem on a server where PHPUnit kept reporting “aborted” no matter what tests were run. That was a pretty annoying bug. I never did find out what the problem was as I moved onto other problems and chalked that error up to some undiagnosed weirdness on that particular server.

From time to time I would get asked on Twitter if I had ever solved the problem. My answer was always “no, and if you do solve it please let met know how you fixed it.” Today, my friends, was the day.

Kudos to Damian Katz for coming up with a solution. Without digging deeper into it, I would imagine that you could alter the phpunit.xml file (if you are using one in your setup) to automatically use the -dzend.enable_gc=0 option every time you run a test.

How did he figure it out? He tracked down this bug that seemed to be displaying the exact behaviour that I had reported way back when. It seems odd that garbage collection would cause this problem, but perhaps it was a perfect storm of lots of small tests causing over-active garbage collection which in turn causes the PHP interpreter to abort whatever it is running.

This is another case where I would love someone with knowledge of PHP internals to try and explain why it is happening.

CodeMash 2012 Review

2012-01-16T00:00:00-05:00

Last week I attended CodeMash 2012 as a speaker and gave my talk on building testable PHP applications (don’t forget to buy the book that the talk inspired). Held in Sandusky, Ohio in the cold of January it is an interesting conference. Unlike most conferences that I attend, it is multi-disciplinary. There were talks on .NET, Java, Ruby, Python, and two lonely talks about PHP.

For me, this is somewhat uncomfortable. It is only natural that people who use certain tools tend to flock together at a conference. I joked that where were only 3.7 PHP developers (with someone stating that the 0.3 part is part of someone that I cut off for not writing tests) but the fact was that unless you knew members of multiple communities you were going to have to be content to feel a little excluded.

Anyway, I did have a really good time hanging out with my friends and making some new ones. It’s a really good idea to talk to as many people as you can, even they are not using the same tools you are using. You can gain some perspective.

Let’s go over the talks I attended and my thoughts on them.

It’s the Little Things (Brad Colbow)

Brad is a freelance graphic designer from the Cleveland area. He also does an online comic called The Brads so, as expected, his presentation was full of all sorts of awesome visuals. Also, I thought that I talked fast, this guy made me look like one of the Ents from Lord of The Rings.

The main take-away I got from his talk was that it never hurts to pay attention to the little things. I realized that this holds true from both an UI/UX perspective and in your code. A bunch of bad little things always turn out into one huge bad thing.

His slides describing the blog posting he made about how crazy it was to check out an audio book from his local library were awesome.

Vagrant: Virtualized Environments Made Simple (Matt Stine)

Matt took what he said was normally a 3 hour tutorial and crammed it into a very informative 1 hour talk. It did help that I had been considering the use of Vagrant anyway, but I got a better picture of more practical uses of Vagrant. I did not know about veewee as a tool to help you create your own Vagrant boxes.

This is of interest because I had a hankering to play around with an Arch Linux VM but found the existing base boxes out there, well, not working with the version of Virtualbox I had installed. I think veewee will let me pursue that further.

Breaking the Sound Barrier with Node.js on Windows and Azure (Glenn Block)

While sitting in this talk I took a picture and tweeted about how it was time to make myself “personally uncomfortable” by sitting in a talk involving a technology that I don’t use running on a platform I don’t use. What I was hoping to find out was some more basic information on Node.js and why it is such a big deal.

To be perfectly honest, the more I see Node.js the more I start to understand the allure of it. What really bothers me is that so many people are gambling their applications on a tool that hasn’t even reached 1.0 yet. I understand the power of how Node.js gets things done. Evented + concurrency + JS is a platform that has lots of appeal to people who maybe haven’t done that sort of work before. Not that I’m including myself in the group that “gets it” but at this point in time I’m still on the “I will use Twisted/Tornado if I need to create event-driven, high-concurrency web servers.”

Maybe when Node.js gets to 1.0 I will think differently.

Azure itself didn’t really thrill me that much, but I do see the advantage of that environment to people who have already made a large investment on the server side in Microsoft. What is really admirable is that they are making some great contributions back to the open source community. The SDK for running Node.js on Windows is open source, and I must give them credit for that.

Code Kata and Analysis (Jim Weirich)

This was a talk that I went into expecting to be surrounding by Ruby hipsterism. After all, this is the guy who created Rake and works for a well-known Ruby consultancy. Once I saw him I knew I was in for a treat. Older guy. White hair. Unix-style beard. Runs emacs.

I was not disappointed.

A code kata is a programming exercise that serves a few purposes. You do it in order to learn how to implement common algorithms in different languages and learn how to solve the easy problems by habit, with the goal of then freeing up your mind to solve hard problems.

The code kata this time was to create an arabic number to Roman numerals converter. The language of choice was Ruby (no big surprise there) and I was even happier to see that it was a test-driven kata using Rspec.

This brings me around to a topic that makes me super uncomfortable. It seems to me that testing has become an integral part of the Ruby and Python ecosystems. It does not appear to be common at all in the PHP and .NET communities. I care about this sort of stuff so it does pain me to see such opposition in the form of apathy and excuses amongst PHP programmers. I wish I could surround myself with developers who feel like I do about the value of automated testing in delivering solid applications. Maybe in the next life.

It was also good to see someone walk you through the whole TDD practice and refactor stuff after getting a particular set of tests working. In the end we had two great examples of code: an easy-to-understand converter and tests that are easy to understand.

Dealing with Information Overload (Scott Hanselman)

One of the more dynamic speakers I saw, Scott gave a talk to a packed room on tips for dealing with information overload. From him I learned some new techniques I’m going to be phasing in:

No reading email until noon, and then “office hours” from a certain time onwards when anyone can speak to you
Instapaper is awesome because I can flag things to read later and get them delivered to my Kindle on Friday afternoons for some after-dinner reading
If this, then that is an extremely interesting tool that can be used to trigger events that help you delegate things into the future
The Pomodoro Technique is a very useful tool to help with short-term focus. Cut out the distractions and concentrate for 25 minutes. Even I think I can do that.
Keep your email responses very short (he reccomends Five Sentences) and instead write a longer response to that issue and post it somewhere others can find it, whether it is a company wiki or on your own blog. That way, next time someone asks you that same question you can point them there

Excellent talk full of lots of excellent ideas. I knew it was gonna be awesome when he put this video up to entertain us before his talk.

A Few of My Favorite [Python] Things (Mike Pirnat)

Mike is a long-time Python programmer who shared with us a “series of lightning talks crammed into one presentation”. While I have some basic Python knowledge I did see a bunch of really interesting libraries and other tools that I did not know about. He covered so much, so fast that I was glad he was going to post a link to the slides on his blog.

It also reminded me how, coming from the PHP world, languages like Ruby and Python really are more terse. You might not think it matters (or even hate significant whitespace) but for some people the elegance that can be found in Ruby and Python code often makes it easier for others to understand what you are doing. I know that I feel like I’m doing more with less when I write Python code.

Then I gave my own talk. I was disappointed that I actually ran out of time(!) for it. In the scheme of things, 90 slides is about 20 too many for a talk on as dense a topic as unit testing tools, strategies for writing code, and complementary tools to make the whole thing easier.

I have identified abotu 20 slides that I think I could chop out and turn into 2 slides of the “and if you’re interested in some tools that I think are helpful” type.

I had a good time (including hanging out at the Kalahari’s water park with my kids) at CodeMash 2012 and hope to go back for CodeMash 2013.

Why so little PHP?

I was asked about why I felt that there were so few talks about PHP and what could be done to make it better. Well, when only 7 people even submit PHP ideas I think there is a problem of visibility.

Given that this conference covers a wide variety of technologies I think it is safe to say that there was quite a bit of overlap. 31 .NET talks? Really? There is nothing wrong with .NET but do you really want to exclude an entire sector of the web programming world at your conference?

Also, there is the issue of speaker reimbursement I know that the official policy of CodeMash is “we cannot guarantee speaker reimbursement”. That’s cool but that will also stop talented people who are not willing to travel from where they are to the Cleveland airport and then hitch a ride to the Kalahari. Maybe in the PHP world we have gotten spoiled by those who run conferences since they tend to cover a significant portion of the costs a speaker might incur to get there.

There is no easy answer to how to get more PHP people to submit talks and to attend the conference in general. It probably also doesn’t help that the conference sold 1200 tickets in 20 minutes, making it that much harder to get a ticket.

I’d love for more PHP folks to come to this conference and get their horizons expanded with a glimpse at what other communities, just as full of energetic people but with different focusses, are doing.

Better Remote Code Development

2012-01-05T00:00:00-05:00

I am struggling to find a better way to handle a new paradigm in development. At Moontoast all our servers are up on EC2, which includes the two dev servers (one for each project) that I use every day.

I’ve been playing around with Sublime Text (because of the awesome vim bindings) and was thinking about why do I always have to be logged into the remote server to do my work. Normally I connect using SSH, then attach to my tmux session and then fire up vim.

This is okay but it lacks a certain elegance in it’s approach. As a result I end up using things like Dropbox to synchronize my .vimrc file so I get a consistent experience across environments. That’s ok for now, although I’m having some weirdness due to vim 7.2 vs. vim 7.3 issues in one of those environments.

I also need to be able to use git to commit changes, create and merge branches, etc. I need to be connected to the server to make that happen. But that is starting to feel like a limitation to me.

What I’m chafing against is having to deal with multiple environments all the time to get my work done. I want to use ONE editor, tweaked out exactly the way I want to edit code in multiple locations. I want to use ONE tool for handling git work in multiple locations.

Any thoughts on how to achieve what I want? There are occasions when I do need to connect to a server (say to tail a log while testing a Gearman job) but damnit it’s the second decade of the 21st century. This should be a SOLVABLE problem.

So for now I’m trying out SublimeText2 as my main editor and started using Transmit (sorry Expandrive but I am finding Transmit to be a lot more responsive) so the missing piece if for me to be able to commit code changes on a remote server WITHOUT being logged into that remote server.

In a perfect world I am using virtual machines for my dev work, but I haven’t convinced our awesome sysadmin to put in the time necessary to make it happen for me.

Let me know potential solutions for my desired setup.

The Grumpy Programmer’s Guide To Building Testable PHP Applications

2012-01-03T00:00:00-05:00

After the middling success of my first programming book I had been thinking about writing another book. When the end of October rolled around and National Novel Writing Month was about to happen I decided to take the plunge and write something.

It didn’t turn out to be The Great Canadian Novel but I did collect my thoughts on what goes into building PHP applications in such a way that they are easily testable. I called it “The Grumpy Programmer’s Guide To Building Testable PHP Applications”.

When I wrote my first book I did everything in Pages but that was before the rise of e-readers. This time I wanted to offer my book in PDF, epub and mobi formats. So I started up using AsciiDoc since I felt it gave me the most options for converting the text into other formats with a little bit of programming work. I created a Makefile (old school, baby!) that took the text and turned it into a PDF.

I had thought about doing the book directly in Docbook as it provided the best way for conversion purposes, but I found all the XML to be getting in the way of, you know, actually writing the thing. Plain text with a little bit of mark-up seemed to be the best fit for my brain.

As my self-imposed deadline got closer I felt like I was in for a lot of late nights fiddling around with layouts to get things to an acceptable level. Lucky for me an alternative came along.

I follow Reg Braithwaite (better known as @raganwald) and saw that he had a new ebook out called “What I’ve Learned From Failure” which he published with some help from Leanpub. The system looked awesome: write your book in Markdown (which I already use for this blog) and then it would create ebooks in PDF/epub/mobi for you, handling all the distribution and payment collection for you. Holy fuck, this is what I wanted.

So I took the plunge and signed up for an account. I started tweaking my text to use the Markdown-specific formats the system requires and little by little the book is being massaged into it’s new shape.

I’ve gotten some great feedback from my technical reviewers and will be incorporating those changes into the book as soon as I’ve converted it over to the Leanpub structure. I’m giving a talk on the topic the book covers at Codemash 2012 and will launch the book while I’m there, offering those who attend my talk a discount as a way of thanking them for travelling to the shores of Lake Erie in the middle of winter.

So please visit the home of my book online and until the book is published feel free to add your name to the mailing list. As soon as it’s available you will be notified. I’m still tweaking the pricing so feel free to suggest to me how much you’d be willing to pay for such a beast. It will probably come in between 40 and 50 pages in length.

The Year of the Struggle

2011-12-30T00:00:00-05:00

For those who know me outside of the context of my ranty tweets and occasionally-lucid blog posts you will know that I am an avid simulation baseball game player. I’m about to start my 15th season in my current league and it has taught me a great many lessons about myself and the importance of having a set of skills that turn independent bits into a cohesive team.

(If you care the league uses a game that it creates itself using freely-available data from Retrosheet and if you’ve ever come across Strat-O-Matic-Baseball you might have an idea of how game-play works)

I’ve had middling success in that league: some really crappy seasons where I clearly did not know what I was doing along with making the playoffs 3 years out of 4 complete with a trip to our league champsionship series. Many times I’ve had teams where the skills of the players (who are merely statistical representations within the confines of the game rules) simply aren’t a good fit.

If you have lots of power, a good bullpen, but guys who don’t get on-base you will find yourself winning by a lot or losing by a lot because power comes and goes. If you have great pitching but a bad offense (similar to what I have for our upcoming 2012 season) then you will find yourself in many close games but you will probably lose more of them than you win because your offense will be unable to come through when needed.

I’ve also learned that I am a decent field manager (in terms of playing the games and understanding strategies), lousy at evaluating hitters in our draft (my past 4 first-round picks have been hitters who have been injured and/or underachieved), great at evaluating pitchers, and very impatient in the trade market when trying to improve my team.

I went through a terrible season last year due to a combination of bad luck and impatience in making trades to keep a fringe playoff contender in the playoff hunt in 2010, then crashing to lose almost 120 games in 2011. So I struggle to find ways to build a better team, and better identify the combination of skills I need for my team to achieve success.

So what does all this have to do with programming? I feel that my current set of skills are a terrible match for today’s programming environment and it’s making me feel a set of emotions I thought I ditched when I left high school: jealousy, fear, angst and regret.

I’ve always prided myself in thinking that I have spotted trends in programming before everyone else. I first saw the value of frameworks in 2003-2004. About the same time I discovered SimpleTest and thought that it would be a very valuable tool. It’s almost 2012 and I still feel the same way, but I feel like the programming community I belong to still doesn’t give a shit.

In February of this year I jumped on an opportunity to work with some people I respect in the PHP community and joined Moontoast. It had been a while since I had done the startup thing and thought it would be a good change for me. At my previous employer I had to do everything myself for my projects and I looked at Moontoast as an opportunity to work with a team again.

Instead I struggled. Struggled to cope with two applications written using heavily-modified versions of two frameworks. Struggled to understand the business that relied on these applications. Struggled with the idea that the things I felt really, REALLY mattered simply don’t matter to the business as a whole. It’s not a lie when I say it hurt. Hurt a lot to miss deadlines and let down team members. Hurt a lot to struggle to make things like automated testing and simpler product development FUCKING WORK and not achieve anything. Struggled while wondering if I was getting honest feedback from fellow employees who might not want to really say what they feel because they are not an asshole like I often am.

On top of that, I felt like the volume of tweets and blog posts I cranked out achieved nothing that I could hang my hat on and make my ego stop screaming at me in my head. I still care immensely about best practices, automation, and building applications that can be easily tested. All too often I feel like I am the only one who wants it so bad I pound my fists on my desk like a little boy having a fucking temper tantrum.

The feedback loop I want just isn’t there any more. Perhaps my troll-like existence in my home office is wearing me down and making it more difficult for me to actually get along with people. My ego is getting in the way of success in a team setting and I am struggling to find a way to figure out how to change things for the better.

PHP is the tool I’ve been using pretty much non-stop since 1998. It won the web battle. It really did. But things are changing and expectations are changing amongst the next wave of programmers. Elegance in code matters. Ability to support concurrency at the code level matters. Perhaps it shouldn’t but it does. I think of all the investment I’ve made in PHP and I find myself constantly asking the question “did you do the right thing?”

I have found that the PHP community is very supportive of me and my ideas, so I’m probably not as terrible a person as I make myself out to be. But the PHP community as a whole really seems to think that much of the stuff I consider to be important is just a waste of time, getting in the way of getting things done so you can go home after work.

The crowd I follow online is a self-selecting one. Most of them care about the same things (except there are those who disagree with me on the importance of automated testing as a critical part of development) but what about those who aren’t there? How many of them give a shit about clear, concise code? How many of them have actually created an application for their employer that had a complete testing suite (in 14 years I’ve been able to do that ONCE) for it? How many of them have automated their deployments as a result of a merge of code in their version control system?

It’s more likely that I am being selfish and demanding recognition for the things I’ve done and don’t really deserve that recognition. I joke with my wife that I am a “big deal on the Internet” and #internetfamous on Twitter, but am I really? Or am I just loud and repeat myself over and over again. Am I just being tolerated or are there people for whom my thoughts are resonating and they are going out and kicking ass in a way I have been unable to?

Maybe this old saying is true: those who can, do, and those who can’t, teach.

2011 was the year of the struggle for me. I don’t want 2012 to be like that but I am not sure how because I’m scared that I’m becoming useless and irrelevant and that the things I am passionate about have no place in the modern PHP community, the only one I’ve ever really been a part of.

Hope you have a good 2012. I’ll be here, struggling.

Better HTTP Request/Response in PHP

2011-12-12T00:00:00-05:00

I just started up a podcast called Developer Hell with my fellow piss-and-moan-driven-development practitioner Ed Finkler . It basically consists of Ed and I sitting down and just having an open and unedited discussion about whatever programming-related topic we feel like rambling on and on about. Our first episode covered the always-riveting topic of “What We Hate About PHP”.

Now even though it is just us complaining about stuff in PHP that we don’t like, regardless of the true merits of our complaints, it sparked a discussion on Twitter about just what should be done to make handling PUT and DELETE requests better in PHP.

$_POST and $_GET are workhorses of the PHP world, used in any application that accepts web requests from somewhere. Sometimes accessing them is wrapped inside objects or filtered out using the filter() extension of even Inspekt but I think this illustrates the problem.

Laura Thompson made the suggestion that PHP should look to port the concept of the Request object that Python uses over to PHP. As a cryptopythonista I think that is an excellent idea. Elizabeth Smith did ask for some constructive ideas on solving this problem. While my C skills are non-existent, I thought I’d delve a bit into more higher-level thoughts about this.

I think the fact that we have $_POST and $_GET lulls some of us into the false sense that we should have $_PUT and $_DELETE objects, since that would map to the commonly-desired set of HTTP verbs that REST likes to use. But what should be inside those things, or should we be moving towards a more Pythonesque solution where a Request object, as part of core or via a only-really-for-the-brave- PECL extension?

What I’m really after is for an easy way to detect that a PUT or DELETE request has been made. Existing solutions for these are not pretty and ( in my opinion) non-intuitive. With my thoughts that PHP is quickly turning into a tool used for web service API end points that don’t require insane levels of concurrency, the idea that there is no native support for figuring out if you’re been given a PUT or DELETE request is disheartening.

The PECL HTTP extension is a step in the correct direction, but it too has some rough edges and would require a lot of polishing before it could be moved into the core where it really belongs.

Again, what I’m looking for is an easy way for PHP to be able to know what type of request it is dealing with so I can make my PHP applications more aware of what it being asked of them. It seems that pretty much every framework implements their own Request/Response objects, so that should be an indication that this kind of code abstraction is a desired core PHP feature.

Share your thoughts with me in the comments.

War-room Driven Deployment

2011-12-05T00:00:00-05:00

For a long time I have been an advocate of the philosophy that organizations as a whole, not just the engineering team (if there is such a thing where you work) should be striving towards the goal of treating any deployment of your code into production as a non-event. Sadly, there is much work to be done from both ends of the spectrum.

When deployments do go wrong the reaction is fairly predictable. It’s almost like the five stages of grief, sometimes known as the Kuebler-Ross model. You’ve probably heard about them: Denial, Anger, Bargaining, Depression, and Acceptance.

The common thing that happens during those two first stages is that people are chewed out for their role in the deployment failure, and in many cases the anger is legitimate. If you haven’t written code that spectacularly failed in a production environment I will happily call you a liar.

What also then happens is that the non-technical people push for what I call a war-room environment for dealing with deployments. By this I mean that, for lack of a better phrase, orders come down from on-high that whenever a production push is going to be done that everyone needs to stick around and be there “just in case” something goes wrong.

At the news of the new directive, the engineering staff roll their eyes and mumble to each other about how the other side “doesn’t get it” and how “this won’t solve these problems”.

Both sides are right, but only up to a point.

I’m sure that some of my readers are familiar with the concept of perception vs. reality. When you apply this to deployment failures, the perception becomes that the entire team responsible for building the application is to blame and that by making sure that everyone should be there it fixes the problem. In my opinion, the reality is that the failure occurred due to the actions of one or two individuals and making the whole team suffer is counterproductive. I feel this way especially when I am directly responsible for what has happened.

Finding bugs in development are magnitudes cheaper to fix than production ones, and you will feel less pressure to fix them than if they are discovered in production. To paraphrase Darth Vader, you know in your heart that this true. Therefore the engineering side of the equation really needs to push back hard on the idea that ALL HANDS ON DECK WE ARE DEPLOYING is not just making the problem worse before it can get better.

I’ve had my fair share of time with war-room driven deployment. I understand why it is viewed as a solution to the problem of failed deployments. It seems very logical. By having everyone there when it happens you are more likely to be able to quickly solve any problems you find. However, I feel like it is masking the true problem in that everything that you find yourself doing when practicing war-room driven deployment can be done before you even get to that point.

It also becomes an issue of trust. Frequent failed deployments results in the engineering side of things to be labeled as sloppy, uncaring, not understanding the cost of lost opportunities. All of those things may or may not be clear, but I think forcing everyone to stick around “just in case” makes the situation worse.

By using war-room driven deployment you are making all stakeholders nervous about deployments. Everyone will be exhorted to think of everything that could go wrong with the deployment and if something does go wrong, the stress level gets cranked up to 11 and there is frantic activity to try and solve the problem.

It is often hard to explain to non-technical people that they should stay calm and understand that the engineering team knows what went wrong and has a real-world solution in mind to fix whatever problem happened to minimize the chances that it happens again in the future. I don’t have any easy answers on how to solve that problem other than staying calm yourself and offering real-world, practical solutions to the problems.

If you put all the effort into making production pushes a non-event, then you will not need war-room driven deployment. All you need is one person to push a button and then everyone can go home.

Scope is not a mouthwash

2011-11-25T00:00:00-05:00

One of the worst-kept secrets now is that I have started working on a book about some practical techniques for building testable applications. I’d say I’m about half done, and the book will be ready for sale when I give my talk at CodeMash 2012 on the same topic.

Why do I care about all this stuff that causes friction for programmers? Because there is so much I want to learn and fragile, untestable applications get in the way of me becoming the programmer I want to be. Naturally I am also working on an application that will serve as a companion to the book, a hand-crafted web application that will embody the techniques I describe in the book. It is also very humbling to “eat your own dogfood”, meaning build you applications using the practices you are espousing.

So, I set out to show people in my chapter of the book about decoupling objects via the magic of dependency injection how to use this awesome thing known as a dependency injection container. Although they are more suited for very large complex applications like web application frameworks but I thought “I could see someone like a younger version of me wanting to mess around with something like this even though it might be totally inappropriate for the application. Let’s do it!”

For this I am using Pimple, an incredibly small but effective dependency injection container. Easy to use, simple and effective documentation, just what I was looking for. I also noticed that Pimple supported the use of closures (or anonymous functions) as a way of storing a dependency.

Then things got stupid.

I altered the bootstrap file for my application (it’s using the no-framework framework approach) and set up an instance of Pimple and wrote this cool-looking code that stores a mapper for one of my models in it:

$container = new \Pimple();
$container['db_connection'] = function ($c) {
    return new PDO(
        'pgsql:host=localhost;dbname=ibl_stats', 
        'login',
        'pass'
    );
};
$container['franchise_mapper'] = function ($c) {
    return new IBL\FranchiseMapper($c['db_connection']);
};

What you don’t see in this snippet is that before this happens, I am using a PSR-0 compliant autoloader. Go Google for it, it’s hard to explain in a few short words and I am also using namespaces to show people just how easy they are to use. I love autoloading. Spares me dealing with ‘require_once’ and ‘include_once’ and also worrying about if I’ve required or included a file somewhere else.

I reload my test page and…what the hell? “Cannot find class IBL\FranchiseMapper”

That…that…that cannot be! It’s RIGHT THERE! I have unit tests that run just fine and the autoloader grabs it there too!

So I start asking for help on Twitter. “It must be Pimple causing the problem.” Pimple’s creator smacks me down and I look at the code and say “of course it can’t be Pimple. Keep looking!”

Next I figure there is something specific to closures that is causing the problem. I reach out for more help, trying things willy-nilly to try and figure it out. I contemplate using Xdebug to trace what is going on in the autoloader. Not very easy to debug autoloaders. Finally I see something (of course I cannot find the link now) that talks about closures and scope. That is when I realize how fucking stupid I have been.

So, let’s talk about scope for a second. Besides being the brand name for a mouthwash, scope is a topic that some programmers get tripped up on. Despite my 14 years of PHP experience I often feel like it is 1 year repeated 14 times. Scope deals with variables (and in this case namespaces) and where they can be accessed from.

You have global scope, meaning that the variable/object/function can be accessed from anywhere. Then you have the scope of something inside a standalone function. Unless (mind you, this is PHP and I cannot speak for other languages) you mark it as global, it can only be accessed from inside that function. Now, you can declare a variable/object/whatever as being accessible inside that object via the use of (for example) $this->foo but that variable, and you can also slap public/protected/private to decide who can see it when you interact with that object.

To roll back to me, my problem was not one of variable scope. It was one of namespace scope. With the introduction of namespaces to PHP you now have the concept of the “global namespace”, which is where all the existing PHP functions and objects live. If you want to be pedantic (and Wotan knows I sometimes enjoy pedant status) you should be prefixing all your calls to things like mysql_query() with a backslash. Try it out and see if I’m telling the truth.

So I’m scratching my head looking at this code. “What the hell, dude. I’ve got nothing to lose by adding a backslash in front of IBL.” Of course it worked. But why did it work?

(Grumpy developer’s note: I was asked on Twitter to expand a bit on the reasons behind why things work this way. I am not a PHP internals guy so everything I say here is just an educated guess)

With the introduction of namespaces, PHP has to operate under some assumptions. If you are going to remain backwards-compatible with a lot of code and support namespaces then you have to make some rules too. As far as I can tell, when PHP encounters an object or function, it assumes that it is in the global namespace first. This is how you can do things like $query = mysql_query() and have it not complain. No prepended backslash, it assumes it is in the global namespace.

Now, let’s say I am trying to do this:

$mapper = new IBL\FranchiseMapper($container['db\_connection']);

There is no pre-pended backslash but there is a backslash after the IBL, therefore PHP can assume that there are namespaces involved. Then (I think) it goes and looks to see if there is currently an ‘IBL’ namespace that it has been asked to use. Because of the autoloader it is aware of that namespace, so then it happily loads it.

So why does it behave differently inside closures? I am not 100% sure, but if I had to make an educated guess I would say that when trying to resolve namespaces inside a closure, the interpretor doesn’t assume that it is already inside the global namespace, that it is in a namespace of it’s own. Hence the need to explicitly declare that you are starting with the global namespace (prepending the backslash) and then working your way down.

If any PHP interals contributors read this blog post I would appreciate a follow-up comment to confirm and/or explain how this works.

So, it looks like closures are stricter and was reminding me, via that error message, that I need to be aware of the current scope of the namespace (is that even the right way to put it?) and prepend that backslash so my autoloader (which is hooked into spl_autoload_register) can, you know, find and create that object for me. So here’s how it looks now

$container = new \Pimple();
$container['db_connection'] = function ($c) {
    return new \PDO(
        'pgsql:host=localhost;dbname=ibl_stats', 
        'login',
        'pass' 
    );
};
$container['franchise_mapper'] = function ($c) {
    return new \IBL\FranchiseMapper($c['db_connection']);
};

$mapper = $container['franchise_mapper'];

Yay! I can create a mapper via the dependency injection container.

I know that a lot of beginning to intermediate javascript programmers *cough*me*cough* have to pay attention to variable/function/object scope issues as well. Also, don’t be like me when there are no tests for your code and start randomly changing things in hopes that it works.

It usually doesn’t.

Building a Gimmebar Sidebar

2011-11-07T00:00:00-05:00

One of the interesting side effects of getting involved in your programming language community of choice is that you often get access to cool projects being done by your community buddies before the rest of the riffraff (aka the general public). One such project is Gimme Bar, a “capture anything you want and read later” web site. Created by the fine folks at Fictive Kin, Sean Coates and Ed Finkler were kind enough to not only give me access to their application (My user type apparantly is “The Betanaut”) but also an early peek at their API.

Now, it’s changed a bit since their earliest work but now they are starting to promote not only the site but showing how easy it is to use their API to play around with not only your content but other’s public-facing content too. I fooled around with the API and tried to write some Python stuff but lots patience when trying to work with OAuth from the command line. Hey, it’s not their fault that I couldn’t be bothered to slog through it all.

Funkatron was kind enough to put together a blog post about building a Tumblelog with Gimmer Bar and PHP so I told him I would write up what I did on my own blog.

One of my intents was to use the Gimme Bar API to pull in a list of my latest Gimmies for the sidebar on this blog. Ed had posted a very simple example in Javascript (I cannot remember where it is) so I took it and ran with it to modify it to work with the blogging software used here.

I went and asked some questions in the Freenode IRC channel for Octopress and found out that they recommended the use of JS libraries that are compatible with Ender.js. Okay, so that was no problem. So what to use to make the actual AJAX request call to the API? Reqwest was mentioned by some as the best fit for using with Ender. Reqwest is an easy-to-use JS AJAX library that the #octopress guys suggest I use since jQuery doesn’t currently play nice with all the other JS libraries that Octopress is using.

Finally, I used Underscore to help with templating.

Even though I’m not much of a JS whiz, I was able to take Ed’s example and very quickly modify it to work exactly the way I wanted it to. Below is the code I used to putt in my 10 latest Gimmes and insert them into the DOM exactly where I wanted them.

<section>
  <h1>My Latest Gimmies</h1>
  <ul id="gb_assets"></ul>
  <script type="text/javascript">
  reqwest({
      url: 'https://gimmebar.com/api/v0/public/assets/grumpycanuck.js?jsonp_callback=?',
      type: 'jsonp',
      success: function(data){
          var list_tpl = "<% _.each(records, function(record) { %> <li><a href='https://gimmebar.com/view/<%=record.id%>'><%=record.title%></li></a> <% }); %>";
          var list_html = _.template(list_tpl, data);
          $('#gb_assets').html(list_html);
      }
  });
  </script>                                                                                                                                                     
</section>

For those who aren’t JS-savvy, I’ll explain what goes on here:

Execute an AJAX call that…
calls the Gimme Bar public API to get the latest 10 public posts…
and on success generate an unordered list of posts…
linking to the item on Gimme Bar and show the title…
and insert it into the DOM where I asked it to

Easy peasy, lemon squeezy. Ender.js + Reqwest + Underscore + 15 lines of markup and JS gives me the latest 10 posts. Hope this inspires you to play with the Gimme Bar API yourself.

Infrastructure Debt

2011-11-03T00:00:00-04:00

You might have come across the term “technical debt”, used to describe the small mistakes that are made in your code base as the application grows and requirements change. You end up with a big tangled mess if you are not very careful about how you make those changes. If you’ve ever worked on an application where you were afraid to make changes for fear of breaking something, then you have run into technical debt.

It’s very easy to build up technical debt: you put a quick hack in because you feel like you are under pressure to get a task accomplished. It takes courage to push back and say “no, damnit, I need time to fix this particular problem correctly!”. I understand, and it’s okay. We can hug it out if it will make you feel better.

Technical debt can be dealt with via ruthless refactoring and wrapping your application in tests that poke and prod at the edge cases that your application deals with. But there is another type of debt you end up dealing with. One that is even more difficult to deal with and change. I call it infrastructure debt.

Infrastructure debt is debt that you build up because you have not been paying attention to the process of creating the environments your application will exist in and have not been paying attention to the process of how your code gets from development into production. In my opinion, infrastructure debt is much more difficult to “pay off” than technical debt. Why? It is often very difficult for people to understand that it even exists.

Before I go any further, if you are not using a version control system to keep track of changes to your code, then please stop reading and go and install one. Having versions of your files with extensions like .1 and .old is the worst kind of infrastructure debt that you could possibly have. It’s the 2nd decade of the 21st century. Version control is an almost 40-years-old, well-understood concept. Nothing I suggest will work if you insist on acting like version control is of no use to you. Passing around diff patches is not version control, my friends. It’s denial. Don’t tell me just because people used to do it in the past that it’s okay now given all the available options. People used to think that the earth was flat once too.

Let’s start with an easy infrastructure debt to pay off: inconsistency in development environments. These days it is pretty easy for someone to create a development environment for their language of choice along with associated components like databases. The problem? That it’s pretty easy for someone to create the development environment of their choice.

You end up with your developers all having environments that are slightly different due to choice of source for packages or preferences on versions of components. I have PHP 5.3.8 via a custom Homebrew recipe while one of my co-workers has the stock version of PHP that comes with OS-X Snow Leopard. He has a version of MySQL that is two versions ahead of mine. See how easy it is to get out of sync?

You have two options. The first one is the draconian one, where you force everyone to use the same operating system and the same tools to minimize the chances of conflicts between development environments. Given how that leads to developer unhappiness, I prefer a more sane approach: use tools like VMWare or VirtualBox to let your developers do their work on identical virtual machines while giving them the freedom to use the development and debugging tools of their own choosing.

Also, by using a virtual machine you can encourage developers to push the envelope and test destructive things like library upgrades or database changes. Messed it up? Just fire up a new virtual machine and try again. When you combine it with good version control practices you should be able to recover from your mistakes even faster.

Just make sure to take the time to create virtual machines that are as close to an exact match as your production systems as you can. Honestly, the only difference between your development environment and your production environment should be the data your application is manipulating. It’s not really that hard to even create a cluster of virtual machines on your own computer to simulate an architecture that has application servers, a database server and a memcached server.

It also helps you to get new developers up to speed quickly. “Oh, working on Project Alpha? I’ll email you the link on the wiki where you can download the virtual machine for that project.”

Check out Vagrant for a great option for developers wishing to do their work in a virtual machine.

So, having injected some structure into your development environment there is really one last frontier where infrastructure debt builds up like the credit card bills of a shopaholic: deployment processes.

I really think there is one rule: if you cannot do your deployments with one command then you are DOING IT WRONG. If you can type the commands you are doing into a shell then you can script them so you don’t have to type them in again. If you can script deployment, then you can automate deployment. If you can automate deployment then you now have a consistent and repeatable process that will behave the same way every single time you deploy.

At Moontoast we have automated deployment in place. When we merge code into our staging and production branches, a version control hook triggers our deployment tool. It checks out our code into it’s proper location on the production server. It runs database migrations that need to be done. It starts and restarts various services that are used by the application. In short, it does everything that I see far too many of my colleagues doing manually. Stop that. Clean up your infrastructure debt by making the process of deploying your code a non-event.

If you want to automate your deployment process, examine tools like Capistrano or Phing or you could even use a tool like Jenkins and use it’s concept of Builds to automate the process of taking your latest code changes into production.

Note that I said the PROCESS, because technical debt might mean that your application is fragile enough (doesn’t have enough tests and/or test coverage) that the changes you made might cause it to behave improperly or even crash the system altogether. Not a good feeling, so by investing the time into automating your deployment process you can eliminate at least one headache.

I think the main point I am trying to get across with respect to infrastructure debt is that you really need to examine your processes beyond just coding. If you find yourself constantly fighting the “works on my computer” battle, you likely have serious infrastructure debt. If you find yourself constantly referring to a checklist whenever you do a deployment, you likely have serious infrastructure debt.

Whenever I bring this topic up, I hear many of the same excuses that I hear when I advocate a commitment to writing tests and automating the execution of those tests. What is it that people have against this sort of thing?

I have found that people terribly underestimate the amount of manual work they do to accomplish a task. For many, it is a fear of losing control that leads them to conclude that automated deployment cannot be trusted, or that developers shouldn’t have the freedom to pick the tools that they use to build their applications.

Technical debt is real, make no mistake. But so is infrastructure debt. Just like the cost of finding bugs in production is more than finding them in development, the cost of fixing infrastructure debt gets more and more expensive. Just think of the large number of problems solved by eliminating inconsistencies between development environments. Add to that the ability to eliminate user error during manual deployments of your application. Why would you not want to have this?

The decision to start paying down your infrastructure debt is not an easy one, because it will often require many changes to be made by many people. Most programmers are creatures of habit and dislike change, even when it is obviously better. Don’t be scared of change, be scared of the debt growing in your code base and in your infrastructure. It won’t go away and there is no government bailout on the way to fix it.

As always, I welcome your thoughts via email, Twitter and comments on this blog. What sort of infrastructure problems have you come across and how did you fix them?

The Grumpy Programmer’s Guide To Remote Working, Part 2

2011-10-24T00:00:00-04:00

As mentioned in the previous post about remote working, you do have to work a lot harder when you’re not in the same office as your co-workers in order to simply get things done. If you took my advice you’ve got yourself a sweet setup at home, with a comfy chair and a nice desk. But that is only the first battle in the on-going war that is successful telecommuting. In this post I want to talk about some tools that I use that I feel help me Get Shit Done when working from home.

First, since you don’t have the instant feedback loop of a fellow cubicle dweller or even office-sharing partner, you need tools that can act as a second set of eyes for you before you open what is often an expensive communication channel to your co-workers. Not expensive in terms of money (unless you have to call people long distance and don’t either use VOIP at home or Skype) but expensive in terms of time. Making that initial “communication handshake” is expensive in time and commitment.

So for me, the first set of eyes I use are Firebug or Chrome’s web developer tools, depending on which browser I’m using. Those let me peek at the HTML being generated and debug any Ajax calls your application is making. It used to be my primary tool until my boss asked me why I wasn’t using Charles, an awesome debugging proxy application for OS-X, Windows and Linux.

Charles acts like what it says: it’s a web debugging proxy. It sits between your browser and your next work connection, monitoring and showing you what connections you are making. Why is this awesome you might ask? I can record an entire series of clicks on a web page and play them back later for testing purposes. I like Charles for a few reasons. This is helpful because one of our main applications at Moontoast is a flash-based store and Charles will show me the request that the flash SWF is making and I can see the response that is coming back. I am not a flash developer, but I do need to make sure that code I write that is supposed to talk back to the SWF (via AMF) is returning correct values. I like to record Ajax calls and repeat them over and over until I am happy with the response coming back. Please, please, PLEASE check out Charles. It is worth every penny.

So now that I have someone else helping me debug my Ajax and AMF calls, what other tools do I find useful. Well, if you don’t have your editor already tricked out to reduce the amount of work you have to do outside of your editor to get things done then I advise you to actually learn your editor inside and out. As a vim user I spend a lot of time evaluating plugins that can help me out. I use Syntastic to run my code through a code linter every time I save it. I use Fugitive to work on my Git commits and merges. All in all, it’s important that you be as productive as possible using the tool you write your code with day in and day out.

Of course, it goes without saying that you should be probably be writing your own tests for your code that you can automate. Don’t worry, this is not going to turn into another sales job for automated testing. PHPUnit, PHPSpec, Selenium, please use something that acts as an automated tester for you. I mean, it seems to work for me so maybe it can work for you.

So that’s really it in terms of tools that I have found that I used more and more as I telecommuted more and more. My criteria for evaluating any tools is this: is it easy to use, can it automate something I don’t want to do manually, and does it easily integrate into my existing workflow. Firebug and Charles fit that criteria. Plugins for my editor fit that criteria. CLI test runners fit that criteria.

In other words, any tool that can make you more productive in an office setting is even more important when working remotely with a team. Anything that you can do to reduce the testing and integration step benefits you and your team, and helps to convince management that their decision to allow you to continue to work remotely is a good one.

I’m interested in hearing other people’s experiences telecommuting and tools that they have found to be helpful. Comment in this post, send me some email (chartjes AT this domain) or hit me up on Twitter.

The Grumpy Programmer’s Guide To Remote Working, Part 1

2011-10-10T00:00:00-04:00

Well, I couldn’t very well call this post MOAR OFFICE PORN as MWOP suggested.

I have talked before on this blog about stuff related to what goes into being a successful remote worker. I’ve been telecommuting for 4-1/2 years now and it’s the best thing that ever happened to my career. It allowed me to grow into the programmer I am today by teaching me the importance of being disciplined, learning how to communicate effectively in a remote environment, and how to not take shit from anyone just because I wasn’t in the same office as them. This will be the first article in a series about what I think are some successful strategies for being a good remote worker. Today’s lesson is about creating an effective workspace.

When I used to do my killer commute to work every day (1-1/2 hours in, 2 hours out) I had the misfortune of working for employers who did not believe that creative people (yes, that means you mister programmer) needed quiet and private (or at least semi-private) working space in order to get things done. Instead I usually ended up either in the cubicle farm or in that phrase that inspires the most fear for me: the open concept office. Except for the managers of course. They always had offices with doors that could be closed.

So now that you work at home, guess what? You can finally create the environment that will allow you to easily get into that state of focussed work that some refer to as The Zone. No working at coffee shops for this grumpy guy. Small tables, crappy chairs, and people staring at you while you’re trying to work. I’m not doing top-secret work for the government, I’m just trying to build some rock-solid web applications. Eyes on your own crappy life, please.

For me, I started off working in the unfinished basement of my house on a make-shift desk that consisted of an old desktop from a broken desk (well, not really broken but I couldn’t find a bunch of lock nuts to put it back together after I moved) on top of two filing cabinets. I bought a decent chair (more on that later) and was okay to work. Until it was winter in my part of Canada. The basement was pretty much unheated so by 2 PM my hands were numb.

So at my wife’s insistence I moved into the dining room. Almost the same problem as working from the coffee shop, except I had a better chair and no people starting at me. Just the cats, but they never debugged my code properly anyway. Plus she complained about the reproducing paper up the table: bills I paid that hadn’t moved down to the filing cabinet in the basement, printouts of instructions and scoresheets for my simulation baseball league. The time came for us to consider trying to use the space in our basement for something productive. So we sank $25K into renovating the basement: laundry room, wreck room for the kids (get the joke?), two storage areas, and a home office for me.

So, what is the configuration that allows me to get into the proper frame of mind to program? Well, the first one is a door that I can close. Call me a princess but sometimes I need to block out the noise from the rest of the house in order to get things done. Especially when the kids are home from school during the last two hours of my normal working day. They argue and make lots of noise when watching TV, so the door gets closed. If I get especially grouchy I lock the door too and make them call me via our phone’s intercom system.

I’ve never liked the idea of putting on headphones to block out the outside world. I find long-term headphone wearing to be hot and uncomfortable. Yes, I used to wear earbuds during my long commute but I wasn’t trying to solve any problem other than mentally cooling down from a long day at work. So an office with a door I can close means I can crank up the tunes if required but also give myself a nice quiet zone when I really need it.

Next essential item? Another piece of furniture other than a chair that you can sit on. Until recently all I had was my awesome “fat man chair” (big comfy leather chair that can handle up to 350 pounds of weight). Sometimes I wanted to move away from my desk to work out a problem with paper and pen, or even just to lay down for a minute while talking in a meeting. Hey, don’t blame me because I work from home. Make it happen for yourself before picking on me!

Anyway, we recently pulled up the grotty carpet in our house and had laminate flooring installed. I got the people who did the floor to help me move on part of our three-piece sectional down into my office. Now I have a fainting couch to sit on. Most useful piece of furniture in my office. Do not underestimate the importance of having something else to sit on.

Now, more about the desk. I still have my terrible desk, but with my awesome adjustable chair I have comfort while sitting my large frame down. I’m 6’5”, 285 lbs of grumpy programmer. The world is not made for people my size. It was worth the money ($300 at the local office supply place, they had only one in stock) for my chair, so I always recommend people try out several chairs and get the one that the think they could sit in for the longest. Yes, I know that standing desks are all the rage right now, but I think you are trading one set of problems for another. Maybe standings desks work for you.

My next home office purchase will be an adjustable desk from Ikea to get the perfect height for working, as I find my current setup to be just a tad too low. Again, having the ability to adjust the height of your desk to an optimal height based on your chair is critical too. I’d love to fork out the $1500 to get one of those awesome adjustable Geek Desks but that will have to wait for me to set aside enough money for purchase and delivery.

So, now my actual working setup. I have a 15” MBP generously purchased for me by my employer. Connected to that I have an Apple keyboard and mouse, and view things via a really nice 23” LCD monitor. What I really want is one of those Apple Studio Displays, but again that will have to wait until I can set aside enough money for such a purchase. Right now, I’d say that is my optimal configuration. I very rarely refer to the laptop screen, and like the amount of real estate I have now. Perhaps I could move to a two-monitor setup but I need more physical space on my desk to make it happen. Hrm, I will have to measure the new desk to see what will fit…

Anyway, in part two I will talk about how my programming style and support tools have changed since I’ve been working remotely.

‘Blogging framework for hackers’ is the right label

2011-09-14T00:00:00-04:00

I don’t regret moving from a Wordpress-powered blog to one that is created using a static site generator. But I can definitely say that the journey to a static site (initially Jekyll and now using Octopress) is not a path I would recommend to anyone who is not comfortable debugging issues from the command line or unable to write conversion tools from scratch depending on the situation.

While I do like how easy Octopress makes it to manage my blog I find the whole setup to be, like so many other tools and web applications I’ve come across, to be brittle. Too many moving parts. Too much reliance on things being perfect and if they’re off by even the tiniest bit everything falls apart. I am by no means innocent of creating brittle applications but I try and go out of my way these days to create things that are as robust as possible in the constraints of the existing system.

For me it was RVM, the “Ruby Version Manager”. I like to call it “what virtualenv would look like if Ruby guys got their hands on it”. On the surface it sounds awesome: it sets up an environment containing the Ruby binary you want and the Ruby gems you want, isolating this application from whatever the defaults were on the system. Just like virtualenv. Sounds incredibly useful. But when it breaks, it breaks and there is NO way to tell what the hell has happened.

Somehow through deleting the .rvmrc file (the RVM config file) I managed to break my install of Octopress. This happened at the same time I upgraded the Octopress components by pulling from the master. I had no fucking idea what had happened, all I knew was that it wouldn’t generate any more blog posts for me and was reporting errors about going too many stack levels deep, mal-formed YAML, and Unicode errors. Funny how none of that seemed to have mattered to Octopress before I updated.

In the end, I had to completely remove the install and start it up again from scratch. This is not what I call a successful error-resolution process. In my mind I shouldn’t have had to do this at all, but through the removal and re-addition of one file everything was completely messed up. One upside was that in the process of getting things back to “normal” I created two scripts that I hope can be of use to people who ran into the same problems I did.

I used the stock migration script for Jekyll for Wordpress users who had their posts in a database. The script reads all your posts in and then spits out new posts in the Markdown format that Jekyll expects. The new version of Octopress likes Ruby 1.9.2 which, as far as I can tell, is much less forgiving of mistakes in YAML and Unicode characters. I don’t use Ruby so I am not as familiar with these issues. It sure sucked when I was searching for solutions to my problems.

So, I had 427 posts that had been converted and there was no way I was going to search through each one by hand to resolve the issues. So I needed two scripts: one to parse all the posts to check for invalid YAML and one to force re-encoding of the content of each post into UTF-8 (because that’s how Jekyll and I roll). Being a polygot programming wannabe, the first script for YAML validation was in Python:

#!/usr/bin/env python
import getopt
import os
import re
import sys
import yaml

from yaml import load, dump

try:
    from yaml import CLoader as Loader, CDumper as Dumper
except ImportError:
    from yaml import Loader, Dumper

class Usage(Exception):
    def __init__(self, msg):
        self.msg = msg

def main(argv=None):
    if argv is None:
        argv = sys.argv

    try:
        try:
            print "Checking markdown files in the current directory…"

            #root = "/Users/chartjes/Documents/atthekeyboard/source/_posts"
            for r,d,f in os.walk('.'):
                for file in f:
                    fp = open(file, 'r')
                    contents = fp.read()
                    marker = "—"
                    a = contents.find(marker)
                    b = contents.find(marker, a + len(marker))
                    try:
                        yaml_to_parse = contents[a:b]
                        yaml.load(yaml_to_parse)
                    except:
                        print "%s is invalid YAML" % file
        except getopt.error.msg:
            raise Usage(msg)
    except Usage, err:
        print >>sys.stderr, err.msg
        return 2

if __name__ == "__main__":
    sys.exit(main())

It took me a while to find some code to extract just the YAML from the posts, and I am not 100% happy with the result but it did the job and identified a handful of posts that made it through the migration and didn’t properly wrap post titles with colons (which YAML thinks is bad) in quotes. With the bad YAML fixed I then had to fix any bad characters in the posts. For that I dragged out my copy of “Programming Ruby” (proudly proclaiming on the cover that it was a Second Edition covering Ruby 1.8) and stole some code from a conversion script someone wrote for Wordpress blogs if you dumped the contents as XML.

#!/usr/bin/env ruby
require 'fileutils'

markdown_files = Dir["*.yml"]

markdown_files.each do |m|
    puts "Converting #{m}"
    f = File.new(m)
    post_contents = f.read
    content = post_contents.encode("UTF-8")
    new_file = File.new(m + "-new", "w+")
    new_file.write(content)
    FileUtils.mv(m + "-new", m)
end

Like I said, I’m happy I switched but the brittleness of the whole setup is something I find annoying. I realize that Octopress probably started it’s life as one person’s set of tools they were using to manage a Jekyll blog. The process of turning that extraction into a general-purpose tool is a difficult one and I want to say thank you to Brandon Mathis for providing quick and useful information in the #octopress channel on Freenode while I tried to figure out how to fix the problems I was facing.

(Edit: On Twitter Octopress’ creator felt I was saying that everyone must endure what I went through, which isn’t the case. My point was that the whole setup seems fragile to me and that one little mistake shouldn’t make everything blow up like it did. Now that I know what to look out for, this shouldn’t happen again. Cheer up Brandon!)

Two Sides to API Building

2011-09-13T00:00:00-04:00

I’ve been spending a lot of time lately building APIs for projects at work. One is a SOAP API and the other an RPC API (no, not full REST but it is aware of GET vs. POST for certain RPC calls). One of them was built to support an internal project (server-side component for a JS-powered online store) while the other is to help a 3rd party extract data from our own services for integration into their own. Building both of these led me to the epiphany (maybe obvious to you but not to me) that there are really two sides to an API. They are not necessarily the sides you are expecting.

One reason to build an API is to expose data that is in your system. If you’re really forward-thinking, you wrote the API first and then your applications follow the form of a Service-Oriented Architecture. That is the sweet spot for a lot of applications and I cannot lie when I say I want to move some of the infrastructure towards that, but I am drifting off topic. This is the most common form of an API – you select the data you wish to expose to other people. The advanced version of this, which is essential to building an SOA, is to then allow users to do actions via the API that create, update, or delete data in your system.

The second reason to build an API is to allow for integration with other services. Sounds a lot like the first reason for making an API, right? The big difference is that instead of the people building the API (meaning you and/or me) are not in control of what gets exposed. The party doing the integration will be telling you what data they will need exposed. It’s a subtle difference but one that I think is important to understand. Under the first scenario, you are in control and (to use a bad cliche) hold the keys to the kingdom, deciding what data gets exposed.

In the second scenario you are not in control and are relying on 3rd parties to tell you what data they need. Depending on the actual relationship between you and the 3rd party, you might have a chance to move this scenario back to the first one. Keep this in mind next time you are building an API and adjust your expectations accordingly.

Renovating Sucks

2011-09-07T00:00:00-04:00

My apologies for the big mess here. As some of you who follow me on twitter might be away, I accidentally blew away all the source for this blog while trying to fix a syntax highlighting problem with some older posts. I thought I had a back-up of the source on my external drive but it turns out that I was not telling it to back up my home directory, for whatever reason. How awkward.

Luckily I sill had the dump of data from my old WordPress blog in a database so since I was still going to use Jekyll, I used one of the provided converters and got back to work. I was still missing 13 posts though. Those were fairly easy to recover thanks to the awesomeness that is Pandoc. What is Pandoc? It bills itself as a “universal file converter” and it does a pretty good job. I used it to convert those old posts in HTML format into Markdown. Now that I had all my posts (stretching back to October of 2005) it was time to renovate.

I’m not using just Jekyll. I decided to use Octopress, which is a suite of tools written in Ruby that sit on top of Jekyll. I edit my posts in Markdown as usual, and then it’s “rake generate && rake deploy” to make new posts. Of course, it wasn’t quite that simple but that is the basic version of it.

(Edit: I previously had said I was going to drop comments from the blog but because I kept the same URL structure, the Disqus plugin for Octopress was able to easily find them. Another small victory for Octopress)

Finally I apologize to people who are reading my feed if I have just blasted them with 800+ posts due to me setting the blog up again.

Anyway, I’m back and expect the info to start flowing again.

A Day in the Life At Moontoast

2011-08-09T00:00:00-04:00

I had jokingly asked on Twitter for topics for blog posts for this week, planning on blogging something last night too, but I ran out of time before I headed out to play softball. Let’s here it for dedication to task! Matthew Turland had suggested I talk about the day in the life of your grumpy programmer at Moontast. Not missing an opportunity to be a smartass, I asked Matthew if he really wanted to know before my boss piped up that it involved lots of “blood, sweat and swear”. Thanks to Terry Chay for that awesome quote.

Given that Moontoast is currently hiring some PHP developers (ahem, Server Side Engineers) to either work in our offices in Nashville, Boston or remotely in North America, perhaps this is a very timely post. Don’t worry, I’m not going to reveal any trade secrets or ask you to peer review the code I’ve written. I’ll save that punishment for those folks we actually hire. Instead, I want to focus a bit on work flow and our infrastructure.

With most of the team in Nashville, that means they are on Central time. Since I’m in the Eastern timezone, that gives me the first hour of the day to focus on planning my day. Yes, I’m sure you shocked that I actually make notes and try to plan ahead of time the things I want to work on that day. Lately I’ve been working on a SOAP-powered API. Don’t worry, I can hear you snickering while you read that. It’s meant to be compatible with Magento’s for strategic business purposes.

As part of my tasks I also identify who on the team I need to speak to if they are a potential block. Being a remote worker means that communication is incredibly important, and quick turnaround time is essential to avoid me sitting in my basement wasting my time. With my list of potential targets for my anger secured, I connect to the dev environment I’ll be using and start coding away.

At Moontoast all our stuff is up in the cloud. EC2, S3, ELB, MySQL on RDS and MongoHQ. My two dev environments are up there too, so I spend pretty much all day using Vim. I did try using Expandrive to mount remote directories and then edit things via MacVim (as I do prefer it) but I found that it was impossible to use my most essential Vim plugin to find the files I needed to work on. So I instead duplicated my Vim setup on my two dev instances and away I go.

I end up learning a lot of devops stuff while I build the applications. Specifically learning how to make all the different pieces talk nicely to each other, and then creating configurations that allow me to pull in data for testing purposes from pretty much any environment we are using. If you have not learned the power of tunnelling database connections over SSH you have missed out on some powerful tools.

We also use Git for our version control. It took me a few months to get comfortable using it (still run into the odd problem but they are fixable with a little research or swearing until someone helps) and I think I’m okay with it now. I don’t do anything fancy like cherry-picking commits to build artisanal pushes that are a thing of beauty. We keep things simple and stick to this workflow, with the added change of using a stage branch.

Oh, and if that wasn’t confusing enough, the two different products we have use two different branching structures. Fixing that AND getting rid of the stage branch are high on the list of system integration tasks I want to do. That and fixing our current deployment system.

Right now we use something that is actually quite awesome called Whiskey Disk, a tool that (almost) delivers on the promise of “embarrassingly fast deployments”. I do like it but I feel that we’re moving to a system that needs to be more tailored to my goals of continuous delivery of code. We have a cron job setup that monitors for changes to several branches, and then triggers a deployment if it notices a change. It is cool, but I would prefer some more control. Sometimes I feel like waiting 10 minutes for changes to show up is unacceptable. Some of my objections are also of a philosophical nature. They are doing deploys using in-place updates of code instead of copying code into a directory, swapping symlinks, and then flushing the opcode cache. A minor detail to some, but if you start relying on pieces like this you really need to do it right.

Magical deployments is cool, but I feel like we are moving towards a scenario where we need more control of what gets deployed where. One thing I do like is the concept of separate repositories for the application and the config files for the application, allowing for pushing just config changes out instead of the whole app. I find it handy, but again, I have a desire for more control on what goes where. My perfect tool is a cross between WePloy and the recently-released Deployinator. Etsy’s tool looks the closest to what I like, but we shall see. Something in PHP is more comfortable as Deployinator is a Sinatra app and my Ruby is WAY out of date.

So, when I work I am either doing coding in a ‘hotfix’ branch to fix a bug or a ‘feature’ branch when creating new things. For example, my SOAP API work is in the ‘feature-api’ branch. I commit my stuff often, usually after I’ve tested stuff out (more on that later) and I also push to my branch quite often as well. Mostly because it provides proof to the team that I am working on stuff as all pushes are tracked and a bot emails the dev team AND posts a message in our IRC channel that code has been pushed.

You notice I talked about tests. The amount of tests we have on what is a fairly complex set of products is appallingly low. I do feel shame, and while building the API I’ve been using PHPSpec as the testing tool. While I’ve been a big believer in unit testing in the past, it seems to me that a Behaviour Driven Development approach is likely to achieve better results overall. BDD lets me talk to our product managers and very easily convert their criteria for a feature successfully working into tests. I’m in the early days of PHPSpec and BDD so I’m sure I’ll have more thoughts on this in the future. Either way, the time to commit to tests is now. I’ve been through the death march of watching a team trying to write tests after the fact for a large application. Not pretty.

To be perfectly honest, having good tests is like having six-pack abs – everybody wants them but nobody wants to do the work to get them. Tests are the first step down the continuous delivery path, because if you write tests then you can have the running of those tests automated via continuous integration. I’ve spoken about CI before, so I won’t waste any more time on it.

So, I have code I’m writing in Git branches and are writing tests for all new features I’m doing. Next step is to figure out how to complement the work I’m doing with PHPSpec with the use of Cucumber to let me test the rest of the site. There are some challenges due to the use of Facebook Connect for authentication, and heavy use of Ajax in the applications. They are great from a user perspective, but a pain in the fucking ass to try and write automated tests for. While Cucumber is aimed more at Ruby folks, Cucumber is also able to test non-Ruby apps. Again, it will be an uphill struggle but one that I think will pay off in allowing increased delivery speed.

Now, we do have meetings every single day. We are using some scrum-like system, but it basically boils down to talking to each other for 5 to 10 minutes every day to go over what we’re working on to see if anyone needs help or to tell them if you’re done early and can tackle another problem. Yes, sometimes I get stuff done ahead of schedule and then declare my desire to spend time on a another task, usually something related to infrastructure work. Our meetings are usually just before my lunch time, and often they result in my slightly modifying what I’m working on in the day. I have some lunch and then get back to work.

So there you have it: a day in my life at Moontoast. There are a lot of cool challenges to make the whole infrastructure better, but they are the type of challenges that I think will not kill you but really make you stronger. Wish me luck!

Brain Dump for June 2011

2011-06-29T00:00:00-04:00

Wow, it’s been almost a month since I blogged anything. So I was inspired to ask my twitter peeps to suggest some topics:

PHP legend Derick Rethans suggested I talk about food, specifically biscuits. Well, I must confess, that I do love a good biscuit. Every time I visit the US I always check to see if the restaurant I’m in serves biscuits and gravy. Mmm, mmm, good. Biscuits and gravy is something you just don’t see on most Canadian menus.

But all joking aside, I wanted to share some thoughts that have been rattling around in my brain this past month.

PHP is undergoing what I think could only be described as a personality crisis. First we had a well-intentioned guy making a bunch of patches he did to PHP available. The reaction to this was predictable: the internals team responses ranged from meh to outright dismissal due to his outsider status. Predictably, users of PHP liked a lot of what he proposed. It’s no secret to those who have followed PHP internals, even sporadically, that there is some dysfunction there. It’s understandable: it’s an open source project without a real BDFL and the goals and desires of the internal PHP core team are very different from those, like me, who just use the language and are not in a position to contribute any meaningful changes at the implementation level.

Next we had the #phpneeds meme that made the rounds on Twitter, culminating in the boys at Orchestra.io putting together a page outlining some of those requests. Again, much of what was being said there makes sense but the same old friction came up: the people with the ideas they want to see implemented are not in a position where they can implement them. My C and C++ skills are non-existent (although I have hopes that Zed Shaw’s proposal to do “Learn C The Hard Way” sees the light of day) so all I can do is complain about my inability to effect real change to PHP.

I even made my own contribution to #phpneeds that I thought now correctly puts PHP in the proper light: it is a server-side solution that needs to decide if it wants to be bold or meek. Look, I am not going to hash out all the old arguments here, but I think there are some important issues to discuss. The big one that I think gets ignored is that the core of PHP needs to decide if PHP is going to continue to try and stay at the front of the pack when it comes to it’s suitability as a modern solution, or end up like (don’t hate me for saying this) Perl. By this I mean that it’s still a really good language, but it’s been pushed aside and left to those who either have to support legacy applications or is being used by people who are strong enough to ignore the catcalls from everyone else. Perl can still get the job done, but it’s no longer a consideration for anyone starting a new project…unless they have many, many years of Perl experience and don’t give a shit about how it “looks” to be using a particular tool.

I’m not immune to this sort of thing either: I wish to branch out and build applications using other languages and tools, but PHP is the thing I know the best. I worry about the future: can I build what I call Web Pi sites (Web 3.14159, get it?) full of real-time information and distributed across multiple machines (cloud or otherwise) using PHP as the server-side solution? What changes have to occur at the bottom of the stack to keep PHP as a viable choice for building web apps? Can I figure out how all this shit works so I can actually build something like this?!?

PHP gets the job done, of that there is no doubt. Sometimes I yearn for elegance in the languages I use, but then I realize it makes me sound a lot like a whiny douchebag who hasn’t shipped anything but is quick to criticize other people’s choices. I criticize my choices all the time, whenever I find myself fighting the components used to build the applications I work on. A good author once said “write what you know” and for me that continues to be PHP. Perhaps it’s time to focus less on the language and more on the problem you’re trying to solve. I’ve taken to heart the concept that the language you use for your application is simply an implementation detail, because you can build shitty apps in any number of languages. Because at some point it may be another language that is the one that I know. Things change fast (says the man looking warily at the marshalling forces of Node.js) so be ready to make a paradigm shift when the time is right.

Which brings us back via rambling path to PHP. I think PHP 5.3 actually represented a large improvement for PHP itself: it encouraged people who build things around PHP to re-examine what they were doing and make things better. Witness all the new 5.3+ frameworks being built. Rather than being annoyed by there being 100+ PHP framework like I have in the past, I should look at this as evidence of progress. I might never use one of those new frameworks, but it’s evidence that those who guide the future of the language are trying to make things better. It may not be happening at a pace that people are happy with, but it is happening.

Now where did I put my copy of “Creating Web Pi Architectures”…

Book Review – CakePHP 1.3 Application Development Cookbook

2011-05-30T00:00:00-04:00

(Note: Packt Publishing provided me with a copy of this book in PDF and MOBI formats for the purpose of me doing a review)

Once again, the fine folks at Packt Publishing have asked me to review a book. I must confess that I had mixed feelings reviewing CakePHP 1.3 Application Development Cookbook. First, the only CakePHP work I do these days is maintaining an old, old CakePHP web site (not even version 1.2) and it’s being re-written in Python + Django. Second, I know the author personally and I like Mariano so I was worried about what would happen if I read the book and didn’t like it. So I took the approach that most fit what my current status with CakePHP is: someone who has used it in the past, but hasn’t used it for a while and is looking for practical examples of how to accomplish specific tasks with the framework.

I personally find the “cookbook” style of books to be the most helpful. They enable me to quickly find answers to the problem “show me how to do something realistic using your tool”. Hello World doesn’t cut it for the tools I need to use. A common workflow for me is finding an example of how to do something, and then start tweaking it to meet by particular needs. I certainly cannot be the only one who learns how to do things this way. So Mariano can rest easy since I found his book to be exactly the type of guide an experienced programmer who is not familiar with a specific tool can use.

Like any well-thought-out book, they cover the main components of CakePHP. One of the thing I liked was that you didn’t have to necessarily read the book from front-to-back in order to get something from it. Within each chapter, you will find info on accomplishing common generic tasks using framework-specific code. A lot of it didn’t look too foreign to me, so I’m comfortable that 1.3 is not a huge leap forward in terms of compatibility. Of course, your mileage may vary.

I was also extremely happy to see a chapter dedicated to showing people how to use the CLI shells available to you via CakePHP. Back in the day I wrote a really crappy attempt to create a read-eval-print loop for people choosing to use CakePHP. I called it “the testing shell” and I did a really crappy job of promoting it. It never got a lot of traction because of PHP’s focus on the web and lack of testing culture. Which dovetails nicely into a chapter in this book on writing tests.

While it’s not great that CakePHP ended up not using PHPUnit to begin with, but having SOME testing is better than NO testing. Congrats to Mariano for sticking his neck out and making sure people understand why testing is important and how easy it is to add in tests. Honestly, it’s good that you can also see the tests written for the framework as well. If you tweak things and try to extend CakePHP, you have a pretty comprehensive test suite to find out if you’ve broken anything else.

So I will recommend “CakePHP 1.3 Application Development Cookbook” for the intermediate to advanced programmer who is looking to learn how to accomplish specific tasks using CakePHP 1.3. I think beginning developers are much better off actually learning some PHP before trying to use a book like this. YOu should not be just cut-and-pasting code without being able to understand what the code itself is doing.

Testing Web Services in Zend Framework 1.x

2011-05-02T00:00:00-04:00

Typing that title made me feel weird, but the truth is that we are getting close enough to ZF2 (at least in Release Candidate form) to be released, so for posterity’s sake I will start differentiating between 1.x and 2. Makes me feel better at either rate.

Before I continue, I want to thank Giorgio Sironi for his very helpful blog post about clever mock objects that showed me the technical details on solving my particular problem, along with @elazar and @onyxraven for their suggestions.

I needed to test a remote web service for an application at work. Right away I knew I wanted to use Test Doubles to do it. I wanted to create a mock object to represent the web service I needed to speak too, and create some data fixtures containing known responses to queries so that I could test the code without speaking to the web service itself. A key feature of a unit test is that it should (unless not avoidable) never connect to a web service or a database. Those sort of things are probably better suited as either components tests or acceptance tests. I’m no testing expert, but that is an approach that seems to resonate with me.

The problem I ran into was that one of the web service calls would react differently depending on what parameter is was passed into. Reading the existing PHPUnit documentation did not provide me with enough hints on how to solve this particular problem, so while I dug around duckduckgoing for a solution I asked Twitter for help. One blog post and 3 tweets later, I had the underpinnings for my test. Check it out here

A walk-through is probably required in order to explain what I did.

First, we create a mock object using the Mock Builder interface and map it to the web service object we wanted to use. Then, I loaded in the fixture data I wanted to use, and also populated two arrays that are a critical part of determining which response to return based on the parameter passed in. Finally, I create an expectation object for the ‘get’ method I am trying to test, telling it that I am stubbing out 3 request-and-response pairings. Then I pass the excpectation object and the two arrays containing acceptable parameters and responses into a different method so that it figures out how to handle my request for that method. It took me a while to get this model working in my head to the point where I could explain it.

To make things even easier to test, I created a Client object that accepts an instance of the web service as a parameter, thereby using Dependency Injection / Inversion of Control to make it possible to actually test this thing. Doing it this way, the Client really has no idea that it is in fact talking to a mock web service instead of a real one. All it does know is that it is getting back JSON responses from this “web service”, and the code takes care of the rest.