Web Acceptance Tools Suck

October 31st, 2014

Wow. Almost a whole year since I posted something. Time to change that.

It is not a coincidence that I write this blog post on Halloween 2014. Because I want to talk about something that should make your skin crawl and want to turn all the lights on in the location where you are reading this.

I am talking, of course, about writing automated web acceptance tests.

At ZendCon2014 I did a talk about how to use Behat and Mink to write some automated acceptance tests for your web sites. You know, the type of thing where you drive a browser using some code and it pretends to click on things...and you lie to yourself about what value all this work is giving you.

I also made the classic mistake of waiting too long to go over my slides for this talk and realized I was using versions of these tools that were extremely out of date. I then spent 6 hours updating the code, instead of watching my friends give awesome talks about PHP and complementary technologies.

This sucked. Sucked big time.

As far as I can tell, we are currently at the point in the PHP community with these tools like unit testing tools were 10 years ago when I first decided I needed to find ways to never work 120 hours of overtime in 6 weeks leading up to Christmas ever again.

Behat and Mink are both very powerful tools, but their documentation is lacking and the code samples review what I think is a very large disconnect between how the creators of these tools use them and how the rest of us use them. I have no idea if there is even any blame to be handed out, unless you subscribe to the theory that shitty documentation is the fault of the people who created the projects in question.

I am not diminishing the work of those who created these things. They are great programmers, and I am just a grumpy guy who wants things to work a certain way and not act like obstacles. I like my tools to be complementary and easy to figure out how to bend to my will.

But when I look at how brittle these things are, and how slow they are, it makes me wonder if I wasted my time learning how to use them.

For those who don't understand what I am getting at, or think I am acting like the drama queen they secretly hope I am, consider this -- the best way to identify elements on a web page is to use tools that expect you to either fully understand CSS and also how XPath works.

Folks, this is a shit show.

Now, it is entirely possible I am simply Doing It Wrong, and will be happy to be corrected. Take a look at the code in this repo AND DESPAIR. This code is brittle, has to maintain state between test steps manually, and the tests take forever to run because you have the overhead of starting up a browser and painfully crawling through the DOM to find things.

(By the way, who is the person that decided the DOM was the best way to internally represent elements on a web page? If it was Tim Berners-Lee, well, it strikes me as a decision that is biting us in the ass now but was probably totally logical at the time)

I am hopeful that someone out there is working on a better set of abstractions and tools that will make the types of things we are asking Behat and Mink and Page Objects and PhantomJS to do a lot better.

Send me your thoughts and ideas on Twitter or to chartjes@grumpy-learning.com and I will do a follow-up post.

Test Spies and Mockery

December 27th, 2013

While recording some screencasts I was struggling to figure out how to get PHPUnit's built-in object mocking tools to allow me to create what is known as a "test spy". I talk about them briefly in my PHPUnit Cookbook but I think that what I wanted to do in this instance was beyond what PHPUnit could give me.

I had some code-under-test that had a conditional statement inside a foreach() loop (aggravating my desire to use object calisthenics) and I wanted to make sure that both branches of the conditional statement got executed.

I first tried something like this:

    // $db is our mocked database object based off stdClass for testing
    $db->expects($this->once())
        ->method('query')
        ->with($update, ['id' => 1]);
    $db->expects($this->once())
        ->method('query')
        ->with($delete, ['id' => 5]);

I was using Aura.Sql and it's Update and Delete objects. I wanted to be sure that I was using both objects.

I also tried using $this->at(0) and $this->at(1) as well, I got errors ranging from "method query was not mocked" to problems complaining about expected values not showing up at the expected sequence.

I knew there had to be a better way, but I really wanted just to use PHPUnit's built-in mocking. I couldn't figure it out. So instead I turned to a mocking library that I knew supported test spies: Mockery.

The code reads a lot smoother:

    // m is an alias to \Mockery
    $db = m::mock('stdClass');
    $db->shouldReceive('query')->with($update, ['id' => 1])->once();
    $db->shouldReceive('query')->with($delete, ['id' => 5])->once();

The first thing that jumps out at me is that the Mockery version looks cleaner. Well, really, it's only one less chained call. But looks do count for something.

More importantly, my test worked the first time with no weird error messaging about unexpected behaviour.

So the next time you are writing a unit test and need to create spies on methods of a mocked object, I cannot recommend enough that you take a look at Mockery.

The Power of the BrowserProxyMob

November 18th, 2013

At work I have been involved with an effort to put some automated front-end testing in place. The combination of Behat, Mink running tests using PhantomJS is a good one for this. Open source, easy to configure, handles JavaScript-heavy pages reasonably well.

There was just one wrinkle in our plans: our use of local host files.

Somewhere back in the mists of time it was decided that in order to properly test our web applications without pummelling our production assets (like image farms, content servers, API end points) that a "hostfile generator" would be created and is accessible on every machine in our development VM network.

You hit /hosts and it spits out a file that you can use on your laptop to alias production assets / resources / whatever to things on your development server. I use a program on my MacBook called Gas Mask that lets me quicky switch between host files when testing out sites on various development boxes. I am sure a similar tool exists for Windows and Linux users.

So, initially the only way to make Behat play nicely with the host files was to manually over-write the /etc/hosts file on the development server with the host-specific one the generator gives us. That way all the host aliases in place.

This is fine and all, but I am pretty sure our sysadmins would never allow the user that runs our continuous integration jobs have the proper permissions to overwrite the host files. If a test run were to crash and generate fatal errors, we would end up with a machine with a bad host file in place.

So clearly what was needed was a proxy. After doing a little bit of digging around I found a solution: BrowserMobProxy. While it bills itself as a tool for helping web developers "watch and manipulate network traffic from their AJAX applications" it also has a feature that is of great interest to our problem: it supports the ability to create aliases to hosts...oddly enough just like we do with our host files at work.

So, BrowserMobProxy is great in that you can send requests to REST-style but I didn't want to mess around with manual calls, so I was happy to find that my friend Adam Goucher had written a PHP library for interacting with it. I forked his code, cleaned it up a little and it's now available via Packagist and ready to install via Composer.

So, let's get started with what I did.

Here's my composer.json file for the project

{
    "require": {
        "behat/behat": "2.4.*@stable",
        "behat/mink": "1.4@stable",
        "behat/mink-extension": "*",
        "behat/mink-goutte-driver": "*",
        "behat/mink-selenium2-driver": "*",
        "chartjes/php-browsermob-proxy" "dev-master",
        "sauce/connect": ">=3.0",
        "sauce/sasuage": ">=0.5",
    },
    "config": {
        "bin-dir": "bin/"
    }
}

We are experimenting with using SauceLabs for testing our sites on mobile devices (in case you were wondering).

With that stuff installed, I next downloaded a copy of BrowserMobProxy (herafter referred to as BMP) and copied the CLI runner and JAR files into the 'bin' directory. Next, I installed PhantomJS in /usr/local/bin on my server.

Next, we setup our Behat configuration file to point to PhantomJS when doing our tests using Selenium2's web driver capabilities:

phantomjs:
    context:
        class: "FeatureContext"
    extensions:
        Behat\MinkExtension\Extension:
            default_session: selenium2
            javascript_session: 'selenium2'
            base_url: http://synacor.com
            selenium2:
                wd_host: http://127.0.0.1:4444/wd/hub

Now, we get to the tricky part.

BMP requires you to connect to it first to get it to start a proxy connection running on a different port than the one the main service runs on. In my Behat test runner script I use the PHP BMP library to create a connection, and it assigns it to the first available port in a range you can specify at run time.

Because I can count on this value to be the same all the time, I then start up an instance of PhantomJS telling it to run in "accept requests like I am WebDriver compatible" proxy all requests through BMP

/usr/local/bin/phantomjs --webdriver=4444 --proxy=http://localhost:9091

In my Behat context class I add in a step that reads in the host file via our host file generator and assigns the host aliases to BMP so that when Behat uses PhantomJS to load web pages, it will use the host aliases set within BMP.

I know that it's pretty convoluted, but I know with 100% certainty that it works.

If you spot anything here that doesn't make sense, let me know and I will update the post.