Docker as a testing tool

January 31st, 2019

Yes, Docker can be used as a tool to help you out with some testing problems. Let me show an example of how it makes some load testing easier.

One of the projects I support at Mozilla is the push notification service that we run. Among the tests that I do for them is a load test using an internal tool that the developers of the service created. It is a Twisted application that runs using PyPy.

No, I do not know why they made those choices. Further more, it doesn't matter because I have to use it so I roll with the weirdness.

Installing it locally on my MacBook Pro was straightforward -- the docs even cover expected weirdness with some support libraries. But now that I have switched over to using Windows at work I ran into some difficulties getting all the dependencies installed in Windows Subsystem for Linux.

In fact, one of the developers uses Windows...and told me of the experience he had at failing to get it to work. "I just use VMWare Player and an Ubuntu VM".

I tried to get that working...and ran into a bunch of issues where I had to disable something called "Windows Defender Credential Guard". Which bugged me because I like to keep security mechanisms in place normally. I followed instructions FROM MICROSOFT on how to do it and VMWare Player still wouldn't run. Kept giving me the same error.

Okay, on to plan B -- Docker.

The first question was "how do I make this work?". I theorized I needed to do something like this:

  • find a base container of Ubuntu 18
  • get inside a running version of that image
  • install all the required dependencies
  • verify that the load testing tool works
  • make a copy of that container
  • push that copy up to Docker hub for teammates to use

Some searching revealed that I could get a running Ubuntu container and connect with it:

docker run --rm --it ubuntu

This downloaded the image and gave me access to it via a Bash shell.

With that shell, I started installing all the packages I would need to install the load testing tool, including adding the package repository for PyPy so I could install it with the package manager.

Once I got all the dependencies installed, I made sure that the tests could run. I even found out that there was a bug in the documenation. ;)

With a Docker image that had a working installation of the load testing tool, the next step turned out to be more straight forward than I had thought.

The Docker feature I needed was called snapshots. Here is what I did.

With the Docker container still running, I opened up another shell (in this case I am using Powershell) and used the command docker ps to get the name that had been assigned by Docker to my running container.

With that name I used the following command to save a version of that container:

docker commit name_of_container ubuntu:ap-loadtester 

Then I tagged the container so I can push up to Docker hub under my account:

docker tag ubuntu:ap-loadtester chartjes/ap-loadtester

Last, I pushed it up to Doker hub

docker push chartjes/ap-loadtester

So now I have:

  • an Ubuntu container I can share
  • that has all the dependencies installed
  • can be used by teammates to load test the push service

Again, I emphasize that a tester with some programming experience can create some really useful testing tools by leveraging the same tools and environments used to build whatever you are testing.

Learn To Test Like A Grumpy Programmer - Part 2

January 16th, 2019

(If you use PHP, you can learn how to write automated tests for your code via my "Learn To Test Like A Grumpy Programmer" course over at LeanPub)

In this blog post I wanted to talk about some of the lessons I've learned at the day job about how to use tools and techniques we are familiar with in the developer world to make our job of testing things easier.

Mozilla's Push Notification Service

One of the projects I do QA work for is the push notification service that Mozilla runs. Yes, I know a lot of people get supremely annoyed by push notifications. Yes, they are heavily abused by people. Mozilla tries to use them in a way not designed to annoy you. But this is the Internet after all, where everything annoys somebody somehow.

My work for that team involves doing testing of the service whenever a new version is ready to be released. The process goes like this:

  • the development team tags a new release along with a changelog
  • the operations team deploys this new release to the staging environment
  • I create a BugZilla ticket to track the deployment and testing results
  • I run a series of tests against the service on staging, recording my progress there

If all my tests pass on staging, I give approval to deploy that version of the service to production. The process for that is:

  • the operations team deploys the new release to production
  • I create another ticket to track the deployment process and testing results
  • I run the same tests and add an additional set of load tests to make sure the service is responsive

So what tests do I run?

API Contract Tests

I wrote some tests using pytest that make API calls using known values and verify that we are getting the responses that we expect.

These tests usually are not difficult to write and I did experiment with making them asynchronous using pytest-asyncio so I could learn how asynchronous code works in Python. If you had a very large test suite, I could see it being useful to speed up the process. This particular test suite is not big enough to warrant that.

For my fellow PHP folks, there are some plugins for PHPUnit that can run your tests in parallel. For one example check out PHPChunkit

Load Tests

I only run the load tests on the production version of the service because the number of nodes that handle push notification requests is higher in production. These tests are designed to put some non-trivial load onto the system and examine the output from the nodes for any error messages.

VAPID Testing

I use two different Android devices for these tests. Each device has latest stable and nightly releases of Firefox on them. One device is configured to point at staging, the other at production.

A web page is loaded that uses Javascript to generate VAPID-based notifications.

In the summer of 2018 I did experiment with seeing if I could automate these tests so I could use an online service offering cloud-accessible Android images. I did make some progress but it seems like some of the pieces needed to do things like make sure I don't have to click on the browser to accept notifications are either really brittle or don't exist, depending on what automation tools you are using.

Desktop Notification Tests

The person who did this testing work before me created a Nodejs app that serves up an web page with some forms and buttons on them. You fill in some values and it creates a series of push notifications that you visually verify work. He then put it inside a Docker container to make it easier for others to run those tests.

This test used to be on a personal web page of a (now-former) Mozilla employee.

Desktop WebPush Tests

These tests are to make sure that a feature of the service where you can group WebPush messages together by topic, only displaying the last one, works correctly.

The process for these tests are:

  • checkout a GitHub repo that contains the test
  • configure everything according to the documenation
  • start up the "topic server" via the CLI, which is serving up a one-page Cyclone web application
  • click on a button fire up a service worker that receives requests to send a WebPush notification
  • then use another CLI tool to fire off notifications (with or without a topic)
  • visually verify that the notifications appeared

Given that I already had a Docker-based solution, I decided it was time to turn this test into something where I only have to click a few buttons instead of running things via the CLI.

The first step was to update the existing one page app to have two more buttons. So I edited the HTML to add some buttons and then added some JavaScript that made calls to two new URL's for the app.

I then added those two URL's to the Cyclone app, porting over code from the CLI "topic pusher" tool to generate the notifications.

There was already a Dockerfile for the application, so I used that and then spent some time building and rebuilding and debugging the application to make sure it did actually make it so all I had to do was the following:

  • download the Docker image
  • start running the image (making sure to tell Docker to forward the correct port)
  • click on the Subscribe button and see the message that the service worker is running and the browser asks to accept notifications
  • click on the button that sends a single notification and visually verify output
  • click on the button that sends multiple notifications grouped by topic and visually verify output

So now I have a test that I can point other folks to that they can use without having to install any dependencies in their local environment other than a web browser and Docker.

Programming Skills + QA work == Solving Interesting Problems

I am far from the only person who has this skill set. But having some ability to create your own purpose-specific tools means that the people around you get to benefit.

Often the tools (and automation solutions) that folks use to test things are proprietary and not open to be modified. So you expend a lot of energy trying to bend a tool towards a new purpose.

There are other folks out there like me who busy creating wrappers around hard-to-use tools or creating new solutions with the goal of making what used to be difficult a lot easier. Encourage those people and promote what they are doing!

Learn To Test Like A Grumpy Programmer - Part 1

December 18th, 2018

(You can see more stuff like this blog post if you buy my "Learn To Test Like A Grumpy Programmer" course over at LeanPub)

I wanted to show folks how someone who has been writing tests for PHP in anger since 2003 actually implements new features for a web site, driving them with tests.

More importantly, I want you to focus on how deliberate and methodical I am in doing this work. I know it's an overused word, but discipline is a key factor in successfully using a TDD approach for your tests. Skipping steps or taking short cuts is a great way to end up with weird errors that you spend a very long time debugging.

I am building [OpenCFP Central])(https://opencfpcentral.com), which is a companion web application for OpenCFP. OpenCFP is a PHP web application that conference organizers can install and use to collect talk submissions from prospective speakers.

OpenCFP Central is a 100% free service.

Right now you can register for an account with OpenCFP Central. For conference organizers, you can obtain OAuth tokens so that people can use their OpenCFP Central account to login and submit talk ideas to OpenCFP. This helps to reduce one of the biggest complaints, which was "why do I have to keep creating OpenCFP accounts?!?"

The next feature I am building is allowing OpenCFP Central users to create talks and store them there. Once I've built that out, then I will add functionality to OpenCFP itself to retrieve a list of talks for a user who authenticated using OpenCFP Central and allow one-click submission of the talk to that OpenCFP instance.

OpenCFP Central is being built using Laravel starting with version 5.7. This is my first Laravel app but so many of the concepts it presents are not revolutionary or even unexpected for folks who have been doing web development with these "front controller style" web application frameworks.

So, with all that out of the way, let me show you how I implemented the first bit of functionality of the new feature: displaying an empty talk creation form for an authenticated user.

I started with a test that assumed everything was working:

namespace Tests\Feature\Feature;

use App\User;
use Tests\TestCase;

class TalkPage extends TestCase
{
    /**
     * @test
     */
    public function it_displays_the_talk_form()
    {
        /**
         * As a logged-in user
         * When I follow the route for "click here to create a talk"
         * I should see the talk form being displayed
         */
        $user = Factory(User::class)->create();
        $response = $this->actingAs($user)
            ->get(route('create_talk'));
        $response->assertStatus(200);
        $response->assertSeeText('Create A Talk');
        $response->assertSeeText('Title');
        $response->assertSeeText('Description');
        $response->assertSeeText('Other Details');
    }
}

Of course, it failed because I did not have the route created. So I went and added this route to routes/web.php

Route::get('/talk/create', 'TalkController@create')->name('create_talk');

Now the test fails because I don't have a TalkController. I used the artisan CLI tool to create an empty TalkController and modified it to use the auth middleware.

namespace App\Http\Controllers;

use Illuminate\Support\Facades\Auth;
use Illuminate\View\View;

class TalkController extends Controller
{
    /**
     * Create a new controller instance.
     *
     * @return void
     */
    public function __construct()
    {
        $this->middleware('auth');
    }
}

The test fails because I do not have the expected method for TalkController created yet. Let's add that in:

namespace App\Http\Controllers;

use Illuminate\Support\Facades\Auth;
use Illuminate\View\View;

class TalkController extends Controller
{
    /**
     * Create a new controller instance.
     *
     * @return void
     */
    public function __construct()
    {
        $this->middleware('auth');
    }

    /**
     * Show the application dashboard.
     *
     * @return \Illuminate\View\View
     */
    public function create(): View
    {
    }
}

Test still failing (as expected) because it's not doing anything. I'll add some code where I grab our authenticated user and ask it to display a specific view.

<?php

namespace App\Http\Controllers;

use Illuminate\Support\Facades\Auth;
use Illuminate\View\View;

class TalkController extends Controller
{
    /**
     * Create a new controller instance.
     *
     * @return void
     */
    public function __construct()
    {
        $this->middleware('auth');
    }

    /**
     * Show the application dashboard.
     *
     * @return \Illuminate\View\View
     */
    public function create(): View
    {
        $user = Auth::user();
        return view('talk.create')
            ->with('user', Auth::user());
    }
}

Now the test fails because that view does not exist. I need to create the appropriate directory structure to group views related to talks and create a blank one that uses our existing layout.

@extends('layouts.app')

@section('content')
    <div class="container">
        <div class="row justify-content-center">
            <div class="col-md-8">
                <div class="card">
                    <div class="card-header">Create A Talk</div>
                    <div class="card-body">
                        <p>
                            Use the form below to create a talk that can be submitted to OpenCFP instances that are
                            using OpenCFP Central
                        </p>
                    </div>
                </div>
            </div>
        </div>
    </div>
@endsection

Test still fails because there is no form information in there. So I created a partial view that contains the form fields I want. I did it without the opening and closing form tags because I plan on reusing this form snippet elsewhere. As part of a future test I need to modify this snippet to set default values as part of verifying you can actually succesfully create or update a test. Sometimes it's okay to think a little bit ahead. This strikes me as a very minor detail.

<div class="form-group">
    <label for="talk_title">Title</label>
    <input type="text" class="form-control" name="talk_title" />
</div>
<div class="form-group">
    <label for="description">Description</label>
    <textarea class="form-control" name="talk_description"></textarea>
</div>
<div class="form-group">
    <label for="other">Other Details</label>
    <textarea class="form-control" name="talk_other"></textarea>
</div>

Next I updated the main "create talk" view to use this form and wrapped it form tags that POST the results to another method.

@extends('layouts.app')

@section('content')
    <div class="container">
        <div class="row justify-content-center">
            <div class="col-md-8">
                <div class="card">
                    <div class="card-header">Create A Talk</div>
                    <div class="card-body">
                        <p>
                            Use the form below to create a talk that can be submitted to OpenCFP instances that are
                            using OpenCFP Central
                        </p>
                            <form method="post" action="/talk/create">
                                @csrf
                                @include('talk.form')
                                <button type="submit" class="btn btn-primary">Create Talk</button>
                            </form>
                    </div>
                </div>
            </div>
        </div>
    </div>
@endsection

Now the test passes!

I hope this blog post helps you understand what a TDD flow with an app under real conditions looks like.