Using markers in pytest

January 15th, 2018

At the current day-job I use pytest for writing what I call "configuration tests" for the various services and tools that I am responsible for doing QA work for.

One of these is Kinto, a "generic JSON document store" that we wrote (and open-sourced) at Mozilla and is used by a lot of services.

These days there are three "flavours" of Kinto that I have to worry about:

  • kinto-dist which is the core of Kinto
  • kinto-settings which is the Kinto core plus a bunch of records representing settings stored for users of Firefox
  • kintowe which is the Kinto core and some extensions for it that we use to track records of what web extensions Firefox users have stored

Whenever we have a new version of any of these "flavours" that need to be tested, it is deployed to a target environment (either staging or production) and tests that I wrote are run by our operations team's deployment tool making a call to a CI server run by the QA department. This CI server then runs all the tests I wrote.

But I had a problem. There were some tests that needed to be run no matter what "flavour" we were deploying, and then some tests that belong only to one "flavour". Previously I just had duplicate repos but wanted to consolidate them.

I enlisted the help of one of our team tool-makers Dave Hunt while at our All-Hands meeting in Austin in December of 2017 to go over some better strategies and to make better use of our CI server.

After scribbling a lot of stuff on a white board to get Dave up to speed, he suggested moving all the tests into one repo and using a feature of pytest called "markers" to figure out what "flavour" we were testing.

Note: at the time of writing I was doing this work with Python 3.6.3 and pytest 3.3.2.

Pytest makes heavy use of decorators to extend its functionality. Fixtures (which can be best described as helper methods for tests) are declared using annotations, and then these fixtures are available to any test method that you pass it into as a parameter (with some exceptions based on the scope of a fixture).

Here's an example of a fixture that reads in a configuration file and returns an object that contains those values:

def conf():
    config = configparser.ConfigParser()'manifest.ini')
    return config

To use it, I just pass it in as a parameter like this:

async def test_server_info(api, conf, env):
    res = await api.server_info()
    data = await res.json()
    expected_fields = aslist(conf.get(env, 'server_info_fields'))

    for key in data:
        assert key in expected_fields

    for field in expected_fields:
        assert field in data

To use a marker, I simply add an annotation to tell pytest "mark this test as belonging to dist" or any other "flavour" I want this test to belong to. Check out the annotations for this test:

async def test_version(api, conf, env, apiversion):
    res = await api.__version__()
    data = await res.json()
    expected_fields = aslist(conf.get(env, 'version_fields'))

    # First, make sure that data only contains fields we expect
    for key in data:
        assert key in expected_fields

    # Then make the we only have the expected fields in the data
    for field in expected_fields:
        assert field in data

    # If we're passed an API version via the CLI, verify it matches
    if apiversion:
        assert apiversion == data['version']

There is a lot going on here, let me break it down:

  • we are marking this test as being able to run asynchronously using Python 3's asyncio module
  • we want this test to be run any time we are testing dist
  • we want this test to be run any time we are testing settings
  • we want this test to be run any time we are testing webextensions
  • we are passing in an API helper as a parameter
  • we are passing in a configuration helper as a parameter
  • we are passing in a helper that reads a target environment parameter from the CLI
  • we are passing in a halper that reads an API version parameter from the CLI

Here's a test we only want run when we do a kinto-settings deployment:

def test_plugins_signatures(env, conf):
    client = Client(
        server_url=conf.get(env, 'reader_server'),
        collection, records, timestamp = get_collection_data(client)
        if len(records) == 0:
            pytest.skip('blocklists/plugins has no records')

        assert verify_signatures(collection, records, timestamp)
        assert verify_signer_id(collection, 'onecrl_key')
    except KintoException as e:
        if e.response.status_code == 401:
  'blocklists/plugins does not exist')'Something went wrong: %s %s' % (e.response.status_code, e.response))

I also needed a way to figure out what markers were set so I could use the correct value from our configuration file (depending on the environemnt). Pytest has a request fixture that is globally available to any test.

I have another helper that uses Swagger to parse an API spec that the developers built for me. I pass the request object to the API helper, and look for specific markers:

def api(event_loop, conf, env, request):
    api_definition = 'dist_api_definition'

    if 'settings' in request.node.keywords:
        api_definition = 'settings_api_definition'
    elif 'webextensions' in request.node.keywords:
        api_definition = 'webextensions_api_definitions'

    return API(conf.get(env, api_definition), loop=event_loop)

(By the way, you can find all this code in the GitHub repo for these tests)

So, when we are doing a deployment of kinto-dist to our staging environment, we can run the tests this way:

pytest -m dist --env=stage config-tests/

Then the tests we want to get run, get run. I'm not sure what other ways there are to organize your tests, but this is a method that works and makes sense to me. Got any comments or suggestions? Email me or contact me via Twitter using details in the sidebar.

January 2018 board game night

January 14th, 2018

For many years I have been getting together with a group of people that I met online when I used to play a lot of the paper-and-pencil role-playing game Hero System. It started getting harder and harder to get everyone together to play, which makes any kind of long-running campaign impossible to maintain. If you don't know from session to session who is going to be available, how can you plan anything?!?!

So a few years back I decided the path going forward was for us to play board games instead. That way if someone can't make it, it's not as big a deal as having a character whom you built an entire campaign around not showing up.

An email goes out saying it's time to play, we all indicate what Saturdays work for an upcoming month (it's always a Saturday) and once we get consensus we make plans to show up at someone's house (we all take turns) and a good time is had by all attendees.

I apologize for the lack of pictures of the various board states but the group is usually on my case to "put your damn phone away, Chris!!!" so I am trying to be a better friend and tame my obvious phone addiction. This time out 6 of the 8 members of our group showed up.

One of the more interesting challenges is making sure that we have games that can handle that large number of people.


We played this game while we waited for the last two members to show up.

My mother got me a beautiful hand-crafted board game called Hadz that reminded me of the old game Sorry with some betting involved.

The game looks like it would take a while to go all the way through and I am pretty sure we are missing out on some of the strategies that might lead to better outcomes. It was fun yelling at the top as it failed to knock a marble into the locations you had bet on.

Joking Hazard

If you've played Cards Against Humanity you will be familiar with this type of game. In Joking Hazard you are trying to build the best three-panel comics with cards that contain the type of material that is best played either by people who know each other very well or by people who are easily amused by toilet humour or aggressively-sexual content.

Not really my game but man some of the comics that got created were quite dark. We played two rounds of that and then prepared for the main event.

Space Cadets

In this game you are a bunch of Space Cadets who are trying to complete a mission as the crew of a spaceship. Each turn the crew members are trying to solve a puzzle specific to their assigned station (helm, engineering, weapons, etc.) in order to advance the mission. It's frantic, you always feel stressed and I'm not sure I really want to play it again.

We've played it at least 4 times and have never one. In fact, I have no idea how anyone ever completes the mission because the quality of puzzles that need to be solved is quite uneven, and the critical station (tractor beams) requires you to memorize the number, colour, shape, and location of puzzle pieces by flipping them over two at a time.

The game can be exciting while you yell at crew members who don't complete their tasks and put the crew in danger, but I think I'm done with it.

Quartermaster General

This game can be best described as "Axis and Allies that you can actually get done in one night." Quartermaster General is a game where all the players take the role of one of the Axis or Allied powers in World War 2 and play out the war using a series of cards and units representing tanks, ships, and aircraft. The game itself took us about 2 hours but the first 30 minutes was just going over all the rules and doing some sample turns.

I really enjoyed this game. I got to be Germany, and I saw how the game creators did a great job of representing the various power levels of the combatants in the war. All of us made plenty of mistakes, and in the end the Allies won victory by 8 points over the 20 turns. The owner of the game said that he'd never seen a game go to turns. I will definitely want to play this again.

There is an expansion set for that shakes things up via additional cards that allows things to be more of a 50-50 proposition for the Axis. It was pointed out to use that there were basically for scenarios all based on on how well one side starts

Axis fast + Allies fast = Allies win Axis fast + Allies slow = Axis wins Axis slow + Allies fast = Allies ROFLstomp win Axis slow + Allies slow = Allies win

Making it 50-50 allows for strategies to matter rather than simply duplicating history.


  • would play Hadz again with a smaller group to kill some time
  • not that enthused about Joking Hazard
  • don't want to play Space Cadet any time soon
  • would definitely play Quartermaster General again

What A Grumpy Programmer Uses - 2018 Edition

January 5th, 2018

I really like browsing through Uses This to see what folks from all sorts of industries use to get the job done. Since they are unlikely to invite me, here's my own details. I share in hopes of discovering new tools for doing my job and showing folks who haven't been doing this for 20 years what "professional programmers" use. You might be surprised.

Who are you, and what do you do?

Monday to Thursday am I working for Mozilla as part of the Firefox Test Engineering group where I test web services that Firefox talks to, along with deployment testing of some internal developer tools and a generic JSON document store that we built ourselves.

On Fridays I continue to work on Grumpy Learning stuff. I'm currently writing a "how to test legacy PHP code" video training course for a large online education provider and are organizing a personal-development course called GrumpyConf that will be held March 22-24, 2018. No new books on the horizon and I currently only appearing at phptek 2018 in late May of this year.

If you want me to come speak at your event in the fall, get in touch.

What hardware do you use?

My main device is a 13-inch 2016 MacBook Pro with Touch Bar with 16 GB of RAM and a 512GB SSD drive. It runs in closed-mode 99.999% of the time plugged into a Dell P2715Q display and I use the Apple Magic Keyboard and Magic Mouse 2. When I travel I bring a Roost laptop stand with me. Being a Large Human using the Roost has made it so I can actually work in environments other than my configured-for-me home office.

I also make heavy use of an iPhone 6S Plus mostly for messaging and Twitter use. I also have a Samsung Chromebook that I can do some PHP web development work on but it is not powerful enough for the day job. I tried it out to see if it was suitable as a road laptop for a world where border crossings can quickly turn ugly. Right now it's not.

I listen to a lot of music (yay Spotify) and since I did notice the poor quality of audio from my laptop I splurged and bought a DragonFll USB DAC along with the Jitterbug USB data and power noise filter. I now have awesome, clean-sounding music coming from my desktop speakers.

And what software?

I use macOS (always using the latest stable version of it -- patch your systems people!) as my development environment and some flavour of Linux (usually Ubuntu) as a deployment environment.

When I am doing Python (which is what I use at Mozilla) I am using PyCharm and if I am doing PHP I am using PhpStorm. For everything else I am using Neovim. I used to be a Vim-only person but over the years I have found the value that IDE's provide me to be invaluable. They have totally-acceptable-to-me Vim bindings anyway.

I make heavy use of the CLI and use iTerm2.

For work I rely on the following tools every single day:

  • IRCCloud for talking to co-workers
  • Mozilla uses Google Mail and Google Calendar, which works nicely with my phone
  • Vidyo for video conference -- great product, works well
  • Firefox web browser. I run the Nightly version

When I do my podcast with Ed, I use Skype with the Call Recorder plug in. I send the recording of my side to him and he fixes it all up.

On my iPhone I use the following apps quite a bit:

I also use all sorts of other apps (messaging ones mostly) like any long-time user of an iOS device.

I am a competitive-casual Magic: The Gathering player, mostly playing the Legacy format (but with a splash of Vintage and the random cube or EDH game mixed in). For many years I was frustrated with my lack of success at the tournament level. I was lucky enough to fall in with a group of people who made me realize that I actually enjoyed the social aspect of the game the most.

What would be your dream setup?

I've worked from home for 11 years now -- I never want to go back to a full-time office experience ever again. I'm totally happy with my computer and my display and all the other tools I use. I tend to upgrade things incrementally instead of in big shots. I like consistency and the ability to quickly get back to work when upgrading any of my equipment.

I hope this helps! Got any more questions or suggestions of things I should check out. Hit me up on Twitter or via the email link in the sidebar.