Testing XML-based API's

January 18th, 2017

At the day job I deal with making sure that a number of different services get tested sufficiently that we have confidence that what's being deployed and then used by Firefox is correct. This is a big change from what I did before I got to Mozilla - I mostly wrote unit tests and integration tests for code. These services are like black boxes as far as QA is concerned -- I do have some insight into the code itself by being involved in code reviews to make sure unit and integration tests are still in place but my task is to make sure the service is functioning as expected.

Since it's 2017, most of the services around here are returning JSON-based responses but Balrog returns XML that is consumed by Firefox itself to let you know any updates that need to be made to Firefox itself or any add-ons you've installed.

Normally I would write some pytest scripts that use the Requests library and then parse the XML responses. These days I like to create what I call "API contract" tests that look at the API responses and verify not that the contents of the response make sense, but we look at the "shape" of the responses to make sure we are getting expected fields and other content. This helps us catch any unexpected changes to the body of the response and also verify the documented calls to API's are still correct.

So I was all set to dive into parsing XML in Python (shoutout to lxml) when my co-worker Tarek suggested an easier path is to use DTD (Document Type Definition) files instead. "It would be easier than what you are trying to do and probably faster." Tarek is very wise.

So the process for the test would become something like this:

  • make a request to a known Balrog API endpoint
  • grab the XML response
  • validate it against the DTD
  • assert that the validation worked
  • profit!

I know I mentioned I am using Python for this but the same principles apply to any language where there are tools available to manipulate and validate XML using a DTD.

So, after conversing with the main developer on the project I got a list of sample URL's and got some details about what fields where required, what ones were optional, and some potential different responses. Next I needed to create some DTD's that act as our validation set. For example, here's a typical response from Balrog:

<updates>
    <update type="minor" displayVersion="50.1.0" appVersion="50.1.0" platformVersion="50.1.0" buildID="20161208153507" detailsURL="https://www.mozilla.org/en-US/firefox/50.1.0/releasenotes/">
        <patch type="complete" URL="http://download.mozilla.org/?product=firefox-50.1.0-complete&os=win64&lang=en-US&force=1" hashFunction="sha512" hashValue="1c2cea9770c814c58058c66ad9f99c678bf1612c8e05960fe415772383c4ab5e293eafef51b8b574307667a880c567d71b0d32c89d2c65dae02f68967991f8f7" size="56892755"/>
        <patch type="partial" URL="http://download.mozilla.org/?product=firefox-50.1.0-partial-50.0.1&os=win64&lang=en-US&force=1" hashFunction="sha512" hashValue="52dab15fb6cbfb3a324a117fb5f1576a0b5947555d1a4535e0f5735a918e817d7c03f5300b3624883758b3a9300a9061a0e190087f1653fef39eb77b81311f69" size="13185929"/>
    </update>
</updates>

(Apologies for the long line lengths)

FYI I used this documentation to figure out what I needed to do to make the DTD work.

Now, what does the DTD look like?

<!ELEMENT updates (update*) >
<!ELEMENT update (patch+) >
<!ATTLIST update
type CDATA #REQUIRED
displayVersion CDATA #REQUIRED
appVersion CDATA #REQUIRED
platformVersion CDATA #REQUIRED
buildID CDATA #REQUIRED
detailsURL CDATA #IMPLIED
licenseURL CDATA #IMPLIED
showPrompt CDATA #IMPLIED
showNeverForVersion CDATA #IMPLIED
actions CDATA #IMPLIED
openURL CDATA #IMPLIED
notificationUrl CDATA #IMPLIED
alertUrl CDATA #IMPLIED
promptWaitTime CDATA #IMPLIED
backgroundInterval CDATA #IMPLIED >
<!ELEMENT patch (#PCDATA)>
<!ATTLIST patch
type CDATA #REQUIRED
URL CDATA #REQUIRED
hashFunction CDATA #REQUIRED
hashValue CDATA #REQUIRED
size CDATA #REQUIRED >

There is a lot going on in here so let's break it down. This DTD describes the elements and attributes I am expecting to find in this type of response from the service.

  • There needs to be a top element updates
  • It has zero or more update elements (the * indicates that)
  • Each update has a bunch of attributes that are required or optional (that's what #IMPLIED refers to)
  • Each update should have one or more patch elements
  • Each patch element has some required attributes

The Python code for a test is pretty simple. Again, substitute your own preferred tools in your own language of choice.

'''
We're using lxml for the XML parsing and validating
url contains the full URL to the API endpoint
'''
repsonse = requests.get(url)
root = etree.XML(response.text)
# Load our DTD file
f = open('./api-tests/updates.dtd')
dtd = etree.DTD(f)

# Validate the response against the DTD and show the errors if it fails
valid = dtd.validate(root)

if valid:
    assert True
else:
    print(dtd.error_log.filter_from_errors())
    assert False

As a rule I try and find libraries and tools that solve problems for me rather than write my own. In this case, lxml and the use of DTD's was the right choice than my initial thoughts of using XPath to hunt for attributes inside the elements.

Hope this gives you a different perspective on strategies for testing API's.

How A Grumpy Programmer Secures Their Laptop

January 9th, 2017

On the /dev/hell podcast episode I recorded with Ed last night, I got the chance to talk at length about my early experiences with my new laptop. According the 'About This Mac':

  • MacBook Pro (13-inch, 2016, Four Thunderbolt 3 Ports)
  • Processor 2.9 GHz Inetl Core i5
  • Memory 16 GB 2133 MHz LPDDR3
  • Startup Disk Macintosh HD
  • Graphics Intel Iris Graphics 550 1536 MB
  • Pretentious Level High

Thanks to a focus by Apple on People Not Like Me, I was able to get up and running really quickly on my new laptop. The Migration Assistant worked perfectly except for not copying over some saved game files for a Steam game that I play quite a bit (Football Manager 2016). Especially when I have a setup that requires the use of SSH keys and applications all configured to my liking, this was awesome.

My next thought turned to security. Clearly we are in a era where attempts to access people's computers is on the rise. Not that I am thinking I am the target of a shady cabal of l33t hax0rs being paid by shadowy security forces of governments that don't like my politics, but I want to at least make them work a bit. So I want to share what I decided to do.

I've been using FileVault for a long time (in fact, it was a requirement for me if I wished to use my own equipment while working for Mozilla).

After that, you have all sorts of options. After seeing a Tweet from someone mentioning a bunch of tools that can help increase the security of your laptop I decided to take the plunge.

First, I installed Little Snitch. It monitors all my network connections and provides me with a bunch of options to allow or deny the connection, forever or just for a limited time. Starting with this tool I had to (and still are, to a minor extent) acknowledge and decide what to do about a ridiculous number of connection attempts by all sorts of programs. For the older crowd, I feel like I am playing some new version of Everquest. So. Much. Furious. Clicking.

Not content to develop repetitive stress injuries to my right hand, I installed Little Flocker. It's a good complement to Little Snitch -- it watches for any interactions with files, looks for keystroke loggers, and checks for malware. More. Clicking.

Next up was to install Micro Snitch to tell me any time my webcamera and microphones were being used. More alerts to acknowledge but at least my microphone only turns on when I need it to. So far.

Finally I installed BlockBlock to let me know if something keeps trying to install malware in known locations. Just another layer of security for someone to overcome. They clearly indicate that the application is in beta, so keep that in mind.

With those apps installed and running and configured, I massaged my very sore wrist and started reading this awesome document at the suggestion of a kind soul on Twitter. Lots of great stuff in there that you can do and raises interesting points about deciding what type of threats you are looking to protect yourself from. Here's a list of the advice from it that I followed:

  • patch everything when updates are available
  • frequent system backups (shoutout to Backblaze)
  • full-volume encryption
  • third-party firewalls
  • Disable Spotlight Suggestions
  • Use Homebrew
  • use dnsmasq, DNSSEC and dnscrypt
  • turn off captive portal
  • use Privoxy as a local web proxy

I plan on implementing some of the other recommendations, but that's what I started with. For Mac users, please read through that document. So much good stuff along with explanations of why you should do it.

Hope that helps!

From macOS to Windows 10 - Part 3

November 28th, 2016

Could Be Called 'Revenge Of The Comfortable'

Welp, the Surface Book sat on my desk next to my MacBookAir in it's Henge Dock and didn't get used beyond Monday. I had some rough times with it that made me retreat back into the comfortable arms of macOS.

In our previous post I mentioned some things I had to take a look at. How did that go?

Needed an HTTP client similar to Paw

Ugh -- nothing I found was similar enough and I also encountered something that became a recurring theme -- how much work did I want to do in order to master a skill but using a different tool? The answer was "not much".

That's probably a personal failing but at age 45 I'm not sure how much time I want to spend remapping those hard-fought memory mapped skills. I'm sure you are starting to guess what the final conclusion might be.

Connect To External Monitor And Keyboard

I didn't bother pairing the bluetooth keyboard I'm currently typing on to the Surface Book, but getting the monitor connected was. Multiple hours spent trying to figure out what the problem was. Was it the drivers? Was it my old monitor? What the hell was happening here -- mini display port to HDMI works just fine on my Mac. In the end, it would only work when I used a mini-display-port-to-VGA connector. It's 2016 and I was highly disappointed.

This then prompted another hour of me searching around looking at monitors (hey, a 4K one sounds great) to discover it wouldn't work with my current laptop and might not work correctly with a Surface Book (not all apps scale properly).

Better Hosts File Management

"Just edit it with Notepad" -- said by people that never used Gas Mask

Battery Life Is Weird

One of the reasons I thought the Surface Book would be enticing is I could detach the screen and use it as a tablet. That actually worked okay...but I would run low on battery after about an hour of usage. That is way less than what my ancient iPad 3 gives me.

I don't really use my laptop much unplugged, but a tablet that has really poor battery life isn't that great.

Maybe The Best Change Is No Change

Look, I know people are going to think I'm weak-willed about this. Yes, the Win10 platform has made leaps and bounds. I did find it jarring to use, and I was actually able to do everything I needed to do at my day job with it. Bash on Win10 worked great (except for curl not working correctly). Atom was a more than suitable editor. Firefox works just fine on Win10. I could do most of what I want to do on Win10. But I would have to relearn a bunch of tools. I'm not sure I want to do that.

I have to give back the Surface Book when I get back from a work meeting in Hawaii (yes, sucks to be me) in a few weeks, so the decision is far from over. One review I read said don't get it if you have a newish MBP while another felt that Apple had built a great machine for hackers.

Where does this leave someone with a 4-1/2 year old (that's 45 years in internet time) MacBook Air? Even more confused than before.

I don't need the Touch Bar because my laptop will run in clamshell/lid-closed mode approximately 99% of the time. What I really wanted was a MacBook that has 32 GB of RAM. I can't get that right now. But will there be one available in April? I would be super-pissed if that happened after I bought a 16GB one.

I'm still more indecisive about it than ever.