A Different Twitter Experience

March 27th, 2017

It's no secret that I am on Twitter a lot. I mean, really a lot. Almost 100k worth of posts a lot. But as I use it more and more I find I was spending way more time treating Twitter as read-only, when what I wanted was to actually interact with people! Maybe that is a naive position to take, but it is what it is.

Then I stumbled upon this blog post where someone described an experiment they were trying out. The code for it was written in Node and I couldn't get it to work so like any pragmatic programmer I used the tools I am most comfortable with and wrote a version in PHP.

So every morning a cron job kicks off on a server and looks at the last 30 days of my Twitter timeline. It looks for users who I have "interacted" with during that time frame. What is "interaction"? It's defined as:

  • you replied to, favourited, or retweeted a tweet of mine
  • you favourited or retweeted a tweet I was mentioned in
  • I mentioned you in a tweet
  • I replied to or favourited a tweet of yours

The results so far have been interesting. I definitely see a different group of people on my feed than I have expected. Which is entirely the point! It has made me use Twitter differently...which again is the point. I'm far freer with my favourites and retweets than I was before. I'm also slowing down to actually read my timeline rather than just scroll through it super quickly like a trained monkey or something.

It's also making me be a lot less judgemental of the people who do end up on this list. While some of the people I follow are definitely blasting out stuff I am not all that interested in, I'm willing to take a chance and not just mute them as soon as I don't like the minutiae of their life. Diversity is a good thing and filter bubbles are not.

Another side effect of this has been some people proclaiming "I didn't know you were following me!" when I say something to them. Hate to burst your bubble but I'm not following you -- you are on my 'cycle' list and I can see your Tweets.

I know some folks follow accounts that send out emergency information and things like that -- I work from home so things like that are not of great use to me. YMMV etc etc.

If you're looking to have a different experience with Twitter, try my code out and see what kind of list it can generate for you. I'm going to keep using this for the near future because I feel like it's a worthy experiment. In fact, some Twitter accounts I felt were essential to follow have just slid off the list and I haven't missed them.

Fighting Fear and Loathing In Crested Butte

March 12th, 2017

This past week (March 5th-10th) I went to Crested Butte, Colorado to attend the Winter Tech Forum with my fellow Mozillian Matt Brandt. He'd been encouraging me for about a year to come out for a visit and see the conference for myself.

I went intending to push myself out of my comfort zone. I pushed a little too hard.

WTF (pretty sure it was named this on purpose) is an Open Spaces conference. This means that there are no set speakers -- attendees self-organize and the goal is really for smaller sessions where it's more of a discussion than your traditional lecture-style session. It seems to me it leans more towards having a number of attendees in the 30-40 range -- Bruce Eckel, one of the organizers for this event, mentioned he felt things started to change once you get up to 50 people. This year's event was just a little over 20.

So, the first thing that caused me some stress was the very first session where Bruce explained how everything worked and the attendees introduced themselves. When it was my turn I introduced myself and talked about how I came into this event with no expectations and was going to try and just go with the flow.

It was very strange to go from being a known quantity after all these years of giving talks at PHP conferences to being absolutely nobody at this event. I'm not going to lie -- it was a very uncomfortable feeling for me. There were four newcomers, I only knew Matt, and it seemed like EVERYONE ELSE knew each other.

So the sessions started going and there were some interesting discussions:

  • a quick intro to Machine Learning led by someone who worked for a consultancy specializing in it
  • I led a discussion about how I manage social media and a discussion about whether or not to interact with haters and trolls
  • offered some input and my experiences to a discussion on remote working

The afternoons were ours to do with as we wished. A lot of attendees liked to ski or do other activities. Since I am usually against sweating while outdoors, I retreated to my charming little Airbnb and kept on doing a big refactor of OpenCFP to add in consistency for the use of forms and replacing Sentry (the authentication and authorization library) with it's replacement from Caratalyst, Sentinel.

Tuesday was more of the same -- open discussions in the morning and then we were free to do whatever we wanted in the afternoons. Tuesday's sessions for me were:

  • Matt led a discussion on app and web accessibility issues
  • What had people done with hardware devices and ideas for the Wednesday hack day

I didn't go to the last sessions, I instead went to go hack some more on OpenCFP in order to have something to show off on Wednesday night. Had some lunch at one of the many fine restaurants in Crested Butte, and went back to my Airbnb.

Wednesday I spent the entire day working on more OpenCFP stuff -- adding tests to the application is a very slow and deliberate process and I am trying to be a Good Open Source Project Maintainer. After dinner everyone went back to the conference venue (the local parish hall) and I fought against my fear and loathing by going first and doing a quick 5 minute demo of OpenCFP. What else did we see?

  • demo of using machine learning to detect angry tweets
  • demo of a Slack bot that uses Markov chains of some Twitter accounts to generate output
  • Alexa-as-a-mentalist demo
  • Using an Echo dot to turn a disco ball off and on
  • Adding some Alexa skills to give people information about Crested Butte
  • A Brainfuck compiler written in Clojure
  • Using a Leap Motion controller to allow people to cast spells from Harry Potter by hand
  • A quick demo of Pony

There were two other demos that I cannot remember and didn't take notes for, sorry folks.

Dinner that night was at an awesome restaurant that specialized in dishes for sharing -- the 12 of us had one of (almost) everything on the menu. Probably the best $50 I ever spent on a dinner.

Then Thursday happened.

I went to some sessions but my brain wouldn't let go of a fear of going to the progressive dinner event. It starts off by going to a house (many of the attendees rented houses for the event) for appetizers, then a different house for the main course, and a third place for dessert.

I watched sessions on:

  • keeping things like login and passwords your app needs from being leaked out
  • a jumpstart for a functional programmer getting into Python

Then I went back and hacked some more, had lunch, and went back to my Airbnb.

It's been a very long time since I've had a panic attack. The last time was when I was going to do a joint talk with Stuart Herbert at ZendCon and he couldn't come due to medical issues.

My brain just latched onto:

  • 20 people
  • you don't know any of them
  • it's too many people in one small house
  • I WILL NOT LET YOU GO

So I remained in my Airbnb and had a salad I had bought at the local grocery store. I felt bad that I couldn't go -- but many of the attendees approached me on Friday to lend me their support and tell me they have often felt the same thing. That was the one big thing that made this event really, really different -- everyone was 100% serious about giving yourself permission to do as much or as little as you wanted at this event.

Friday was a much, much better day.

  • Session for the new attendees to discuss their experience
  • Integrating remote folks into your teams

Then the closing session where Bruce talked about his own experiences this year, some changes he was thinking about, and his own plans for future events like this in 2017. We went around the circle sharing our own experiences and then we took off for lunch.

I went back to my Airbnb and worked on converting over a CakePHP application from 2.x to 3.4. Of course the ORM has changed things so I have lots of work to do.

Then the finale of the event was awesome -- dinner at a yurt that you could only get to my snowshoeing, skiing, or snowmobile. There were multiple starting points depending on your own comfort level and experience level. I had decided on the 1.5 mile starting point and would snowshoe in. I had never used snowshoes before.

This time I felt comfortable having dinner with the whole group. The snowshoe experience was not going to deter me from making it.

In the end, the experience was GREAT. We walked as the sun set, great conversations to be had and then a great dinner prepared in the yurt for us. I sat at a table with all the other attendees named Chris (there were 4 of us) and I shared the tale of my grandfather's experiences as a German soldier who got caught in the siege at Stalingrad.

The return walk was even better -- walking in bright moonlight combined with a great talk with an attendee about his upcoming job helping to build the infrastructure for an autonomous car company. It made me regret not participating in the progressive dinner the night before.

Saturday I got up early and Matt drove me back to Durango for my flights home. Again, great chats with Matt over the 4-1/2 hour drive and I feel like we really turned a professional relationship into a real friendship. My flights home were uneventful and I grabbed my car from the off-airport valet parking around midnight.

So what are my big takeaways from this event?

First, it's really intimidating to get into meaningful discussions on any topic with a group of strangers. By the end of the week I felt like these people weren't strangers any more -- it felt more like work colleagues. It's clear to me that the event (which is 10+ years old if I recall correctly) acts as a support and networking group for these people. They really care about each others' lives and careers. It was great to see.

The second thing is a bit of weird perspective from me. For a lot of people, they get the impression that what they are doing is unique and then these sort of events show you that others are doing or experiencing the same things.

I had always felt that what I was doing was very similar to other peoples' experiences. Instead, I discovered much of my experience and views on remote working, personal management, and communication skills are quite unique. Attendees told me they really liked how I articulated my views on these topics and that they got a lot out of it. Nice little boost for the ego, to be sure.

So, will I be back? I want to go back and immerse myself more into this group. Hopefully I can convince Ed to come with me.

Thanks to Bruce for being so warm and welcoming and treating me like I had been part of this forever. Extra special thanks to Matt for being my conference buddy. I would not have gone to an event like this without knowing someone.

Finally, this event has made me wonder if my 2018 event idea (GrumpyCon) should be something similar. I am moving in the summer and maybe I get lucky and wherever I end up has the ability for me to allow me to host 30 people for a week of in-depth discussions and experiences about PHP and other topics people want to discuss.

Testing XML-based API's

January 18th, 2017

At the day job I deal with making sure that a number of different services get tested sufficiently that we have confidence that what's being deployed and then used by Firefox is correct. This is a big change from what I did before I got to Mozilla - I mostly wrote unit tests and integration tests for code. These services are like black boxes as far as QA is concerned -- I do have some insight into the code itself by being involved in code reviews to make sure unit and integration tests are still in place but my task is to make sure the service is functioning as expected.

Since it's 2017, most of the services around here are returning JSON-based responses but Balrog returns XML that is consumed by Firefox itself to let you know any updates that need to be made to Firefox itself or any add-ons you've installed.

Normally I would write some pytest scripts that use the Requests library and then parse the XML responses. These days I like to create what I call "API contract" tests that look at the API responses and verify not that the contents of the response make sense, but we look at the "shape" of the responses to make sure we are getting expected fields and other content. This helps us catch any unexpected changes to the body of the response and also verify the documented calls to API's are still correct.

So I was all set to dive into parsing XML in Python (shoutout to lxml) when my co-worker Tarek suggested an easier path is to use DTD (Document Type Definition) files instead. "It would be easier than what you are trying to do and probably faster." Tarek is very wise.

So the process for the test would become something like this:

  • make a request to a known Balrog API endpoint
  • grab the XML response
  • validate it against the DTD
  • assert that the validation worked
  • profit!

I know I mentioned I am using Python for this but the same principles apply to any language where there are tools available to manipulate and validate XML using a DTD.

So, after conversing with the main developer on the project I got a list of sample URL's and got some details about what fields where required, what ones were optional, and some potential different responses. Next I needed to create some DTD's that act as our validation set. For example, here's a typical response from Balrog:

<updates>
    <update type="minor" displayVersion="50.1.0" appVersion="50.1.0" platformVersion="50.1.0" buildID="20161208153507" detailsURL="https://www.mozilla.org/en-US/firefox/50.1.0/releasenotes/">
        <patch type="complete" URL="http://download.mozilla.org/?product=firefox-50.1.0-complete&os=win64&lang=en-US&force=1" hashFunction="sha512" hashValue="1c2cea9770c814c58058c66ad9f99c678bf1612c8e05960fe415772383c4ab5e293eafef51b8b574307667a880c567d71b0d32c89d2c65dae02f68967991f8f7" size="56892755"/>
        <patch type="partial" URL="http://download.mozilla.org/?product=firefox-50.1.0-partial-50.0.1&os=win64&lang=en-US&force=1" hashFunction="sha512" hashValue="52dab15fb6cbfb3a324a117fb5f1576a0b5947555d1a4535e0f5735a918e817d7c03f5300b3624883758b3a9300a9061a0e190087f1653fef39eb77b81311f69" size="13185929"/>
    </update>
</updates>

(Apologies for the long line lengths)

FYI I used this documentation to figure out what I needed to do to make the DTD work.

Now, what does the DTD look like?

<!ELEMENT updates (update*) >
<!ELEMENT update (patch+) >
<!ATTLIST update
type CDATA #REQUIRED
displayVersion CDATA #REQUIRED
appVersion CDATA #REQUIRED
platformVersion CDATA #REQUIRED
buildID CDATA #REQUIRED
detailsURL CDATA #IMPLIED
licenseURL CDATA #IMPLIED
showPrompt CDATA #IMPLIED
showNeverForVersion CDATA #IMPLIED
actions CDATA #IMPLIED
openURL CDATA #IMPLIED
notificationUrl CDATA #IMPLIED
alertUrl CDATA #IMPLIED
promptWaitTime CDATA #IMPLIED
backgroundInterval CDATA #IMPLIED >
<!ELEMENT patch (#PCDATA)>
<!ATTLIST patch
type CDATA #REQUIRED
URL CDATA #REQUIRED
hashFunction CDATA #REQUIRED
hashValue CDATA #REQUIRED
size CDATA #REQUIRED >

There is a lot going on in here so let's break it down. This DTD describes the elements and attributes I am expecting to find in this type of response from the service.

  • There needs to be a top element updates
  • It has zero or more update elements (the * indicates that)
  • Each update has a bunch of attributes that are required or optional (that's what #IMPLIED refers to)
  • Each update should have one or more patch elements
  • Each patch element has some required attributes

The Python code for a test is pretty simple. Again, substitute your own preferred tools in your own language of choice.

'''
We're using lxml for the XML parsing and validating
url contains the full URL to the API endpoint
'''
repsonse = requests.get(url)
root = etree.XML(response.text)
# Load our DTD file
f = open('./api-tests/updates.dtd')
dtd = etree.DTD(f)

# Validate the response against the DTD and show the errors if it fails
valid = dtd.validate(root)

if valid:
    assert True
else:
    print(dtd.error_log.filter_from_errors())
    assert False

As a rule I try and find libraries and tools that solve problems for me rather than write my own. In this case, lxml and the use of DTD's was the right choice than my initial thoughts of using XPath to hunt for attributes inside the elements.

Hope this gives you a different perspective on strategies for testing API's.