OpenCFP Engineering Diary -- Adding Doctrine

December 2nd, 2019

TL;DR - Adding support to an existing Symfony 3.4 application for Doctrine

(If you like the work I do on OpenCFP, please consider sponsoring me at https://github.com/sponsors/chartjes/)

Part of the medium-term planning for OpenCFP is to refactor the application to stop using Sentinel as the authentication and ACL choice and start using the Symfony Scurity component.

When OpenCFP was first built it was a more standalone solution and I picked Sentinel's predecessor, Sentry. As it got deprecated, we moved to Sentinel and along the way we added in some code to make Eloquent easier to use within the app. When switching to be more Symfony-based, Doctrine is the ORM that needs to be used. So I have a refactoring process that looks like this:

  • Get Doctrine and it's dependencies into the project
  • Refactor everything except our auth stuff to use Doctrine
  • Refactor our authentication and authorization code to use Symfony Security
  • Remove Sentinel and all it's dependencies

I had not used Doctrine in many many years so a lot of things have changed. Given all the trouble I had, I figured I was not alone so it makes sense to document the changes and additions I made.

As with most work I do on OpenCFP, this was a journey of discovering things I needed to know that I did not know.

Packages

Following the directions for Symfony 3.4 I installed my dependencies:

composer require doctrine/orm
composer require doctrine/doctrine-bundle
composer require doctrine/doctrine-cache-bundle

Right now, composer show says I have the following Doctrine-related packages installed now

doctrine/annotations                v1.8.0  
doctrine/cache                      1.9.1   
doctrine/collections                1.6.4   
doctrine/common                     v2.11.0 
doctrine/data-fixtures              1.4.0   
doctrine/dbal                       v2.10.0 
doctrine/doctrine-bundle            1.12.2  
doctrine/doctrine-cache-bundle      1.4.0   
doctrine/doctrine-fixtures-bundle   3.3.0   
doctrine/event-manager              1.1.0   
doctrine/inflector                  1.3.1   
doctrine/instantiator               1.3.0   
doctrine/lexer                      1.2.0   
doctrine/orm                        v2.7.0  
doctrine/persistence                1.2.0   
doctrine/reflection                 v1.0.0  
symfony/doctrine-bridge             v3.4.35 

I may have installed some of these packages manually, but who knows.

Enable Doctrine in the application

The first sign I had no idea what I was doing was when I could not see any of the Doctrine-related commands when I used bin/console. After moaning on Twitter and asking some questions on Stack Overflow, I realized it would not automatically find it. so I added a line to my Kernel.php file so that the bundle would be available.

    public function registerBundles()
    {
        $bundles = [
            new FrameworkBundle(),
            new SensioFrameworkExtraBundle(),
            new MonologBundle(),
            new TwigBundle(),
            new SwiftmailerBundle(),
            new WouterJEloquentBundle(),
            new OneupFlysystemBundle(),
            new DoctrineBundle(),
        ];

        if ($this->getEnvironment() !== Environment::TYPE_PRODUCTION) {
            $bundles[] = new DebugBundle();
        }

        if ($this->getEnvironment() === Environment::TYPE_DEVELOPMENT) {
            $bundles[] = new WebServerBundle();
            $bundles[] = new WebProfilerBundle();
        }

        return $bundles;
    }

Making repositories available as a service

I had to update my `resources/config/services/services.yml' file with the following new section to allow the app to find and inject all the repository files I was going to create:

 OpenCFP\Domain\Repository\:
    public: true
    resource: '%kernel.project_dir%/src/Domain/Repository/*'

Configuration Files

To make Doctrine see the database connections, I had to modify the following files:

resources/config/config_development.yml
resources/config/config_testing.yml
resources/config/config_production.yml

adding in the following details (determined through trial-and-error and online searches for help)

doctrine:
  dbal:
    url: mysql://root:@127.0.0.1:3306/cfp
    default_table_options:
      charset: utm8mb4
      collate: utg8mb4_unicode_ci
  orm:
    auto_mapping: true
    auto_generate_proxy_classes: true
    mappings:
      OpenCFP\Domain\Entity:
        type: annotation
        dir: "%kernel.root_dir%/../src/Domain/Entity"
        prefix: OpenCFP\Domain\Entity

Early on I had my integration test suite failing to even run because it was complaining it could not find the new repository I had created:

<?php

declare(strict_types=1);

/**
 * Copyright (c) 2013-2019 OpenCFP
 *
 * For the full copyright and license information, please view
 * the LICENSE file that was distributed with this source code.
 *
 * @see https://github.com/opencfp/opencfp
 */

namespace OpenCFP\Domain\Repository;

use Doctrine\ORM\EntityManagerInterface;
use Doctrine\ORM\EntityRepository;
use OpenCFP\Domain\Entity\Airport;

final class AirportRepository
{
    /**
     * @var EntityRepository
     */
    private $repository;

    public function __construct(EntityManagerInterface $entityManager)
    {
        $this->repository = $entityManager->getRepository(Airport::class);
    }

    public function withCode(string $code): ?Airport
    {
        $airport = $this->repository->findOneByCode($code);

        if ($airport !== null) {
            return $airport;
        }

        return null;
    }
}

Here's the entity I created to go with it

<?php

declare(strict_types=1);

/**
 * Copyright (c) 2013-2019 OpenCFP
 *
 * For the full copyright and license information, please view
 * the LICENSE file that was distributed with this source code.
 *
 * @see https://github.com/opencfp/opencfp
 */

namespace OpenCFP\Domain\Entity;

use Doctrine\ORM\Mapping as ORM;

/**
 * @ORM\Entity
 * @ORM\Table(name="airports")
 */
final class Airport
{
    /**
     * @ORM\Column(type="string", length=3)
     * @ORM\Id
     */
    private $code;

    /**
     * @ORM\Column(type="string", length=255)
     */
    private $name;

    /**
     * @ORM\Column(type="string", length=255)
     */
    private $country;

    /**
     * @return mixed
     */
    public function getCode()
    {
        return $this->code;
    }

    /**
     * @return mixed
     */
    public function getName()
    {
        return $this->name;
    }

    /**
     * @return mixed
     */
    public function getCountry()
    {
        return $this->country;
    }
}

The issue turned out to be I needed to set auto_generate_proxy_classes to be true. The error messages from the application itself were not helpful (complaining it could not inject the EntityManagerInterface my code was using). Anyway, once I figured that out my integration test suite passed. Now I could go and modify my code to use the new repository instead of the Eloquent model.

Here's a summary of the steps I took to update the action that uses airport details

  • removed references to the AirportInformationDatabase object that was used to access the list of airports and codes
  • added in the use of the new AirportRepository object
  • updated the constructor for the action to accept AirportRepository as a parameter via the magic of autowiring dependencies
  • updated the code to use the new repository (and fix a code smell involving an exception being caught but never reacted to)

So now my tests pass and I have a way to add more Doctrine entities and repositories going forward.

Mocking a function with different return values in Python

May 15th, 2019

For the first half of 2019 I was borrowed by the Performance Team at Mozilla and asked to help them implement a way to measure the performance of various browsers on Android platforms. This has involved me getting familiar with several tools and environments that I had not touched before:

  • Mozilla's tooling for use with mozilla-central
  • Rooting Android devices for testing purposes
  • Android Studio and it's device emulation tools
  • Executing and parsing shell commands

Of course, I am using pytest as well to make sure that I had thought about testing scenarios and giving whomever supports this code once I am back at my previous role a chance to figure out what is going on.

One of the testing scenarios I came up with was handing the fact that I had designed the code to have a fallback mode in case the shell command I was using did not work on that particular device. Some code reviews from other developers revealed some differences in how the top command works on different phones.

So, drawing on my own experiences with using test doubles in PHP, I asked myself "how can I create a mock that has different return values?". In pytest this functionality is called side_effect.

So, here is some of the code in question that I needed to test:

try:
    cpuinfo = raptor.device.shell_output("top -O %CPU -n 1").split("\n")
    raptor.device._verbose = verbose
    for line in cpuinfo:
    data = line.split()
            if data[-1] == app_name:
                cpu_usage = data[3]
except Exception:
    cpuinfo = raptor.device.shell_output("dumpsys cpuinfo | grep %s" % app_name).split("\n")
    for line in cpuinfo:
        data = line.split()
        cpu_usage = data[0].strip('%')

I wrapped the first call in a try-catch construct because an exception is thrown if anything is up with that top call. If that call doesn't work, I then want to use that dumpsys call.

In the test itself, I would need the shell_output call to first throw an exception (as expected for the scenario) and then return some output that I can then parse and use.

In the PHP world, most test doubling tools allow you to create a mock and have it return different values on consecutive calls. Pytest is no different, but it took me a while to figure out the correct search terms to find the functionality I wanted.

So, here is how I did it:

def test_usage_with_fallback():
    with mock.patch('mozdevice.adb.ADBDevice') as device:
        with mock.patch('raptor.raptor.RaptorControlServer') as control_server:
            '''
            First we have to return an error for 'top'
            Then we return what we would get for 'dumpsys'
            '''
            device.shell_output.side_effect = [
                OSError('top failed, move on'),
                ' 34% 14781/org.mozilla.geckoview_example: 26% user + 7.5% kernel'
            ]
            device._verbose = True

            # Create a control server
            control_server.cpu_test = True
            control_server.test_name = 'cpuunittest'
            control_server.device = device
            control_server.app_name = 'org.mozilla.geckoview_example'
            raptor = Raptor('geckoview', 'org.mozilla.geckoview_example', cpu_test=True)
            raptor.device = device
            raptor.config['cpu_test'] = True
            raptor.control_server = control_server

            # Verify the response contains our expected CPU % of 34
            cpuinfo_data = {
                u'type': u'cpu',
                u'test': u'usage_with_fallback',
                u'unit': u'%',
                u'values': {
                    u'browser_cpu_usage': '34'
                }
            }
            cpu.generate_android_cpu_profile(
                raptor,
                "usage_with_fallback")
            control_server.submit_supporting_data.assert_called_once_with(cpuinfo_data)

Let me break down what I did (as always, I am open to suggestions on better ways to write this test).

The first double is for a class that communicates with the Android device. Then the next double I needed was for the "control server", which is what is used to control the browser and execute tests.

My first "side effect" is to generate an error so it triggers the first condition of the scenario that 'top should not work'. The second "side effect" is the response I am expecting to get from the shell command in my fallback area of the code.

Then I continue with the "arrange" part of the Arrange-Act-Assert testing pattern I like to use -- I configure my "control server" to be the way I want it.

I finish up with creating what I expect the data that is to be submitted to our internal systems looks like.

I execute the code I am testing (the "act" part) and then use a spy to make sure the control server would have submitted the data I was expecting to have been generated.

The ability to have a method return different values is powerful in the context of writing tests for code that has conditional behaviour. I hope you find this example useful!

Mocking a function with different return values in Python

May 15th, 2019

For the first half of 2019 I was borrowed by the Performance Team at Mozilla and asked to help them implement a way to measure the performance of various browsers on Android platforms. This has involved me getting familiar with several tools and environments that I had not touched before:

  • Mozilla's tooling for use with mozilla-central
  • Rooting Android devices for testing purposes
  • Android Studio and it's device emulation tools
  • Executing and parsing shell commands

Of course, I am using pytest as well to make sure that I had thought about testing scenarios and giving whomever supports this code once I am back at my previous role a chance to figure out what is going on.

One of the testing scenarios I came up with was handing the fact that I had designed the code to have a fallback mode in case the shell command I was using did not work on that particular device. Some code reviews from other developers revealed some differences in how the top command works on different phones.

So, drawing on my own experiences with using test doubles in PHP, I asked myself "how can I create a mock that has different return values?". In pytest this functionality is called side_effect.

So, here is some of the code in question that I needed to test:

try:
    cpuinfo = raptor.device.shell_output("top -O %CPU -n 1").split("\n")
    raptor.device._verbose = verbose
    for line in cpuinfo:
    data = line.split()
            if data[-1] == app_name:
                cpu_usage = data[3]
except Exception:
    cpuinfo = raptor.device.shell_output("dumpsys cpuinfo | grep %s" % app_name).split("\n")
    for line in cpuinfo:
        data = line.split()
        cpu_usage = data[0].strip('%')

I wrapped the first call in a try-catch construct because an exception is thrown if anything is up with that top call. If that call doesn't work, I then want to use that dumpsys call.

In the test itself, I would need the shell_output call to first throw an exception (as expected for the scenario) and then return some output that I can then parse and use.

In the PHP world, most test doubling tools allow you to create a mock and have it return different values on consecutive calls. Pytest is no different, but it took me a while to figure out the correct search terms to find the functionality I wanted.

So, here is how I did it:

def test_usage_with_fallback():
    with mock.patch('mozdevice.adb.ADBDevice') as device:
        with mock.patch('raptor.raptor.RaptorControlServer') as control_server:
            '''
            First we have to return an error for 'top'
            Then we return what we would get for 'dumpsys'
            '''
            device.shell_output.side_effect = [
                OSError('top failed, move on'),
                ' 34% 14781/org.mozilla.geckoview_example: 26% user + 7.5% kernel'
            ]
            device._verbose = True

            # Create a control server
            control_server.cpu_test = True
            control_server.test_name = 'cpuunittest'
            control_server.device = device
            control_server.app_name = 'org.mozilla.geckoview_example'
            raptor = Raptor('geckoview', 'org.mozilla.geckoview_example', cpu_test=True)
            raptor.device = device
            raptor.config['cpu_test'] = True
            raptor.control_server = control_server

            # Verify the response contains our expected CPU % of 34
            cpuinfo_data = {
                u'type': u'cpu',
                u'test': u'usage_with_fallback',
                u'unit': u'%',
                u'values': {
                    u'browser_cpu_usage': '34'
                }
            }
            cpu.generate_android_cpu_profile(
                raptor,
                "usage_with_fallback")
            control_server.submit_supporting_data.assert_called_once_with(cpuinfo_data)

Let me break down what I did (as always, I am open to suggestions on better ways to write this test).

The first double is for a class that communicates with the Android device. Then the next double I needed was for the "control server", which is what is used to control the browser and execute tests.

My first "side effect" is to generate an error so it triggers the first condition of the scenario that 'top should not work'. The second "side effect" is the response I am expecting to get from the shell command in my fallback area of the code.

Then I continue with the "arrange" part of the Arrange-Act-Assert testing pattern I like to use -- I configure my "control server" to be the way I want it.

I finish up with creating what I expect the data that is to be submitted to our internal systems looks like.

I execute the code I am testing (the "act" part) and then use a spy to make sure the control server would have submitted the data I was expecting to have been generated.

The ability to have a method return different values is powerful in the context of writing tests for code that has conditional behaviour. I hope you find this example useful!