At Dropbox, we made the switch from testing with unittest to pytest. We love the features, fixtures, plugins, and customizability of pytest. To further improve our experience, we built a couple of tools ( pytest-flakefinder, unittest2pytest) for working with pytest and released them as open source.
We developed the pytest-flakefinder plugin to help with a common problem, flaky tests. Tests that involve multiple threads, or that depend on certain ordering can often fail at a fairly low rate. A few flaky tests aren’t a big deal, but with thousands of tests, they become a huge issue. We used to literally run pytest in a loop or sometimes just copy paste the test code multiple times to see if we could reproduce the failure. With this plugin we can easily have pytest run the test multiple times in a row. And if you combine it with -x --pdb you can just run it until it fails and get yourself a debugger ready to find out what happened. We now run flakefinder on updated tests in CI to detect flakiness proactively.
Another one of the key pytest features we love is assertion rewriting. This pytest feature allows us to see significantly more detailed error messages when asserts fire. In order to take advantage of assert rewriting, the tests must use a raw assert a == b rather than the unittest library’s idiomatic self.assertEqual(a, b).
Test output using unittest style assertions:
test/test_login.py:80: in test
self.assertEquals(login.call_count, 1)
E AssertionError: 0 != 1
assert login.call_count == 1pytest output with raw Python asserts:
test/test_login.py:80: in test
E AssertionError: assert 0 == 1
E + where 0 = <MagicMock name='mock.desktop_login.login' id='140671857679512'>.call_count
When we made the switch to pytest, we had thousands of existing tests generally using the latter form. Fortunately, pytest is compatible with unittest asserts, so our test suite still passed. However, we weren’t getting the benefits of assertion rewriting everywhere. On top of that, we had different testing practices in our codebase, leading to some confusion.
We developed unittest2pytest to convert our existing unittest asserts to pytest rewrite-compatible raw asserts. It’s built on top of the lib2to3 library for automatic code rewriting. This library was able to safely convert most of our code automatically. There were a few hiccups with certain kinds of whitespace and inline commenting, which you can see in our issue reporter. unittest2pytest simply skips over converting things it doesn’t understand.
At Dropbox, we developed pytest-flakefinder and unittest2pytest to improve our experience with pytest. These tools are both open source now, so check them out if you use pytest or are considering switching.
