Testing TYPO3’s Core - Part III: Unit Tests

In the first two parts of this series we took a closer look at the automatic testing integrated into the process of developing TYPO’3 core, at how the core team and other core contributors use these tests and also at the hardware and software needed to handle these procedures.

We take a systematic and meticulous approach to building TYPO3 by means of continual testing and revision. Before new code is merged into TYPO3’s core it undergoes a thorough check to ensure that everything performs as intended. Our process of testing consists of four different steps, each of which need to be completed before a patch can be cleared for use: unit testing, functional testing, acceptance testing and integrity testing.

A unit is the smallest testable part of software. It usually has one or a few inputs and a single output. Unit testing is the software developing process in which these smallest individual components are tested individually and independently to determine whether they are fit to use. It is the most common test level when contributing patches to the TYPO3 core and its function is to validate that each unit performs exactly as designed.

Unit tests were the first tests ever added to the TYPO3 core. They’re our so-called “low hanging fruits” as they are relatively easy to write and maintain, if done correctly. Currently, the TYPO3 core is backed up by roughly altogether 9,000 unit tests. The number of unit tests changes constantly in accordance with changes made to the TYPO3 core.

For instance, with the recent removal of deprecated code in the current core master branch (which will evolve to 9.0), several hundred tests were deleted. At the same time, new tests are also constantly being written. For example, with the rewrite of the form engine code since core v7, more than 600 unit tests have been added in this specific code area.

The system has a lot of code that fiddles around with strings and arrays and these parts are perfect for unit testing. Any short method that splits a string in some way, validates the result and does its magic … is simply priceless to us, because code - by definition - has bugs. Unit tests are really useful here as they handle small code components with various input scenarios, help find all the nifty edge cases which need attention, thus leading to a more stable software.

All’s well that ends well and what started as chaos ended up becoming my first and very positive experience with unit testing. That was back in 2010. TYPO3 v4.3 had been introduced shortly beforehand (2009) and the very new and shiny extension “scheduler” had been integrated into the core of this version. The scheduler calculates the date as to when the next run of a scheduled task should begin. When the scheduler is called manually or via cron, it looks at that date and if it is in the past compared to “now”, a task needs execution. A straight and simple approach.

We started using the extension in a bigger project and found ourselves confronted with some rather confusing issues with regards to the date calculation. For instance, a task timed for weekly execution was being executed every minute of the day it was scheduled to do so. Looking into details we found further issues and broken combinations.The first implementation of the “calculate next execution date” was a 50 or 100 lines of code method that called itself recursive in two or three places. Altogether it was complex, hard to follow and full of bugs. And it had no test coverage whatsoever. In short, we needed a plan to fix the mess and to prevent a nerve-wracking and time-consuming issue like this happening again.

We were able to come up with a much better parser that understands all the little details a unix cron command can handle. No recursion anymore. The code was backed up with more than 200 unit tests that even recognized and tested edge cases like summertime / wintertime switches - happy little details that occur in date calculations. Since that patch was merged in 2010, only a bit of coding style adoptions and other cosmetics have been applied to this code area. There has not been a single substantial bugfix till this day. Written once, it works for years. The point is, this would never have been possible without proper unit tests.

Rewriting this code area and backing it up with unit tests was totally worth it. Sufficient test coverage with unit testing put an end to seemingly non-stop bug fixing sessions and fixed things once and for all. It was this special experience that made me a strong believer of unit testing. The TYPO3 core team has made unit testing an integral part of the workflow patterns we use in daily procedures and this has made things easier to handle and our work more enjoyable.  

Unit testing in TYPO3’s core is based on a programmer-oriented testing framework: PHPUnit. The basic configuration file “UnitTests.xml” defines basic phpunit parameters. A pretty short TYPO3 specific bootstrap is referenced there to calculate some base paths and to initialize the class loader. After that, PHPUnit takes over, finds the tests and executes them. It should be noted that as little as possible is our best practice when it comes to bootstrapping: LocalConfiguration.php is not read, there is no database connection and the system does not know which extensions are active. Isolating the unit tests helps avoid unwanted side effects.

Executing all 9,000 core unit tests is relatively quick: A decent machine easily handles them in less than a minute.

However, executing them once per patch set is not enough in Bamboo: TYPO3 v8 currently supports the two major PHP versions 7.0 and 7.1, so we execute all unit tests on both platforms. And yes, while developers sometimes do have the pleasure of experiencing the “green bar feeling” (meaning that all tests results are green), this certainly isn’t always the case. Sometimes the 7.0 tests are green while the 7.1 test turns out to be red - or the other way round. Translated into English this means: go back, fix bugs and rerun the tests till all results are green.

Additionally, the TYPO3 core has a long standing problem with global scope and state that is hidden in static class properties. This makes it difficult to track side effects between single tests. For example, test number 162 may set a state somewhere and not clean up properly, and then test number 6893 relies on that state. It’s always a hassle to track down stuff like that. A mid-range goal of the TYPO3 core team is to rule out these state issues in one way or the other. Until that happens, we’ll continue using a simple workaround to ensure that unit tests are free of such side effects: their order is shuffled during execution.

In the example above, if the test at former position 6893 is randomly executed before the test at former position 162, it would fail. We configured four additional unit test jobs to execute tests in random order. The result is very favourable indeed. Since they’ve been added to our working procedure, we haven’t had any annoying side-effect bugs within the unit tests anymore - at least not in merged code.

Unit tests cover a single unit (e.g. one method). In our next article on testing TYPO3’s core, we'll be focussing on functional tests, which is a quality assurance process whereby the system is tested against the functional requirements and specifications. The functional test environment of the TYPO3 core allows in-depth testing of complex data manipulation scenarios.

In the meantime, you can also find answers to other frequently asked technical questions, instructional clips and all sorts of helpful videos on our TYPO3 YouTube channel. Come along and join us there. If you find this useful, please share the link on your network.