Testing TYPO3’s Core - Part IV: Functional Tests

This post is the fourth of a series we’re publishing about the workflow patterns we use for testing TYPO3’s core and the multiple stages of testing that changes, i.e. patches, to TYPO3’s core have to pass before being merged into the system. So far, we’ve taken a closer look at TYPO3’s review process in general, the hardware and software this work requires and the first step in the review process: unit testing.

In today’s blog post we’ll be taking a closer look at the second step in TYPO3’s review process: functional testing. So, grab yourself a cup of tea (or coffee) and make yourself comfortable, as we now dig into the details of how functional testing actually works.

With recent major TYPO3 core versions, the team worked hard on some in-depth structural changes to the system. We improved data structures of the language handling and worked on critical parts of the workspace setup. This touched central parts of the system and if done wrongly, bugs can easily lead to complete chaos in the database. So we pay particular attention to each and every detail when working in this area, so as not mess things up. And luckily, so far, we’ve been able to avoid putting TYPO3’s core in jeopardy and have managed to keep things safe and sound at all times (we hope!).

Functional testing is a quality assurance (QA) process. It’s a process whereby software is tested to confirm that the functionality of an application or system is behaving according to specifications and that it conforms with all requirements. The test give answers to the questions “Did we build a correctly working product?” and “Does the software meet the business requirements?”.

Functions are tested by feeding them input and examining the output. While unit tests are about testing single units, functional tests execute multiple components at once and look at the state of the whole system and are thus much more complex. They should cover all the possible paths through a given scenario, including failure paths and boundary cases. Encapsulated and controlled environments are required for this process. The test runner sets up a single TYPO3 instance for each test - including its own database - to execute this single test scenario.

The functional test environment of TYPO3’s core allows in-depth testing of complex data scenarios.

For example:

• Put some example records into table “tt_content”.

• Set up some language specifications.

• Call the DataHandler to localize tt_content records.

• Compare the result in the database with a reference list.

• Set up a frontend and test the localized records are rendered as expected.

There are tons of situations like this in the TYPO3 core and the functional test coverage is pretty impressive, especially when it comes to the central DataHandler codebase. Our functional testing is automated and there are many good reasons to do so: it’s fast, it’s repeatable and it avoids human errors. For this to work safely and soundly, we designed a framework with a rich set of components.

The TYPO3 core framework can deal with many different data relations. One functional test scenario (a so-called “test case”) can provide multiple single tests executed on a specific set of core and fixture extensions.

Single tests need to be encapsulated:

• Each test needs its own controlled environment that is not polluted by an existing instance or other tests.

• Each test needs its own database that is created with all database tables defined by the core and optionally given fixture extensions.

• Single tests of a series of tests (a test case) need a clean (database) state, even if a previous tests failed.

With each different scenario, the testing framework basically sets up a full blown TYPO3 instance per test, to fully encapsulate one from the other. Each instance is separated from all others and has its own database that is cleaned in between single tests.

The basic bootstrap of a functional test case roughly goes through the following steps:

• Create a unique path for each test case in typo3temp and create a basic filesystem layout.

• Link the core and requested extensions into this instance.

• Create a database with a unique name per test case.

• Initialize this database with all tables specified by given list of extensions.

• Create a LocalConfiguration.php and PackageStates.php file so the instance can boot.

• Finally, execute a single test which then can add rows in the database, do some operation and test for expected results.

Currently we have approximately 1,100 single functional tests which split up into about 110 single test cases. Running all of these functional tests means that 110 standalone instances have to be created. With maybe 50 tables needed for a working instance, this easily multiplies to several thousands of database tables and tens of thousands of columns that need to be created for a full run.

This is the big bummer with functional tests: database engines tend to be very conservative when it comes to table creation. Their job is to make very sure integrity is given at any point in time. And they make double sure a table definition has been fully written onto the hard disk before allowing any further writing operation whatsoever. This consumes a lot of execution time making table creation an expensive issue.

Additionally, as mentioned before, TYPO3 is currently not a safe deal when it comes to global state. When a request has been finished, the global PHP state is polluted and can’t be reused for another run. This compiles to a more and more pressing problem and the TYPO3 core is aiming at solving this in the near future. The current solution for functional tests is to fork a single fresh process for each test so as not to pollute the parent environment. This, too, is expensive. And even worse, if a frontend request has to be done for a test, the system is constructed in a way that it creates yet another PHP sub-process. All that is done to separate the environments from each other and this comes with all sorts of drawbacks. It’s slow, additionally debugging in such an environment is a true test to our patience. Oh dear!

With all of this being said, functional test are vital for ensuring that TYPO3 behaves according to specifications and that it conforms with all requirements. So we meet these challenges with our heads held high and also have a few tricks up our sleeves.

9,000 unit tests can be executed in less than a single minute. Executing all functional tests can easily take an hour or even longer. Slow table creation and other details can render functional test execution to be insanely slow. What to do, when patience is being tested like that? Well, we use two strategies to speed things up:

1. The docker containers on Bamboo are RAM driven, so all database “disk” operations are in fact RAM operations. This alone speeds things up to being approximately ten times as fast. While this area does still need proper official documentation, it is relatively easy to run on local systems, too.

2. Another strategy is splitting the functional tests into pieces in Bamboo and executing them as a subset. The single parts are given to different agents simultaneously. At the moment, there are 10 jobs in Bamboo and ideally, each job takes care of about 110 tests. Splitting is done by a straight unix shell script “splitFunctionalTests.sh” located in the core’s “Build” directory. The script counts the number of single tests per test case and tries to balance them. In the end, each job ends up with roughly the same number of tests.

As result, our currently 1,100 functional jobs are pushed through the test plan in just a couple of minutes.

Ok, so we split the functional tests into 10 pieces and have 10 distinct jobs that take care of a subset. Cool. But that only covers a single database system. The functional jobs are created to test all those nasty little database operations and their results. Since switching to doctrine in TYPO3 v8, we support and test multiple different database platforms at the same time:

While developing TYPO3 v8, we always had the functional tests active on MySQL. Every patch had to be green before being accepted. Later, we took a indepth look at our functional tests, this time not only being executed on MySQL DBMS, but also on postgreSQL. Our first test runs with postgreSQL were rather confounding, as almost 800 of altogether 1,100 functional tests failed in their first reviews! In the meantime, we’ve reduced this intimidatingly large number to zero. The postgreSQL based functional tests have been added to the mainline tests and are now required to be green for each patch, too.

Since being integrated the base functionals have signaled “green” every single time and the backend as well as the frontend work on this DBMS without much additional trouble. There have been only a couple of smaller glitches with this platform that were not covered by functional tests.

Note: A well configured postgreSQL is quick and it’s a lot of fun to run TYPO3 on it. Give it a shot!

With regards to the testing, this doubles the amount of jobs: all functional tests are run with MySQL as well as with postgreSQL. This sums up to a total of 30 functional jobs at the moment.

The TYPO3 testing infrastructure supports running tests on MySQL. This daemon is taken care of within the Bamboo docker container agents. We additionally want to run all the tests on postgreSQL, therefore this has been added to each container too. Additionally, some parts of TYPO3 - the caching framework and the locking framework - can put their data into NoSQL databases. These areas are covered by functional tests, too. The docker containers provide Redis, a memcache daemon and other stuff to verify these core parts act as expected.

When it comes to DBMS, the core team currently strives for full Microsoft SQL Server (MSSQL) support. Microsoft is becoming cooler and MSSQL is a viable platform from an open source project point of view: Today, a port of MSSQL that runs natively on linux exists. Seriously! Integrating it into our docker containers was achieved within a day. At the time of this writing, the majority of functional tests works on this platform. We’re looking forward to fixing last things here and to experiencing that “green bar feeling” (meaning that all tests results are green). As soon as functionals are green with MSSQL, TYPO3 core should run without issues on this platform. And we aim to keep it that way. When that is in place, 30 functional jobs have to be executed via Bamboo for each and every patch set. And when we’re done with that, we’ll have a look at the next point on our to-do list, namely SQLite ...

The functional test infrastructure of TYPO3’s core ensures that central (database driven) parts of the system are in a good shape and major parts of the system are not destroyed by any given patch. The effort of creating and maintaining the tests as well as the infrastructure in this area is significant, but definitely worth it. It verifies the stability the framework is known for, which is improving patch by patch.

Unit and functional tests are in-depth testing scenarios. The application layer - the browser - is still missing. We’ll have a look at this in the next part of this series (of altogether eight!).

In the meantime, you can also find answers to other frequently asked technical questions, instructional clips and all sorts of helpful videos on our TYPO3 YouTube channel. Come along and join us there. If you find this useful, please share the link on your network.