Testing TYPO3’s Core - Part VI: Integrity Tests

To ensure that TYPO3 operates in coordination with all other systems in the same environment, specific tests are applied. In our previous posts, we’ve taken a closer look at unit, functional and acceptance testing and now we’ll take a closer look at the fourth and final hurdle in our process of testing TYPO3’s core: integrity tests (=integration tests, system integration test).

Integrity tests are used to check the system as a whole, with all of its subsystem components and elements. Communication paths between different parts of the system are tested to verify the integrity of the system on different levels and to show that all modules work correctly together. They test the depth and degree of integration and give answer to the question “Does it work for all edges cases and corner cases?” Integrity tests also check layers of an application working together and can be used when looking for bugs caused by two or more parts of the system working in unison. So obviously they’re an essential part of TYPO3’s quality assurance process.

Lately, the TYPO3 core team has put quite some effort into refactoring the JavaScript side of the system. Outdated frameworks have been dropped as well as lots of JavaScript being split into pieces.

This is a major step forward and now finally allows unit testing of JavaScript. The core team also introduced the test runner Karma as a developer dependency and we are now also using Jasmine as a framework for testing JavaScript code. Jasmine doesn’t depend on any other JavaScript frameworks and doesn’t require a DOM. It has a clean syntax so tests are easily written. Up until now, not that many tests have been written, but this area is evolving and we’re looking forward to more progress and test coverage.

As hundreds of contributors work on the codebase, establishing a common code style that is followed by all contributors is essential. However, fighting for style details can be time consuming and can lead to off-topic discussions that don’t handle the technical changes anymore, but only cosmetical stuff instead.

The TYPO3 core team put an end to these discussions when TYPO3 CMS 7.6 LTS was released. The decision was made to stick to the generally accepted coding guidelines defined in PHP Standards Recommendations with PSR-2.
A cool advantage of this decision is that PSR-2 compatibility can be automatically tested with the PHP-CS-fixer project. And even better, this project allows fixing any coding style violation directly.

This makes sticking to coding guidelines a cakewalk in TYPO3!

If a Bamboo test plan fails due to a coding guideline violation, a developer just executes the fixer script “cglFixMyCommit.sh” and pushes a new version of the patch to “make the test plan happy”. Done. No more fighting about style details and no more time spent manually fixing this by hand.

There are a couple of other tests executed for each patch. Most of them were integrated after experiencing one error or another while doing our work.

Fail - Improve - Succeed - Repeat: the developer’s workflow.

Let’s have a look at some of the lessons we learnt along the way and at how they led to developing additional tests:

• Exception codes
  Each exception thrown by the TYPO3 core has to have a unique identifier (=UID). A link to the official TYPO3 Content Management System wiki is created from it, which allows the community to document additional information like “What raises this exception and how to fix it?” in the wiki (example).
  This system only works if all exceptions actually have an identifier that is unique. Violations to this rule easily happen when using copy and paste. Then a small shell script finds all exceptions in the core and verifies that there’s an identifier and it’s really unique within the project.
  There is a similar test for the localization files in place: The translation system relies on a similar unique identifier restriction.

• ReST file integrity
  Since core version 7, all changes within the core that add features, deprecate or break functionality must come along with a text file explaining the change and how extension authors should handle it.
  The file format is based on ReST. A small helper on forger helps with writing those files. Here too, we have some rules to follow. Each ReST file must reference the according issue number and each file must be tagged with some keyword to specify which parts of the system are affected. Another script in the core checks that those rules are followed. Bamboo executes this script for each patch.

• PHP linting
  PHP has a linter to test if a source file is syntactically correct. The TYPO3 core supports multiple PHP versions in its different major releases. It sometimes happens that a patch is back ported from a core branch that uses a PHP language construct from a younger PHP version which does not exist in older PHP versions. For instance, TYPO3 version 8 requires PHP 7 and above, while version 7 also supports PHP 5.5 and 5.6. The PHP "strict types" feature introduced with PHP 7.0 is actively used in TYPO3 version 8, but can not be used in version 7, that would break the PHP 5.5 and 5.6 support. Such patches need to be adapted when back porting changes. The PHP linter for PHP version 5.5 would find such a violation. Bamboo jobs therefore test patches to see if such a violation slipped through. This gives us a safety net especially important when backporting changes to older TYPO3 core branches. To prevent issues like this, a Bamboo test checks each and every PHP file with all supported PHP versions for such syntactical errors.

• Permissions and git submodules
  Two further simple test scripts check some other parts. For instance, first we need to make sure that the command line entry point to TYPO3 is executable, other file permissions are ok, and we also check that no change is pushed that integrates a git submodule. This actually happened by accident a short while ago and raised quite some issues during deployment. So we now have a happy little test script in place preventing this from happening again.

• Sass linting
  The TYPO3 backend CSS is created from Sass files. We recently added a bamboo job to verify all Sass files are syntactically correct.

The TYPO3 core testing infrastructure not only checks the usual testing areas but is also flexible enough to integrate tests for whatever tasks that arise. We’ve established various scripts to verify that mistakes made in the past will not be repeated and we’ll continue doing so, if and when necessary. This ensures that overall system integrity is given at any point in time.

The TYPO3 core team has had to jump through quite a few loops to achieve all of this. In our next blog post (number six of altogether eight!), we’ll be taking a look at how this was created over time and where it helped us to improve the overall architecture of the system.

In the meantime, you can also find answers to other frequently asked technical questions, instructional clips and all sorts of helpful videos on our TYPO3 YouTube channel. Come along and join us there. If you find this useful, please share the link on your network.