Serious Software Testing: TYPO3 Runs Its 20,000th Build!

Software testing is an integral part of software development. It is one of the most crucial parts of business success, but can also be the most commonly disregarded and undervalued. In its broadest sense it refers to a planned collection of tests and evaluations that identify the accuracy, completeness, quality and security of the software.

The community of open source software TYPO3 takes an incremental, meticulous–and possibly unique–approach to software testing, and with good reason.

Tests are executed to determine the success of a build and ensure that what we’ve created does what it’s supposed to do (and that it doesn’t do what it’s not supposed to do!). Software testing is important to us because if we’re putting software out there, it’s our responsibility to make sure it’s something we can be proud of and that you love to use.

While software testing may seem like an obvious idea, there is only a clear added value if tests are not only in place, but are also performed regularly. Software testing needs to be integrated as an ongoing process in the everyday development of any software right from the beginning. Proper testing includes several different kinds of tests, workflow planning, a systematic review and above all: consistent processes.

Consistency is the key to a successful, long-term and high-quality solution.

If tests are abandoned, software will break somewhere further along the line. On the other hand, if software testing is performed properly, the results allow one to draw general conclusions about the performance and quality of a program. If tests are done frequently they result in added value and a better product.

Over the years, the TYPO3 core team and contributors established a well working test environment that include unit tests, functional tests, acceptance tests and integration tests. Automatic testing procedures are now integrated as a regular and ongoing element in the everyday development process of TYPO3 CMS.

Since mid 2016, a fully automated testing process with Atlassian Bamboo has been used prior to making changes to the TYPO3 core master. Every change that’s made to the TYPO3 core runs through the complete process before getting merged into the system. Our Bamboo system is automated, yet we’re constantly improving and extending the test setup.

Today, we’re proud as punch to announce that the TYPO3 core team ran the 20,000th full test run, end to end, a couple of days ago! Each of these tests have approximately 30,000 single tests per run. So we’re speaking of more than 600 million test executions in less than two years. A special thank you goes to the Leibniz-Rechenzentrum for providing us with computing hardware to perform this excessive number of tests.

• locating defects and making sure they get fixed

• defining what the system is capable of doing

• ensuring that a change does not break something that has been established beforehand

• ensuring that a fixed bug does not pop up again later

• thus reducing the risk that a user is impacted in a negative manner when using the software

Consistent testing offers a genuine added value to a software product. It’s a key for high-quality software development. 

We published a detailed series on the multiple stages of testing and reviewing that changes to TYPO3´s core have to pass before being merged into the system. You’ll find in-depth information in these eight articles:

• Testing TYPO3's Core - Part I: Infrastructure

• Testing TYPO3’s Core - Part II: On Hardware & Dockers

• Testing TYPO3’s Core - Part III: Unit Tests

• Testing TYPO3’s Core - Part IV: Functional Tests

• Testing TYPO3’s Core - Part V: Acceptance Tests

• Testing TYPO3’s Core - Part VI: Integrity Tests

• Testing TYPO3's Core - Part VII: A Historical View

• Testing TYPO3’s Core - Part VIII: Future

The effective performance of any software application or product relies on proven, efficient and consistent testing process. The main idea behind software testing is to locate defects and make sure that they get fixed, thus adding value to the software.

The testing framework in TYPO3 sets a high standard in the software landscape and is constantly being improved and extended. This makes TYPO3 CMS a technical advanced system and a strong choice for enterprises when it comes to website development.

Do you have questions? Or would you like our help in finding an official, certified TYPO3 partner, get in touch!