As explained in the previous guide, Matomo (formerly Piwik)'s test suite contains PHP tests and UI tests. The PHP test suite is written and run using PHPUnit.
If you're creating a new plugin, you may find it beneficial to engage in Test Driven Development or at least to verify your code is correct with tests. With tests, you'll be able to ensure that your code works and you'll be able to ensure the changes you make don't cause regressions.
This can be sometimes hard to decide and often leads to discussions. We consider a test as a unit test when it tests only a single method or class. Sometimes two or three classes can still be considered as a Unit for instance if you have to pass a dummy class or something similar but it should actually only test one class or method. If it has a dependency to the filesystem, web, config, database or to other plugins it is not a unit test but an integration test. If the test is slow it is most likely not a unit test but an integration test as well. "Slow" is of course very subjective and also depends on the server but if your test does not have any dependencies your test will be really fast.
It is an integration test if you have any dependency to a loaded plugin, to the filesystem, web, config, database or something similar. It is an integration test if you test multiple classes in one test.
It is a system test if you - for instance - make a call to Matomo itself via HTTP or CLI and the whole system is being tested.
Because they fail for different reasons and the duration of the test execution is different. This allows us to execute all unit tests and get a result very quick. Unit tests should not fail on different systems and just run everywhere for example no matter whether you are using NFS or not. Once the unit tests are green one would usually execute all integration tests to see whether the next stage works. They take a bit longer as they have dependencies to the database and filesystem. The system and ui tests take the most time to run as they always run through the whole code.
Another advantage of running the tests separately is that we are getting a more accurate code coverage. For instance when running the unit tests we will get the true code coverage as they always only test one class or method. Integration tests usually run through a lot of code but often actually only one method is supposed to be tested. Although many methods are not tested they would be still marked as tested when running integration tests.
Before you start make sure you have setup Matomo, and make sure you have enabled development mode:
$ ./console development:enable
To install PHPUnit, run below command in the Matomo root directory (depending on how you installed Composer you may not need the php command):
php composer.phar install
If your development Matomo is not using localhost as a hostname (or if your webserver is using a custom port number), then edit your config/config.ini.php file and under [tests] section, add the http_host and/or port settings:
[tests]
http_host = localhost
port = 8777
The request_uri needs to be configured for running tests. If your development Matomo is setup in a sub-directory for example at http://localhost/dev/matomo, then your settings should be like this:
[tests]
request_uri = "/dev/matomo"
If you don't use any sub-directory, you can simply set it up like this:
[tests]
request_uri = "/"
Before you run the tests (at least the first time, but you can rerun it any time), run this command to migrate the test database.
$ ./console tests:setup-fixture OmniFixture
A unit test tests only a single method or class and does not use dependencies like the filesystem, web, config, database or any other plugin.
To create a new unit test, use the console:
$ ./console generate:test --testtype unit
The command will ask you for the name of the plugin and the name of the test (which is usually the name of the class you want to test). It will create a file plugins/MyPlugin/tests/Unit/WidgetsTest.php which contains an example to get you started:
/**
* @group MyPlugin
* @group WidgetsTest
* @group Plugins
*/
class WidgetsTest extends UnitTestCase
{
public function testSimpleAddition()
{
$this->assertEquals(2, 1+1);
}
}
We don’t want to cover how you should write your unit test. This is totally up to you. If you have no experience writing unit tests we recommend reading articles or a book on the topic, watching videos or anything else that will help you learn best.
If your test needs access to a test Matomo database, filesystem, or any other dependency — create an integration test:
$ ./console generate:test --testtype integration
The command will as well ask you for the name of the plugin and the name of the test. It will create a file plugins/MyPlugin/tests/Integration/WidgetsTest.php which contains an example to get you started.
The IntegrationTestCase base class provides a setUp() method that creates a test Matomo database and a tearDown() method that removes it. During integration
tests all plugins will be loaded allowing you to write actual integration tests.
The IntegrationTestCase base class also provides two extra methods that can be overridden:
beforeTableDataCached(): IntegrationTestCase will initialize a fixture before running any test. The fixture could add entities, track visits, archive them or affect the database in another way. As an optimization, IntegrationTestCase will cache all this data in memory after the fixture is set up, and simply re-insert all of it into the database tables in the test's setUp() method. This allows us to have a clean slate for each test, without having to destroy the database and re-run the fixture every single time (which would be much slower than just restoring the table state). Some tests, however, add data outside of the fixture in the setUp() method. This can slow the test down and by extension the already long running builds on travis-ci.com. If that data is meant to be static for each test, it can be added in an overridden beforeTableDataCached() method. Then it will be cached along with fixture data.
Some examples of things you can add here are: adding many websites via SitesManager API or adding many users via the UsersManager API. Using the API is far slower than just inserting the data back into truncated tables.configureFixture(): to handle the creation/destruction of the Matomo environment, integration tests use a blank fixture (it can be overwritten just like in system tests to bootstrap a test with some data). The configuration for this fixture may not be ideal for every test, so this method can be used to configure the fixture differently. For example, by default we don't create a real superuser in the system (as in, adding a row to the user table and everything else), and we don't load any translations. Some tests may not want this.There are some common issues that can occur when writing integration tests. These are listed below with their appropriate solution:
protected static function configureFixture($fixture)
{
parent::configureFixture($fixture);
$fixture->extraTestEnvVars['loadRealTranslations'] = true;
}protected static function configureFixture($fixture)
{
parent::configureFixture($fixture);
$fixture->createSuperUser = true;
}Whenever possible, you should be using dependency injection to change behaviour in tests. For example if you don't want to fetch data from a remote service but instead use a local fixture then you could create a file in plugins/MyPluginName/config/test.php where you return a regular array and overwrite any DI dependency.
If DI is not possible or not trivial to use, then you can check if tests are currently being executed by using $isTestMode = defined('PIWIK_TEST_MODE') && PIWIK_TEST_MODE;.
To run a test, use the command tests:run which allows you to execute a test suite, a specific file, all files within a folder or a group of tests.
To verify whether the created test works we will run it as follows:
$ ./console tests:run WidgetsTest
This will run all tests having the group WidgetsTest. As other tests can use the same group you might want to pass the path to your test file instead:
$ ./console tests:run plugins/Insights/tests/Unit/Widgets.php
If you want to run all tests within your plugin pass the name of your plugin as an argument:
$ ./console tests:run insights
Of course, you can also define multiple arguments:
$ ./console tests:run insights WidgetsTest
This will execute all tests within the insights plugin having the group WidgetsTest. If you only want to run unit tests within your plugin you can do the following:
$ ./console tests:run insights unit
To run all unit, integration or system tests, use one of the following arguments:
$ ./console tests:run unit
$ ./console tests:run integration
$ ./console tests:run system
While developing or debugging tests, there isn't the need to always execute all tests in a file or group. Instead, you can execute only a specific test or group of tests by adding the filter option.
$ ./console tests:run path/file.php --filter="test_mymethod"
This will only run the test cases that start with the sepcific method name test_mymethod. This will make troubleshooting this test a lot faster as you don't need to wait until all other test cases finish.
Most unit and integration tests in Matomo test a single class, or at most a matomo subsystem. One test, however, is special in that they don't test Matomo behavior, but instead tests that Matomo is ready to be released. This test is called ReleaseCheckListTest and performs the following types of tests:
Plugins sometimes define their own version of this test.
assertSame over assertEquals so it does an exact comparision (including type)$this->assertSame(1, count($array)) use $this->assertCount(1, $array)Instead of for example
$this->assertEquals( 1, count( $missingTables ) );
$this->assertEquals( 'foobar', $missingTables[0] );
It is much better to use $this->assertSame( [ 'foobar' ], $missingTables );.
This way you will only need one assertEquals and the $this->assertEquals( 1, count( $missingTables ) ); can be removed. More importantly, when there is a test failure, it will show you the entire output/difference of $missingTables vs with the current implementation you would only see something like expected 1, actual 2 which isn't really helpful to know what went wrong. With the suggested assert it will tell you exactly what went wrong and by comparing the entire variable you always make sure there isn't anything that was forgotten.
In an ideal world each test case has only one assert or only tests one specific case. Instead of for example having:
public function test_multiply() {
$this->assertSame( false, $this->report->multiply(0,false) );
$this->assertSame( 1, $this->report->multiply(1,1) );
}
split them into two different methods:
public function test_multiply_shouldReturnFalse_whenOneInputIsNotNumeric() {
$this->assertSame( false, $this->report->multiply(0,false) );
}
public function test_multiply_shouldReturnTheResult_whenTwoNumbersAreGiven() {
$this->assertSame( 1, $this->report->multiply(1,1) );
}
This way the test output will be more verbose when a test case fails and it will be more clear what the case is trying to test.
We shouldn't catch any unexpected exception as otherwise tests would succeed without us noticing when they start failing. Instead we can simply remove the try/catch block. When there is any exception in the future, the test will fail (which is good) and we will get the exception message reported by PHPUnit.
public function test_multiply() {
try {
$this->assertSame( false, $this->report->multiply(0,false) );
} catch (Exception $e) {
Log::log('test failed: ' . $e->getMessage());
}
}
Don't just test the expected way a method might be used. Also pass unexpected values such as null etc.
Say you have an assertion like this: $this->assertNotContains( "_paq.push(['requireCookieConsent']);", $trackingCode );.
Then it can be better to instead simply use $this->assertNotContains( 'requireCookieConsent', $trackingCode );.
Why?
_paq.push([ 'requireCookieConsent']);) then the test would still pass even though it contains the requireCookieConsent call._paq.push(["requireCookieConsent"]);) then the test still works correctly and would still correctly detect any failure if for some reason there's a bug and the requireCookieConsent call is suddenly present in $trackingCode.Note this does not apply to eg assertContains where you want to be specific to make sure we get the expected result. There you would want to write $this->assertContains( "_paq.push(['requireCookieConsent']);", $trackingCode );
Instead of writing a test like this:
public function test_me() {
$this->assertSame(1, $this->square(1));
$this->assertSame(4, $this->square(2));
$this->assertSame(9, $this->square(3));
}
Use a data provider like this:
/**
* @dataProvider getMyDataProvider
*/
public function test_me($expected, $number) {
$this->assertSame($expected, $this->square($number));
}
public function getMyDataProvider()
{
yield 'when number is 1 it should return 1' => [1, 1];
yield 'when number is 2 it should return 4' => [4, 2];
yield 'when number is 3 it should return 9' => [9, 3];
}
This ensures that the test execution won't stop in the middle if one of them fails and makes the test code more readable.
Locate the directory of the processed and expected tests directory for the test you are executing. For example plugins/YourPluginName/tests/System/ or tests/PHPUnit/System/.
You can then compare the two directories for changes. If you are using PHPStorm, then simply select both processed and expected directories and then right click and select Compare Directories. There you can see the changes for each file and update any processed file if needed. If you don't use PHPStorm, then check if your IDE offers a similar feature or use a linux command like diff processed expected.
Once you have updated all expected files, then you need to git add and git commit and git push these changes.
System PHP tests in Matomo typically execute an API method and compare the entire XML output of the API method with an expected XML output.
If you are making changes to Matomo then the result of such an API method may change and break the build. This is an opportunity to review your code and as a Matomo developer you should ensure that any change in the output is actually expected.
If they are not expected, determine the cause of the change and fix it in a new commit. If the changes are expected, then you should update the expected system files accordingly. To compare and update the expected system files, follow these steps:
#45511, then 45511 is the build number.{buildnumber} with the actual build number. ./console development:sync-system-test-processed {buildnumber}.
--expected. You then need to make sure before committing and pushing these changes that every change is actually expected.processed directory. For example tests/PHPUnit/System/processed and plugins/Goals/tests/System/processed. If you are using PHPStorm you can then select both the processed and expected directory and then right click -> Compare Directories. This allows you to review every change of added, changed and removed files and let's you update each expected file individually.git add and git commit and git push the changes to trigger another build run.As a software developer writing tests it can be useful to be able to set breakpoints and debug while running tests. If you use Phpstorm read this answer to learn to configure Phpstorm with the PHPUnit from Composer.