System tests files are in tests/PHPUnit/System/*Test.php
System tests allow to test how major Matomo components interact together. A test will typically generate hits to the Tracker (record visits and page views) and then test all API responses and for each API output. It then checks that they match expected XML (or CSV, json, etc.). If a test fails, you can compare the processed/ and expected/ directories in a graphical text compare tool, such as WinMerge on Win, or MELD on Linux, or even with PhpStorm, to easily view changes between files.
For example using Meld, click on "Start new comparison", "Directory comparison", in "Original" select "path/to/matomo/tests/PHPUnit/System/expected" in "Mine" select "path/to/matomo/tests/PHPUnit/System/processed"
If changes are expected due to the code changes you make, simply copy the file from processed/ to expected/, and test will then pass. Copying files is done easily using Meld (ALT+LEFT). Otherwise, if you didn't expect to modify the API outputs, it might be that your changes are breaking some features unexpectedly.
System tests use Fixtures to generate controlled web usage data (visits, goals, pageviews, events, site searches, content tracking, custom variables, etc.).
Fixtures are stored in tests/PHPUnit/Fixtures
We also have an OmniFixture that includes all other Fixtures. OmniFixture is used for screenshot tests to provide data across most reports.
Remember to update the Omnifixture SQL dump whenever you make any change to any fixture. You can use:
./console tests:setup-fixture OmniFixture --sqldump=OmniFixture-dump.sql
Keeping the OmniFixture up to date makes it easier to see which tests fail after each small fixture change.
If we don't update the OmniFixture then we end up with many failed screenshots tests which makes it hard to see whether those changes are expected or not.
As part of our system tests we generate the scheduled reports (in HTML, PDF & SMS).
Some of these scheduled reports contain PNG graphs. Depending on the system under test, generated images can differ.
Therefore, PNG graphs are only tested and compared against "expected" graphs, if the system under test has the same characteristics as the integration server.
The characteristics of the integration server are described in SystemTestCase::canImagesBeIncludedInScheduledReports()
To create a system test, extends Piwik\Tests\Framework\TestCase\SystemTestCase. Then implement the getApiForTesting() method. This method should return an array of arrays. Each nested array contains information for a single test.
The first element in the array should be one or more API methods or the 'all' string. This determines which API methods whose output should be compared against expected files. The second element should be an associative array that contains a set of options that affect the way the test is run or URL used to invoke the API method. You are allowed to set the following options:
'xml'. The extension of the output is determined by the format.'all'.'day' or array('day', 'month').Some examples:
public function getApiForTesting()
{
$idSite = self::$fixture->idSite;
$dateTime = self::$fixture->dateTime;
return array(
// test a single API method
array('UserSettings.getResolution', array('idSite' => $idSite, 'date' => $dateTime)),
// test all methods in a plugin
array('API', array('idSite' => $idSite, 'date' => $dateTime)),
// test every API method
array('all', array('idSite' => $idSite, 'date' => $dateTime)),
// set some custom request parameters
array('API.getBulkRequest', array('format' => 'xml',
'testSuffix' => '_bulk_xml',
'otherRequestParameters' => array('urls' => $bulkUrls))),
// test multiple dates w/ multiple periods and multiple sites
array('UserSettings.getResolution', array('idSite' => 'all',
'date' => $dateTime,
'periods' => array('day', 'week', 'month'),
'setDateLastN' => true)),
);
}
After implementing getApiForTesting(), add the following test to the file:
/**
* @dataProvider getApiForTesting
* @group System
*/
public function testApi($api, $params)
{
$this->runApiTests($api, $params);
}
This will test every API method specified in getApiForTesting().
Before you can run your tests, you'll have to set the test's fixture. Fixtures add test data to the database by adding websites, tracking visits, etc.
To set a fixture, add a public static field named $fixture to your test class and initialize it below the class definition, for example:
namespace Piwik\Plugins\MyPlugin\Test;
use Piwik\Tests\Framework\TestCase\SystemTestCase;
class MySystemTest extends SystemTestCase
{
public static $fixture = null;
// ...
}
MySystemTest::$fixture = new \Test_Piwik_Fixture_ThreeGoalsOnePageview();
To see the fixtures Matomo (formerly Piwik) defines, see the files in the tests/PHPUnit/Fixtures directory.
You can create your own fixture as well, just extend Piwik\Tests\Framework\Fixture and place the file in the Test/Fixtures/ directory of your plugin.
System tests will generate an expected output file for every API method and period combination. The generated output (also called processed output) is stored in the processed/ subdirectory of your plugin's Test/ directory. The expected output should be stored in a directory named expected/.
When you first create a system test, there will be no expected files. You will have to copy processed files to the expected folder after ensuring they are correct.
When you work on a bug or feature and you push the code to Github then Travis will run our system tests. You might then notice that some system tests are failing because the expected content is different to the processed content. To not having to run the all the tests locally and as Travis might include different plugins it's recommended to download the processed files from Travis and then committing these to make the tests pass. However, it is very much needed to look at each changed file and assess that the changed content is expected. This can be done for example before pushing the changes.
To download the processed files from system tests execute below command. You will need to replace {BUILD_NUMBER} with the number of the build. You find this number when you select a build in Travis where it says eg Build #18. In this case, the build number is 18.
./console development:sync-system-test-processed {BUILD_NUMBER} --expected
This will download all the processed system test results and copies them directly into the expected system test directories. Next you can git add and commit and push them. Before adding or pushing make sure though that all the changes are expected.
Alternatively, you can also remove the --expected option in which case it will copy the downloaded files into your processed filter. You can then use for example the "Compare directories" feature in PHPStorm and only copy over the processed files into the expected directory where the changes are expected.
Only generated N API calls to test but was expecting more for this test.
Sometimes when writing a test you'll see this error in the test output:
Only generated N API calls to test but was expecting more for this test
Want to test APIs: ...
But only generated these URLs:
...
This can have the following possible causes:
get. System tests are designed only to be able to test
get API methods.null or equivalent. To determine
what the root issue is here, you'll need to inspect the variable $nameVariable in this if statement:
https://github.com/matomo-org/matomo/blob/4.2.0/core/API/DocumentationGenerator.php#L40-L41When running system tests that fetch data from API.getReportMetadata, API.getSegmentsMetadata, API.getReportPagesMetadata or API.getWidgetMetadata, the API response includes results from all other plugins. This can make it hard to maintain a passing build as tests might fail because of a change in a different plugin. To limit the response to the current plugin you can use the below example code.
class ApiTest extends SystemTestCase
{
public static function setUpBeforeClass(): void
{
parent::setUpBeforeClass();
self::setAllowedModulesToFilterApiResponse('API.getReportMetadata', array('MODULES_TO_FILTER'));
self::setAllowedCategoriesToFilterApiResponse('API.getSegmentsMetadata', array('CATEGORIES_TO_FILTER'));
self::setAllowedModulesToFilterApiResponse('API.getWidgetMetadata', array('MODULES_TO_FILTER'));
self::setAllowedCategoriesToFilterApiResponse('API.getReportPagesMetadata', array('CATEGORIES_TO_FILTER'));
}
}
It is also possible to write system tests for console commands. These tests should extend Piwik\Tests\Framework\TestCase\ConsoleCommandTestCase.
Example for a test that tests the config:set command:
class MyCommandTest extends \Piwik\Tests\Framework\TestCase\ConsoleCommandTestCase
{
public function test_command_succeedsWhenOptionsUsed()
{
// execute the command with few different options
$code = $this->applicationTester->run(array(
'command' => 'config:set',
'--section' => 'MySection',
'--key' => 'setting',
'--value' => 'myvalue',
'-vvv' => false,
));
// assert exit code
$this->assertEquals(0, $code, $this->getCommandDisplayOutputErrorMessage());
// assert the command actually performed the correct action
$config = $this->makeNewConfig();
$this->assertEquals(array('setting' => 'myvalue'), $config->MySection);
// assert printed command output
self::assertStringContainsString('Setting [MySection] setting = "myvalue"', $this->applicationTester->getDisplay());
}
}