diff options
Diffstat (limited to 'meson/docs/markdown/Unit-tests.md')
-rw-r--r-- | meson/docs/markdown/Unit-tests.md | 270 |
1 files changed, 270 insertions, 0 deletions
diff --git a/meson/docs/markdown/Unit-tests.md b/meson/docs/markdown/Unit-tests.md new file mode 100644 index 000000000..5233a4fe8 --- /dev/null +++ b/meson/docs/markdown/Unit-tests.md @@ -0,0 +1,270 @@ +--- +short-description: Meson's own unit-test system +... + +# Unit tests + +Meson comes with a fully functional unit test system. To use it simply +build an executable and then use it in a test. + +```meson +e = executable('prog', 'testprog.c') +test('name of test', e) +``` + +You can add as many tests as you want. They are run with the command `meson +test`. + +Meson captures the output of all tests and writes it in the log file +`meson-logs/testlog.txt`. + +## Test parameters + +Some tests require the use of command line arguments or environment +variables. These are simple to define. + +```meson +test('command line test', exe, args : ['first', 'second']) +test('envvar test', exe2, env : ['key1=value1', 'key2=value2']) +``` + +Note how you need to specify multiple values as an array. + +### MALLOC_PERTURB_ + +By default, environment variable +[`MALLOC_PERTURB_`](http://man7.org/linux/man-pages/man3/mallopt.3.html) is +set to a random value between 1..255. This can help find memory leaks on +configurations using glibc, including with non-GCC compilers. This feature +can be disabled as discussed in [test()](Reference-manual.md#test). + +## Coverage + +If you enable coverage measurements by giving Meson the command line +flag `-Db_coverage=true`, you can generate coverage reports after +running the tests (running the tests is required to gather the list of +functions that get called). Meson will autodetect what coverage +generator tools you have installed and will generate the corresponding +targets. These targets are `coverage-xml` and `coverage-text` which +are both provided by [Gcovr](http://gcovr.com) (version 3.3 or higher) +and `coverage-html`, which requires +[Lcov](https://ltp.sourceforge.io/coverage/lcov.php) and +[GenHTML](https://linux.die.net/man/1/genhtml) or +[Gcovr](http://gcovr.com). As a convenience, a high-level `coverage` +target is also generated which will produce all 3 coverage report +types, if possible. + +The output of these commands is written to the log directory `meson-logs` in +your build directory. + +## Parallelism + +To reduce test times, Meson will by default run multiple unit tests in +parallel. It is common to have some tests which can not be run in +parallel because they require unique hold on some resource such as a +file or a D-Bus name. You have to specify these tests with a keyword +argument. + +```meson +test('unique test', t, is_parallel : false) +``` + +Meson will then make sure that no other unit test is running at the +same time. Non-parallel tests take longer to run so it is recommended +that you write your unit tests to be parallel executable whenever +possible. + +By default Meson uses as many concurrent processes as there are cores +on the test machine. You can override this with the environment +variable `MESON_TESTTHREADS` like this. + +```console +$ MESON_TESTTHREADS=5 meson test +``` + +## Priorities + +*(added in version 0.52.0)* + +Tests can be assigned a priority that determines when a test is +*started*. Tests with higher priority are started first, tests with +lower priority started later. The default priority is 0, Meson makes +no guarantee on the ordering of tests with identical priority. + +```meson +test('started second', t, priority : 0) +test('started third', t, priority : -50) +test('started first', t, priority : 1000) +``` + +Note that the test priority only affects the starting order of tests +and subsequent tests are affected by how long it takes previous tests +to complete. It is thus possible that a higher-priority test is still +running when lower-priority tests with a shorter runtime have +completed. + +## Skipped tests and hard errors + +Sometimes a test can only determine at runtime that it can not be run. + +For the default `exitcode` testing protocol, the GNU standard approach +in this case is to exit the program with error code 77. Meson will +detect this and report these tests as skipped rather than failed. This +behavior was added in version 0.37.0. + +For TAP-based tests, skipped tests should print a single line starting +with `1..0 # SKIP`. + +In addition, sometimes a test fails set up so that it should fail even +if it is marked as an expected failure. The GNU standard approach in +this case is to exit the program with error code 99. Again, Meson will +detect this and report these tests as `ERROR`, ignoring the setting of +`should_fail`. This behavior was added in version 0.50.0. + +## Testing tool + +The goal of the Meson test tool is to provide a simple way to run +tests in a variety of different ways. The tool is designed to be run +in the build directory. + +The simplest thing to do is just to run all tests. + +```console +$ meson test +``` + +### Run subsets of tests + +For clarity, consider the meson.build containing: + +```meson + +test('A', ..., suite: 'foo') +test('B', ..., suite: ['foo', 'bar']) +test('C', ..., suite: 'bar') +test('D', ..., suite: 'baz') + +``` + +Specify test(s) by name like: + +```console +$ meson test A D +``` + +Tests belonging to a suite `suite` can be run as follows + +```console +$ meson test --suite (sub)project_name:suite +``` + +Since version *0.46*, `(sub)project_name` can be omitted if it is the +top-level project. + +Multiple suites are specified like: + +```console +$ meson test --suite foo --suite bar +``` + +NOTE: If you choose to specify both suite(s) and specific test +name(s), the test name(s) must be contained in the suite(s). This +however is redundant-- it would be more useful to specify either +specific test names or suite(s). + +### Other test options + +Sometimes you need to run the tests multiple times, which is done like this: + +```console +$ meson test --repeat=10 +``` + +Invoking tests via a helper executable such as Valgrind can be done with the +`--wrap` argument + +```console +$ meson test --wrap=valgrind testname +``` + +Arguments to the wrapper binary can be given like this: + +```console +$ meson test --wrap='valgrind --tool=helgrind' testname +``` + +Meson also supports running the tests under GDB. Just doing this: + +```console +$ meson test --gdb testname +``` + +Meson will launch `gdb` all set up to run the test. Just type `run` in +the GDB command prompt to start the program. + +The second use case is a test that segfaults only rarely. In this case +you can invoke the following command: + +```console +$ meson test --gdb --repeat=10000 testname +``` + +This runs the test up to 10 000 times under GDB automatically. If the +program crashes, GDB will halt and the user can debug the application. +Note that testing timeouts are disabled in this case so `meson test` +will not kill `gdb` while the developer is still debugging it. The +downside is that if the test binary freezes, the test runner will wait +forever. + +Sometimes, the GDB binary is not in the PATH variable or the user +wants to use a GDB replacement. Therefore, the invoked GDB program can +be specified *(added 0.52.0)*: + +```console +$ meson test --gdb --gdb-path /path/to/gdb testname +``` + +```console +$ meson test --print-errorlogs +``` + +Meson will report the output produced by the failing tests along with +other useful information as the environmental variables. This is +useful, for example, when you run the tests on Travis-CI, Jenkins and +the like. + +For further information see the command line help of Meson by running +`meson test -h`. + +## Legacy notes + +If `meson test` does not work for you, you likely have a old version +of Meson. In that case you should call `mesontest` instead. If +`mesontest` doesn't work either you have a very old version prior to +0.37.0 and should upgrade. + +## Test outputs + +Meson will write several different files with detailed results of +running tests. These will be written into $builddir/meson-logs/ + +### testlog.json + +This is not a proper json file, but a file containing one valid json +object per line. This is file is designed so each line is streamed out +as each test is run, so it can be read as a stream while the test +harness is running + +### testlog.junit.xml + +This is a valid JUnit XML description of all tests run. It is not +streamed out, and is written only once all tests complete running. + +When tests use the `tap` protocol each test will be recorded as a +testsuite container, with each case named by the number of the result. + +When tests use the `gtest` protocol Meson will inject arguments to the +test to generate it's own JUnit XML, which Meson will include as part +of this XML file. + +*New in 0.55.0* |