aboutsummaryrefslogtreecommitdiffstats
path: root/meson/docs/markdown/Unit-tests.md
diff options
context:
space:
mode:
Diffstat (limited to 'meson/docs/markdown/Unit-tests.md')
-rw-r--r--meson/docs/markdown/Unit-tests.md270
1 files changed, 270 insertions, 0 deletions
diff --git a/meson/docs/markdown/Unit-tests.md b/meson/docs/markdown/Unit-tests.md
new file mode 100644
index 000000000..5233a4fe8
--- /dev/null
+++ b/meson/docs/markdown/Unit-tests.md
@@ -0,0 +1,270 @@
+---
+short-description: Meson's own unit-test system
+...
+
+# Unit tests
+
+Meson comes with a fully functional unit test system. To use it simply
+build an executable and then use it in a test.
+
+```meson
+e = executable('prog', 'testprog.c')
+test('name of test', e)
+```
+
+You can add as many tests as you want. They are run with the command `meson
+test`.
+
+Meson captures the output of all tests and writes it in the log file
+`meson-logs/testlog.txt`.
+
+## Test parameters
+
+Some tests require the use of command line arguments or environment
+variables. These are simple to define.
+
+```meson
+test('command line test', exe, args : ['first', 'second'])
+test('envvar test', exe2, env : ['key1=value1', 'key2=value2'])
+```
+
+Note how you need to specify multiple values as an array.
+
+### MALLOC_PERTURB_
+
+By default, environment variable
+[`MALLOC_PERTURB_`](http://man7.org/linux/man-pages/man3/mallopt.3.html) is
+set to a random value between 1..255. This can help find memory leaks on
+configurations using glibc, including with non-GCC compilers. This feature
+can be disabled as discussed in [test()](Reference-manual.md#test).
+
+## Coverage
+
+If you enable coverage measurements by giving Meson the command line
+flag `-Db_coverage=true`, you can generate coverage reports after
+running the tests (running the tests is required to gather the list of
+functions that get called). Meson will autodetect what coverage
+generator tools you have installed and will generate the corresponding
+targets. These targets are `coverage-xml` and `coverage-text` which
+are both provided by [Gcovr](http://gcovr.com) (version 3.3 or higher)
+and `coverage-html`, which requires
+[Lcov](https://ltp.sourceforge.io/coverage/lcov.php) and
+[GenHTML](https://linux.die.net/man/1/genhtml) or
+[Gcovr](http://gcovr.com). As a convenience, a high-level `coverage`
+target is also generated which will produce all 3 coverage report
+types, if possible.
+
+The output of these commands is written to the log directory `meson-logs` in
+your build directory.
+
+## Parallelism
+
+To reduce test times, Meson will by default run multiple unit tests in
+parallel. It is common to have some tests which can not be run in
+parallel because they require unique hold on some resource such as a
+file or a D-Bus name. You have to specify these tests with a keyword
+argument.
+
+```meson
+test('unique test', t, is_parallel : false)
+```
+
+Meson will then make sure that no other unit test is running at the
+same time. Non-parallel tests take longer to run so it is recommended
+that you write your unit tests to be parallel executable whenever
+possible.
+
+By default Meson uses as many concurrent processes as there are cores
+on the test machine. You can override this with the environment
+variable `MESON_TESTTHREADS` like this.
+
+```console
+$ MESON_TESTTHREADS=5 meson test
+```
+
+## Priorities
+
+*(added in version 0.52.0)*
+
+Tests can be assigned a priority that determines when a test is
+*started*. Tests with higher priority are started first, tests with
+lower priority started later. The default priority is 0, Meson makes
+no guarantee on the ordering of tests with identical priority.
+
+```meson
+test('started second', t, priority : 0)
+test('started third', t, priority : -50)
+test('started first', t, priority : 1000)
+```
+
+Note that the test priority only affects the starting order of tests
+and subsequent tests are affected by how long it takes previous tests
+to complete. It is thus possible that a higher-priority test is still
+running when lower-priority tests with a shorter runtime have
+completed.
+
+## Skipped tests and hard errors
+
+Sometimes a test can only determine at runtime that it can not be run.
+
+For the default `exitcode` testing protocol, the GNU standard approach
+in this case is to exit the program with error code 77. Meson will
+detect this and report these tests as skipped rather than failed. This
+behavior was added in version 0.37.0.
+
+For TAP-based tests, skipped tests should print a single line starting
+with `1..0 # SKIP`.
+
+In addition, sometimes a test fails set up so that it should fail even
+if it is marked as an expected failure. The GNU standard approach in
+this case is to exit the program with error code 99. Again, Meson will
+detect this and report these tests as `ERROR`, ignoring the setting of
+`should_fail`. This behavior was added in version 0.50.0.
+
+## Testing tool
+
+The goal of the Meson test tool is to provide a simple way to run
+tests in a variety of different ways. The tool is designed to be run
+in the build directory.
+
+The simplest thing to do is just to run all tests.
+
+```console
+$ meson test
+```
+
+### Run subsets of tests
+
+For clarity, consider the meson.build containing:
+
+```meson
+
+test('A', ..., suite: 'foo')
+test('B', ..., suite: ['foo', 'bar'])
+test('C', ..., suite: 'bar')
+test('D', ..., suite: 'baz')
+
+```
+
+Specify test(s) by name like:
+
+```console
+$ meson test A D
+```
+
+Tests belonging to a suite `suite` can be run as follows
+
+```console
+$ meson test --suite (sub)project_name:suite
+```
+
+Since version *0.46*, `(sub)project_name` can be omitted if it is the
+top-level project.
+
+Multiple suites are specified like:
+
+```console
+$ meson test --suite foo --suite bar
+```
+
+NOTE: If you choose to specify both suite(s) and specific test
+name(s), the test name(s) must be contained in the suite(s). This
+however is redundant-- it would be more useful to specify either
+specific test names or suite(s).
+
+### Other test options
+
+Sometimes you need to run the tests multiple times, which is done like this:
+
+```console
+$ meson test --repeat=10
+```
+
+Invoking tests via a helper executable such as Valgrind can be done with the
+`--wrap` argument
+
+```console
+$ meson test --wrap=valgrind testname
+```
+
+Arguments to the wrapper binary can be given like this:
+
+```console
+$ meson test --wrap='valgrind --tool=helgrind' testname
+```
+
+Meson also supports running the tests under GDB. Just doing this:
+
+```console
+$ meson test --gdb testname
+```
+
+Meson will launch `gdb` all set up to run the test. Just type `run` in
+the GDB command prompt to start the program.
+
+The second use case is a test that segfaults only rarely. In this case
+you can invoke the following command:
+
+```console
+$ meson test --gdb --repeat=10000 testname
+```
+
+This runs the test up to 10 000 times under GDB automatically. If the
+program crashes, GDB will halt and the user can debug the application.
+Note that testing timeouts are disabled in this case so `meson test`
+will not kill `gdb` while the developer is still debugging it. The
+downside is that if the test binary freezes, the test runner will wait
+forever.
+
+Sometimes, the GDB binary is not in the PATH variable or the user
+wants to use a GDB replacement. Therefore, the invoked GDB program can
+be specified *(added 0.52.0)*:
+
+```console
+$ meson test --gdb --gdb-path /path/to/gdb testname
+```
+
+```console
+$ meson test --print-errorlogs
+```
+
+Meson will report the output produced by the failing tests along with
+other useful information as the environmental variables. This is
+useful, for example, when you run the tests on Travis-CI, Jenkins and
+the like.
+
+For further information see the command line help of Meson by running
+`meson test -h`.
+
+## Legacy notes
+
+If `meson test` does not work for you, you likely have a old version
+of Meson. In that case you should call `mesontest` instead. If
+`mesontest` doesn't work either you have a very old version prior to
+0.37.0 and should upgrade.
+
+## Test outputs
+
+Meson will write several different files with detailed results of
+running tests. These will be written into $builddir/meson-logs/
+
+### testlog.json
+
+This is not a proper json file, but a file containing one valid json
+object per line. This is file is designed so each line is streamed out
+as each test is run, so it can be read as a stream while the test
+harness is running
+
+### testlog.junit.xml
+
+This is a valid JUnit XML description of all tests run. It is not
+streamed out, and is written only once all tests complete running.
+
+When tests use the `tap` protocol each test will be recorded as a
+testsuite container, with each case named by the number of the result.
+
+When tests use the `gtest` protocol Meson will inject arguments to the
+test to generate it's own JUnit XML, which Meson will include as part
+of this XML file.
+
+*New in 0.55.0*