path: root/docs/3_Launch_the_tests.md

# How to launch the tests ?

## Natively during the development

It could be convenient to be able to test the software that you are currently
developing. Then you can ensure that your modifications haven't introduced
regressions, bugs, etc. This depends upon your tests of course.

As previously saw, you need the `test widget` to be able to launch the tests and
you need also to have the `afb-test` binding installed and the `application
framework binder` to be able to execute your tests.

Assuming you already installed those components on your development host, then
proceed as the following from the project root directory:

```bash
cd build
cmake -DBUILD_TEST_WGT=TRUE ..
make
make widget
```

To prepare all files needed for the test launch then use the `afm-test` script:

```bash
# Usage of afm-test command line utility
afm-test --help
Usage: /opt/AGL/bin/afm-test <binding-wgt-rootdir> <test-wgt-rootdir> [-m|--mode <SOLO|SERVICE>] [-t|--timeout <X>] [-l|--lavaoutput]
binding-wgt-rootdir: path to the test wgt file
test-wgt-rootdir: path to the test folder file
-m|--mode: SOLO (1 binder) or SERVICE (2 binders) (Default: SOLO)
-t|--timeout: timeout in second. (Default 3 seconds)
-l|--lavaoutput: Flags indicating the binding to add Lava special test markers.
Error: Test launch failed. Code: 1
# Launching the test from your build project directory
afm-test package package-test
```

### Example with the afb-test selftest suite

Prepare the launch building the `test widget`:

```bash
$ cd build
# Cleaning the previous build files
$ rm -rf *
# Configuration step
$ cmake -DBUILD_TEST_WGT=TRUE ..
-- The C compiler identification is GNU 8.2.1
-- The CXX compiler identification is GNU 8.2.1
-- Check for working C compiler: /usr/lib64/ccache/cc
-- Check for working C compiler: /usr/lib64/ccache/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/lib64/ccache/c++
-- Check for working CXX compiler: /usr/lib64/ccache/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Distribution detected (separated by ';' choose one of them) fedora
Include: /home/claneys/.config/app-templates/cmake.d/00-common-var.cmake
Include: /home/claneys/Workspace/Sources/IOTbzh/gerrit.automotivelinux.org/apps/app-afb-test/conf.d/cmake/00-default-osconfig.cmake
Include: /usr/share/cmake/Modules/CMakeAfbTemplates/cmake/cmake.d/01-build_options.cmake
-- Found PkgConfig: /usr/bin/pkg-config (found version "1.4.2")
-- Checking for module 'json-c'
--   Found json-c, version 0.13.1
-- Checking for module 'libsystemd>=222'
--   Found libsystemd, version 238
-- Checking for module 'afb-daemon>=4.0'
--   Found afb-daemon, version 6.90.0
-- Checking for module 'lua>=5.3'
--   Found lua, version 5.3.4
Include: /usr/share/cmake/Modules/CMakeAfbTemplates/cmake/cmake.d/02-variables.cmake
-- Check gcc_minimal_version (found gcc version 8.2.1)  (found g++ version 8.2.1)
Include: /usr/share/cmake/Modules/CMakeAfbTemplates/cmake/cmake.d/03-macros.cmake
Include: /usr/share/cmake/Modules/CMakeAfbTemplates/cmake/cmake.d/04-extra_targets.cmake
-- Overwrite the CMAKE default install prefix with /opt/AGL
-- Found CURL: /usr/lib64/libcurl.so (found version "7.59.0")
-- Qt's WebSocket AFB Client: Disabled!
-- CURL wrapping helpers: Enabled!
-- Notice: LUA Controler Support Selected
-- Notice: Using default test widget configuration's file.
-- If you want to use a customized test-config.xml template then specify TEST_WIDGET_CONFIG_TEMPLATE in your config.cmake file.
-- Configuring done
-- Generating done
-- Build files have been written to: /home/claneys/Workspace/Sources/IOTbzh/gerrit.automotivelinux.org/apps/app-afb-test/build
# Build the binding
$ make
Scanning dependencies of target test-files
[  3%] Generating test-files
[  3%] Built target test-files
Scanning dependencies of target project_populate_test-files
[  6%] Generating package-test/var/test-files
[  6%] Built target project_populate_test-files
Scanning dependencies of target afb-test-config
[  9%] Generating afb-test-config
Warning: JSON_CHECKER not found. Not verification made on files !
[  9%] Built target afb-test-config
Scanning dependencies of target project_populate_afb-test-config
[ 12%] Generating package-test/etc/afb-test-config
[ 12%] Built target project_populate_afb-test-config
Scanning dependencies of target aftest-config
[ 16%] Generating aftest-config
Warning: JSON_CHECKER not found. Not verification made on files !
[ 16%] Built target aftest-config
Scanning dependencies of target project_populate_aftest-config
[ 19%] Generating package/etc/aftest-config
[ 19%] Built target project_populate_aftest-config
Scanning dependencies of target fixture-files
[ 22%] Generating fixture-files
[ 22%] Built target fixture-files
Scanning dependencies of target project_populate_fixture-files
[ 25%] Generating package-test/var/fixture-files
[ 25%] Built target project_populate_fixture-files
Scanning dependencies of target afTest-lua
[ 29%] Generating afTest-lua
[ 29%] Built target afTest-lua
Scanning dependencies of target project_populate_afTest-lua
[ 32%] Generating package/var/afTest-lua
[ 32%] Built target project_populate_afTest-lua
Scanning dependencies of target afb-helpers
[ 35%] Building C object afb-helpers/CMakeFiles/afb-helpers.dir/wrap-json.c.o
[ 38%] Building C object afb-helpers/CMakeFiles/afb-helpers.dir/filescan-utils.c.o
[ 41%] Building C object afb-helpers/CMakeFiles/afb-helpers.dir/escape.c.o
[ 45%] Building C object afb-helpers/CMakeFiles/afb-helpers.dir/curl-wrap.c.o
[ 48%] Linking C static library libafb-helpers.a
[ 48%] Built target afb-helpers
Scanning dependencies of target ctl-utilities
[ 51%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-action.c.o
[ 54%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-config.c.o
[ 58%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-onload.c.o
[ 61%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-plugin.c.o
[ 64%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-control.c.o
[ 67%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-event.c.o
[ 70%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-lua.c.o
[ 74%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-timer.c.o
[ 77%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-lua-utils.c.o
[ 80%] Linking C static library libctl-utilities.a
[ 80%] Built target ctl-utilities
Scanning dependencies of target aft
[ 83%] Building C object src/CMakeFiles/aft.dir/aft.c.o
[ 87%] Building C object src/CMakeFiles/aft.dir/mapis.c.o
[ 90%] Linking C shared module aft.so
[ 90%] Built target aft
Scanning dependencies of target project_populate_aft
[ 93%] Generating package/lib/aft.so
[ 93%] Built target project_populate_aft
Scanning dependencies of target populate
[ 96%] Generating package/bin, package/etc, package/lib, package/htdocs, package/var, package-test/bin, package-test/etc, package-test/lib, package-test/htdocs, package-test/var
[ 96%] Built target populate
Scanning dependencies of target afTest_build_done
++ Typical binding launch: afb-daemon --name afTest --port=1234 --workdir=package-test --ldpaths=/opt/AGL/lib64/afb:../package/lib --token=
[ 96%] Built target afTest_build_done
Scanning dependencies of target autobuild
[100%] Built target autobuild
# Build both widgets classic and test
$ make widget
[  2%] Built target test-files
Scanning dependencies of target packaging_wgt
[  5%] Generating package/config.xml
[  8%] Generating package-test/config.xml, package-test/bin/launcher
[  8%] Built target packaging_wgt
[ 11%] Generating package-test/var/test-files
[ 11%] Built target project_populate_test-files
Warning: JSON_CHECKER not found. Not verification made on files !
[ 14%] Built target afb-test-config
[ 17%] Generating package-test/etc/afb-test-config
[ 17%] Built target project_populate_afb-test-config
Warning: JSON_CHECKER not found. Not verification made on files !
[ 20%] Built target aftest-config
[ 23%] Generating package/etc/aftest-config
[ 23%] Built target project_populate_aftest-config
[ 26%] Built target fixture-files
[ 29%] Generating package-test/var/fixture-files
[ 29%] Built target project_populate_fixture-files
[ 32%] Built target afTest-lua
[ 35%] Generating package/var/afTest-lua
[ 35%] Built target project_populate_afTest-lua
[ 50%] Built target afb-helpers
[ 79%] Built target ctl-utilities
[ 88%] Built target aft
[ 91%] Built target project_populate_aft
[ 94%] Built target populate
Scanning dependencies of target widget
[ 97%] Generating aftest.wgt
NOTICE: -- PACKING widget aftest.wgt from directory /home/claneys/Workspace/Sources/IOTbzh/gerrit.automotivelinux.org/apps/app-afb-test/build/package
[100%] Generating aftest-test.wgt
NOTICE: -- PACKING widget aftest-test.wgt from directory /home/claneys/Workspace/Sources/IOTbzh/gerrit.automotivelinux.org/apps/app-afb-test/build/package-test
++ Install widget file using in the target : afm-util install afTest.wgt
[100%] Built target widget
```

Launch the test using the mode SERVICE for the `afb-test` because of a recursion
problem which loads 2 times the same binding and causes conflict. So it is
needed to uses 2 binders then each ones loads its binding properly:

```bash
$ afm-test package package-test/ -m SERVICE
1..4
# Started on Tue Oct 30 10:32:46 2018
# Starting class: TestListverb
ok     1        TestListverb.testFunction
# Starting class: TestGetVerb
ok     2        TestGetVerb.testFunction
# Starting class: Test_turning_on
ok     3        Test_turning_on.testFunction
# Starting class: testLockWait
ok     4        testLockWait.testFunction
# Ran 4 tests in 0.001 seconds, 4 successes, 0 failures
~~~~~~~~~~ BEGIN ALL TESTS ~~~~~~~~~~
1..59
# Started on Tue Oct 30 10:32:48 2018
# Starting class: testAssertEquals
~~~~~ Begin Test ~~~~~
~~~~~ Begin Test Assert Equals ~~~~~
~~~~~ End Test Assert Equals ~~~~~
~~~~~ End Test ~~~~~
ok     1        testAssertEquals.testFunction
# Starting class: testAssertNotEquals
~~~~~ Begin Test ~~~~~
~~~~~ End Test ~~~~~
ok     2        testAssertNotEquals.testFunction
# Starting class: testAssertItemsEquals
~~~~~ Begin Test ~~~~~
[...]
~~~~~ End Test ~~~~~
ok     57       testAssertVerbStatusError.testFunction
# Starting class: testAssertVerbResponseEqualsError
~~~~~ Begin Test ~~~~~
~~~~~ End Test ~~~~~
ok     58       testAssertVerbResponseEqualsError.testFunction
# Starting class: testAssertVerbCbError
~~~~~ Begin Test ~~~~~
~~~~~ End Test ~~~~~
ok     59       testAssertVerbCbError.testFunction
# Ran 59 tests in 0.003 seconds, 59 successes, 0 failures
~~~~~~~~~~ END ALL TESTS ~~~~~~~~~~
Tests correctly launched.
```

## Launch test on a target board

If you are trying to launch your test on a target you'll have to use
a test widget which contains test files, fixture and configuration.
Then use **afm-test**:

```bash
# afm-test -h
Usage: /usr/bin/afm-test [-l|--lava] [-v|--verb <verb>] <path>
-l|--lavaoutput: flag that enable Lava test marker to the output. (Default: disabled)
-v|--verb: select a specific verb to launch from the test API. (Default: all)
path: path to the test wgt file
```

By default, the test widgets should be located in /usr/AGL/apps/testwgt. This
will install the widget, launch the tests then display the result on standard
output. After that it will kill test binding and remove it.

### Example on a target

Here is an example:

```bash
qemux86-64:~# afm-test /usr/AGL/apps/testwgt/aftest-test.wgt
afm-test /tmp/aftest-test.wgt
PASS: aftest-test@6.90 started with pid=1649
1..59
~~~~~ Begin Test ~~~~~
~~~~~ Begin Test Assert Equals ~~~~~
~~~~~ End Test Assert Equals ~~~~~
~~~~~ End Test ~~~~~
PASS: 1 testAssertEquals.testFunction
~~~~~ Begin Test ~~~~~
~~~~~ End Test ~~~~~
PASS: 2 testAssertNotEquals.testFunction
~~~~~ Begin Test ~~~~~
~~~~~ End Test ~~~~~
PASS: 3 testAssertItemsEquals.testFunction
~~~~~ Begin Test ~~~~~
~~~~~ End Test ~~~~~
[...]
PASS: 58 testAssertVerbResponseEqualsError.testFunction
~~~~~ Begin Test ~~~~~
~~~~~ End Test ~~~~~
PASS: 59 testAssertVerbCbError.testFunction
# Ran 59 tests in 0.003 seconds, 59 successes, 0 failures
~~~~~~~~~~ END ALL TESTS ~~~~~~~~~~
1..63
~~~~~ Begin Test ~~~~~
~~~~~ Begin Test Assert Equals ~~~~~
~~~~~ End Test Assert Equals ~~~~~
~~~~~ End Test ~~~~~
PASS: 1 testAssertEquals.testFunction
~~~~~ Begin Test ~~~~~
~~~~~ End Test ~~~~~
PASS: 2 testAssertNotEquals.testFunction
~~~~~ Begin Test ~~~~~
~~~~~ End Test ~~~~~
PASS: 3 testAssertItemsEquals.testFunction
~~~~~ Begin Test ~~~~~
~~~~~ End Test ~~~~~
[...]
~~~~~ Begin Test ~~~~~
~~~~~ End Test ~~~~~
PASS: 61 TestGetVerb.testFunction
~~~~~ Begin Test ~~~~~
~~~~~ End Test ~~~~~
PASS: 62 Test_turning_on.testFunction
~~~~~ Begin Test ~~~~~
~~~~~ End Test ~~~~~
PASS: 63 testLockWait.testFunction
# Ran 63 tests in 0.003 seconds, 63 successes, 0 failures
~~~~~~~~~~ END ALL TESTS ~~~~~~~~~~
PASS: aftest-test@6.90 killed and removed
```

The command being : ```afm-test /usr/AGL/apps/testwgt/aftest-test.wgt```

You can see here that everything ran as on your pc terminal.
**Begin Test**  and **End Test** are the
beforeEach and afterEach functions and
**END ALL TESTS**  is the after all functions.

 **PASS :** shows the function that is or was running.