diff options
Diffstat (limited to 'docs/3_Launch_the_tests.md')
| -rw-r--r-- | docs/3_Launch_the_tests.md | 327 |
1 files changed, 327 insertions, 0 deletions
diff --git a/docs/3_Launch_the_tests.md b/docs/3_Launch_the_tests.md new file mode 100644 index 0000000..b577acb --- /dev/null +++ b/docs/3_Launch_the_tests.md @@ -0,0 +1,327 @@ +# How to launch the tests ? + +## Natively during the development + +It could be convenient to be able to test the software that you are currently +developing. Then you can ensure that your modifications haven't introduced +regressions, bugs, etc. This depends upon your tests of course. + +As previously saw, you need the `test widget` to be able to launch the tests and +you need also to have the `afb-test` binding installed and the `application +framework binder` to be able to execute your tests. + +Assuming you already installed those components on your development host, then +proceed as the following from the project root directory: + +```bash +cd build +cmake -DBUILD_TEST_WGT=TRUE .. +make +make widget +``` + +To prepare all files needed for the test launch then use the `afm-test` script: + +```bash +# Usage of afm-test command line utility +afm-test --help +Usage: /opt/AGL/bin/afm-test <binding-wgt-rootdir> <test-wgt-rootdir> [-m|--mode <SOLO|SERVICE>] [-t|--timeout <X>] [-l|--lavaoutput] +binding-wgt-rootdir: path to the test wgt file +test-wgt-rootdir: path to the test folder file +-m|--mode: SOLO (1 binder) or SERVICE (2 binders) (Default: SOLO) +-t|--timeout: timeout in second. (Default 3 seconds) +-l|--lavaoutput: Flags indicating the binding to add Lava special test markers. +Error: Test launch failed. Code: 1 +# Launching the test from your build project directory +afm-test package package-test +``` + +### Example with the afb-test selftest suite + +Prepare the launch building the `test widget`: + +```bash +$ cd build +# Cleaning the previous build files +$ rm -rf * +# Configuration step +$ cmake -DBUILD_TEST_WGT=TRUE .. +-- The C compiler identification is GNU 8.2.1 +-- The CXX compiler identification is GNU 8.2.1 +-- Check for working C compiler: /usr/lib64/ccache/cc +-- Check for working C compiler: /usr/lib64/ccache/cc -- works +-- Detecting C compiler ABI info +-- Detecting C compiler ABI info - done +-- Detecting C compile features +-- Detecting C compile features - done +-- Check for working CXX compiler: /usr/lib64/ccache/c++ +-- Check for working CXX compiler: /usr/lib64/ccache/c++ -- works +-- Detecting CXX compiler ABI info +-- Detecting CXX compiler ABI info - done +-- Detecting CXX compile features +-- Detecting CXX compile features - done +Distribution detected (separated by ';' choose one of them) fedora +Include: /home/claneys/.config/app-templates/cmake.d/00-common-var.cmake +Include: /home/claneys/Workspace/Sources/IOTbzh/gerrit.automotivelinux.org/apps/app-afb-test/conf.d/cmake/00-default-osconfig.cmake +Include: /usr/share/cmake/Modules/CMakeAfbTemplates/cmake/cmake.d/01-build_options.cmake +-- Found PkgConfig: /usr/bin/pkg-config (found version "1.4.2") +-- Checking for module 'json-c' +-- Found json-c, version 0.13.1 +-- Checking for module 'libsystemd>=222' +-- Found libsystemd, version 238 +-- Checking for module 'afb-daemon>=4.0' +-- Found afb-daemon, version 6.90.0 +-- Checking for module 'lua>=5.3' +-- Found lua, version 5.3.4 +Include: /usr/share/cmake/Modules/CMakeAfbTemplates/cmake/cmake.d/02-variables.cmake +-- Check gcc_minimal_version (found gcc version 8.2.1) (found g++ version 8.2.1) +Include: /usr/share/cmake/Modules/CMakeAfbTemplates/cmake/cmake.d/03-macros.cmake +Include: /usr/share/cmake/Modules/CMakeAfbTemplates/cmake/cmake.d/04-extra_targets.cmake +-- Overwrite the CMAKE default install prefix with /opt/AGL +-- Found CURL: /usr/lib64/libcurl.so (found version "7.59.0") +-- Qt's WebSocket AFB Client: Disabled! +-- CURL wrapping helpers: Enabled! +-- Notice: LUA Controler Support Selected +-- Notice: Using default test widget configuration's file. +-- If you want to use a customized test-config.xml template then specify TEST_WIDGET_CONFIG_TEMPLATE in your config.cmake file. +-- Configuring done +-- Generating done +-- Build files have been written to: /home/claneys/Workspace/Sources/IOTbzh/gerrit.automotivelinux.org/apps/app-afb-test/build +# Build the binding +$ make +Scanning dependencies of target test-files +[ 3%] Generating test-files +[ 3%] Built target test-files +Scanning dependencies of target project_populate_test-files +[ 6%] Generating package-test/var/test-files +[ 6%] Built target project_populate_test-files +Scanning dependencies of target afb-test-config +[ 9%] Generating afb-test-config +Warning: JSON_CHECKER not found. Not verification made on files ! +[ 9%] Built target afb-test-config +Scanning dependencies of target project_populate_afb-test-config +[ 12%] Generating package-test/etc/afb-test-config +[ 12%] Built target project_populate_afb-test-config +Scanning dependencies of target aftest-config +[ 16%] Generating aftest-config +Warning: JSON_CHECKER not found. Not verification made on files ! +[ 16%] Built target aftest-config +Scanning dependencies of target project_populate_aftest-config +[ 19%] Generating package/etc/aftest-config +[ 19%] Built target project_populate_aftest-config +Scanning dependencies of target fixture-files +[ 22%] Generating fixture-files +[ 22%] Built target fixture-files +Scanning dependencies of target project_populate_fixture-files +[ 25%] Generating package-test/var/fixture-files +[ 25%] Built target project_populate_fixture-files +Scanning dependencies of target afTest-lua +[ 29%] Generating afTest-lua +[ 29%] Built target afTest-lua +Scanning dependencies of target project_populate_afTest-lua +[ 32%] Generating package/var/afTest-lua +[ 32%] Built target project_populate_afTest-lua +Scanning dependencies of target afb-helpers +[ 35%] Building C object afb-helpers/CMakeFiles/afb-helpers.dir/wrap-json.c.o +[ 38%] Building C object afb-helpers/CMakeFiles/afb-helpers.dir/filescan-utils.c.o +[ 41%] Building C object afb-helpers/CMakeFiles/afb-helpers.dir/escape.c.o +[ 45%] Building C object afb-helpers/CMakeFiles/afb-helpers.dir/curl-wrap.c.o +[ 48%] Linking C static library libafb-helpers.a +[ 48%] Built target afb-helpers +Scanning dependencies of target ctl-utilities +[ 51%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-action.c.o +[ 54%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-config.c.o +[ 58%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-onload.c.o +[ 61%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-plugin.c.o +[ 64%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-control.c.o +[ 67%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-event.c.o +[ 70%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-lua.c.o +[ 74%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-timer.c.o +[ 77%] Building C object app-controller-submodule/ctl-lib/CMakeFiles/ctl-utilities.dir/ctl-lua-utils.c.o +[ 80%] Linking C static library libctl-utilities.a +[ 80%] Built target ctl-utilities +Scanning dependencies of target aft +[ 83%] Building C object src/CMakeFiles/aft.dir/aft.c.o +[ 87%] Building C object src/CMakeFiles/aft.dir/mapis.c.o +[ 90%] Linking C shared module aft.so +[ 90%] Built target aft +Scanning dependencies of target project_populate_aft +[ 93%] Generating package/lib/aft.so +[ 93%] Built target project_populate_aft +Scanning dependencies of target populate +[ 96%] Generating package/bin, package/etc, package/lib, package/htdocs, package/var, package-test/bin, package-test/etc, package-test/lib, package-test/htdocs, package-test/var +[ 96%] Built target populate +Scanning dependencies of target afTest_build_done +++ Typical binding launch: afb-daemon --name afTest --port=1234 --workdir=package-test --ldpaths=/opt/AGL/lib64/afb:../package/lib --token= +[ 96%] Built target afTest_build_done +Scanning dependencies of target autobuild +[100%] Built target autobuild +# Build both widgets classic and test +$ make widget +[ 2%] Built target test-files +Scanning dependencies of target packaging_wgt +[ 5%] Generating package/config.xml +[ 8%] Generating package-test/config.xml, package-test/bin/launcher +[ 8%] Built target packaging_wgt +[ 11%] Generating package-test/var/test-files +[ 11%] Built target project_populate_test-files +Warning: JSON_CHECKER not found. Not verification made on files ! +[ 14%] Built target afb-test-config +[ 17%] Generating package-test/etc/afb-test-config +[ 17%] Built target project_populate_afb-test-config +Warning: JSON_CHECKER not found. Not verification made on files ! +[ 20%] Built target aftest-config +[ 23%] Generating package/etc/aftest-config +[ 23%] Built target project_populate_aftest-config +[ 26%] Built target fixture-files +[ 29%] Generating package-test/var/fixture-files +[ 29%] Built target project_populate_fixture-files +[ 32%] Built target afTest-lua +[ 35%] Generating package/var/afTest-lua +[ 35%] Built target project_populate_afTest-lua +[ 50%] Built target afb-helpers +[ 79%] Built target ctl-utilities +[ 88%] Built target aft +[ 91%] Built target project_populate_aft +[ 94%] Built target populate +Scanning dependencies of target widget +[ 97%] Generating aftest.wgt +NOTICE: -- PACKING widget aftest.wgt from directory /home/claneys/Workspace/Sources/IOTbzh/gerrit.automotivelinux.org/apps/app-afb-test/build/package +[100%] Generating aftest-test.wgt +NOTICE: -- PACKING widget aftest-test.wgt from directory /home/claneys/Workspace/Sources/IOTbzh/gerrit.automotivelinux.org/apps/app-afb-test/build/package-test +++ Install widget file using in the target : afm-util install afTest.wgt +[100%] Built target widget +``` + +Launch the test using the mode SERVICE for the `afb-test` because of a recursion +problem which loads 2 times the same binding and causes conflict. So it is +needed to uses 2 binders then each ones loads its binding properly: + +```bash +$ afm-test package package-test/ -m SERVICE +1..4 +# Started on Tue Oct 30 10:32:46 2018 +# Starting class: TestListverb +ok 1 TestListverb.testFunction +# Starting class: TestGetVerb +ok 2 TestGetVerb.testFunction +# Starting class: Test_turning_on +ok 3 Test_turning_on.testFunction +# Starting class: testLockWait +ok 4 testLockWait.testFunction +# Ran 4 tests in 0.001 seconds, 4 successes, 0 failures +~~~~~~~~~~ BEGIN ALL TESTS ~~~~~~~~~~ +1..59 +# Started on Tue Oct 30 10:32:48 2018 +# Starting class: testAssertEquals +~~~~~ Begin Test ~~~~~ +~~~~~ Begin Test Assert Equals ~~~~~ +~~~~~ End Test Assert Equals ~~~~~ +~~~~~ End Test ~~~~~ +ok 1 testAssertEquals.testFunction +# Starting class: testAssertNotEquals +~~~~~ Begin Test ~~~~~ +~~~~~ End Test ~~~~~ +ok 2 testAssertNotEquals.testFunction +# Starting class: testAssertItemsEquals +~~~~~ Begin Test ~~~~~ +[...] +~~~~~ End Test ~~~~~ +ok 57 testAssertVerbStatusError.testFunction +# Starting class: testAssertVerbResponseEqualsError +~~~~~ Begin Test ~~~~~ +~~~~~ End Test ~~~~~ +ok 58 testAssertVerbResponseEqualsError.testFunction +# Starting class: testAssertVerbCbError +~~~~~ Begin Test ~~~~~ +~~~~~ End Test ~~~~~ +ok 59 testAssertVerbCbError.testFunction +# Ran 59 tests in 0.003 seconds, 59 successes, 0 failures +~~~~~~~~~~ END ALL TESTS ~~~~~~~~~~ +Tests correctly launched. +``` + +## Launch test on a target board + +If you are trying to launch your test on a target you'll have to use +a test widget which contains test files, fixture and configuration. +Then use **afm-test**: + +```bash +# afm-test -h +Usage: /usr/bin/afm-test [-l|--lava] [-v|--verb <verb>] <path> +-l|--lavaoutput: flag that enable Lava test marker to the output. (Default: disabled) +-v|--verb: select a specific verb to launch from the test API. (Default: all) +path: path to the test wgt file +``` + +By default, the test widgets should be located in /usr/AGL/apps/testwgt. This +will install the widget, launch the tests then display the result on standard +output. After that it will kill test binding and remove it. + +### Example on a target + +Here is an example: + +```bash +qemux86-64:~# afm-test /usr/AGL/apps/testwgt/aftest-test.wgt +afm-test /tmp/aftest-test.wgt +PASS: aftest-test@6.90 started with pid=1649 +1..59 +~~~~~ Begin Test ~~~~~ +~~~~~ Begin Test Assert Equals ~~~~~ +~~~~~ End Test Assert Equals ~~~~~ +~~~~~ End Test ~~~~~ +PASS: 1 testAssertEquals.testFunction +~~~~~ Begin Test ~~~~~ +~~~~~ End Test ~~~~~ +PASS: 2 testAssertNotEquals.testFunction +~~~~~ Begin Test ~~~~~ +~~~~~ End Test ~~~~~ +PASS: 3 testAssertItemsEquals.testFunction +~~~~~ Begin Test ~~~~~ +~~~~~ End Test ~~~~~ +[...] +PASS: 58 testAssertVerbResponseEqualsError.testFunction +~~~~~ Begin Test ~~~~~ +~~~~~ End Test ~~~~~ +PASS: 59 testAssertVerbCbError.testFunction +# Ran 59 tests in 0.003 seconds, 59 successes, 0 failures +~~~~~~~~~~ END ALL TESTS ~~~~~~~~~~ +1..63 +~~~~~ Begin Test ~~~~~ +~~~~~ Begin Test Assert Equals ~~~~~ +~~~~~ End Test Assert Equals ~~~~~ +~~~~~ End Test ~~~~~ +PASS: 1 testAssertEquals.testFunction +~~~~~ Begin Test ~~~~~ +~~~~~ End Test ~~~~~ +PASS: 2 testAssertNotEquals.testFunction +~~~~~ Begin Test ~~~~~ +~~~~~ End Test ~~~~~ +PASS: 3 testAssertItemsEquals.testFunction +~~~~~ Begin Test ~~~~~ +~~~~~ End Test ~~~~~ +[...] +~~~~~ Begin Test ~~~~~ +~~~~~ End Test ~~~~~ +PASS: 61 TestGetVerb.testFunction +~~~~~ Begin Test ~~~~~ +~~~~~ End Test ~~~~~ +PASS: 62 Test_turning_on.testFunction +~~~~~ Begin Test ~~~~~ +~~~~~ End Test ~~~~~ +PASS: 63 testLockWait.testFunction +# Ran 63 tests in 0.003 seconds, 63 successes, 0 failures +~~~~~~~~~~ END ALL TESTS ~~~~~~~~~~ +PASS: aftest-test@6.90 killed and removed +``` + +The command being : ```afm-test /usr/AGL/apps/testwgt/aftest-test.wgt``` + +You can see here that everything ran as on your pc terminal. +**Begin Test** and **End Test** are the +beforeEach and afterEach functions and +**END ALL TESTS** is the after all functions. + + **PASS :** shows the function that is or was running. |