aboutsummaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)AuthorFilesLines
2022-12-08agl-test-framework: add "XPASS" and "XFAIL" statesduerpei5-32/+35
Adjust the test framework , add "XPASS" and "XFAIL" states for the result of test case, and delete useless code by the way. Bug-AGL: SPEC-4345 Signed-off-by: duerpei <duep.fnst@fujitsu.com> Change-Id: I5763dc27caef956fb6632fcb59d9f95d671f4766
2022-12-02agl-test-framework: add test 'crashme'yanxk6-1/+136
'agl-test' modified, now, usr would not run dangerous tests like this 'crashme' unless they do self selection by '-m' pytest option. Set default to 'not dangerous'. a new marker 'dangerous' has been add to the pytest.ini. add new method into the agl_test_base.py, to help get path of the 'spec.json', this file help stress tests like 'crashme' to run under different stress levels. for run_tests.py, modifications focus on how to run 'crashme' under that different levels. new marker 'order' would always ensure this 'crashme' test to be placed as the so called 'last' test of all. Bug-AGL: SPEC-4345 Signed-off-by: yanxk <yanxk.fnst@fujitsu.com> Change-Id: Icb36d7c5c52430d89787f4373a2852b1b03ec8a6
2022-08-27agl-test-framework: add test 'stress'duerpei2-0/+57
Add test script for 'stress' Bug-AGL: SPEC-4345 Signed-off-by: duerpei <duep.fnst@fujitsu.com> Change-Id: I1c0c048dc907e5cff5d50396ab28b7550371b32e
2022-11-11agl-test-framework: enable ptest of 'bc'yanxk3-0/+65
we have already submitted patches that enabled ptest of 'bc' to the OpenEmbedded community and the patches got merged. but it is not yet in the YP 4.0.5, so it would skip now. it can be seen in YP 4.1. we now add it into the test framework, for it would be available in the next YP release, maybe, or several versions later. Signed-off-by: yanxk <yanxk.fnst@fujitsu.com> Change-Id: Iffdef472c9449c9e3179c3b3bd1273fb2ec0e032
2022-08-14agl-test-framework: deploy parametric implementation for old ‘glib-2.0’ testyanxk1-34/+6
Single definitions are no longer used in this test framework, they are now replaced by another parametric implementation. The benefit of this is that it minimizes the likelihood of requiring manual maintenance due to the small but frequent updates from the upstream. Bug-AGL: SPEC-4345 Signed-off-by: yanxk <yanxk.fnst@fujitsu.com> Change-Id: I86a0edabb27c7117d8279e1478be94f456894533
2022-08-14agl-test-framework: enable ptest of 'openssl'yanxk3-0/+72
Base on the agl-test-framework, the ptest results of 'openssl' can now be collected, analyzed and then reported. Here are 3 samples that the output result might look like: tests/openssl/run_tests.py::test_openssl[30-test_evp_fetch_prov] PASSED tests/openssl/run_tests.py::test_openssl[30-test_evp_kdf] PASSED tests/openssl/run_tests.py::test_openssl[30-test_evp_libctx] PASSED Bug-AGL: SPEC-4345 Signed-off-by: yanxk <yanxk.fnst@fujitsu.com> Change-Id: Id28a36ce081e8cfb49dcb7af8db369244f1fd258
2022-11-11conftest: get zip file name dynamicallyduerpei1-8/+11
Get zip file name dynamically, and the name of the zip file is formatted as follows: agl-test-log-13.0.1-raspberrypi4-20200808.zip And delete some useless comments Bug-AGL: SPEC-4345 Signed-off-by: duerpei <duep.fnst@fujitsu.com> Change-Id: Id321deb0232388a2ab4271d04ee6c5979a625672
2022-11-10agl-test-framework: deploy parametric implementation for old ‘expat’ testyanxk1-3416/+6
Single definitions are no longer used in this test framework, they are now replaced by another parametric implementation. The benefit of this is that it minimizes the likelihood of requiring manual maintenance due to the small but frequent updates from the upstream. Bug-AGL: SPEC-4345 Signed-off-by: yanxk <yanxk.fnst@fujitsu.com> Change-Id: I326c429b0750e7d7e623f45d36d4e53e5c5b97c4
2022-11-10agl-test-framework: deploy parametric implementation for old ‘zlib’ testyanxk1-24/+6
Single definitions are no longer used in this test framework, they are now replaced by another parametric implementation. The benefit of this is that it minimizes the likelihood of requiring manual maintenance due to the small but frequent updates from the upstream. Bug-AGL: SPEC-4345 Signed-off-by: yanxk <yanxk.fnst@fujitsu.com> Change-Id: I18ee28b6ebce12c61a55e87fef9f1b96f6db958c
2022-11-10agl-test-framework: deploy parametric implementation for old ‘bzip2’ testyanxk1-2015/+6
Single definitions are no longer used in this test framework, they are now replaced by another parametric implementation. The benefit of this is that it minimizes the likelihood of requiring manual maintenance due to the small but frequent updates from the upstream. Bug-AGL: SPEC-4345 Signed-off-by: yanxk <yanxk.fnst@fujitsu.com> Change-Id: I609a4c54edcc2b68a43d759fa60e9eee64d9c2cf
2022-11-10agl-test-framework: enable ptest of 'libxml2'yanxk3-0/+69
Base on the agl-test-framework, the ptest results of 'libxml2' can now be collected, analyzed and then reported. Here are 3 samples that the output result might look like: tests/libxml2/run_tests.py::test_libxml2[Reader regression tests] PASSED tests/libxml2/run_tests.py::test_libxml2[Reader entities substitution regression tests] PASSED tests/libxml2/run_tests.py::test_libxml2[Reader on memory regression tests] PASSED Bug-AGL: SPEC-4345 Signed-off-by: yanxk <yanxk.fnst@fujitsu.com> Change-Id: I8361fe54177d951c36732ebfbe8dd8915975d7a9
2022-11-08agl-test-framework: enable ptest of 'libpam'yanxk3-0/+68
Base on the agl-test-framework, the ptest results of 'libpam' can now be collected, analyzed and then reported. Here are 3 samples that the output result might look like: tests/libpam/run_tests.py::test_libpam[tst-pam_acct_mgmt] PASSED tests/libpam/run_tests.py::test_libpam[tst-pam_authenticate] PASSED tests/libpam/run_tests.py::test_libpam[tst-pam_chauthtok] PASSED Bug-AGL: SPEC-4345 Signed-off-by: yanxk <yanxk.fnst@fujitsu.com> Change-Id: I11c73689b5073ca4bf24b34d128287e029afdb3a
2022-08-11agl-test-framework: Add new method to the plugins/agl_test_ptest_base.pyyanxk1-0/+5
This new method is named 'run_ptest', which means it runs the ptest and does the log process, it is a common method that all ptest cases would require. Bug-AGL: SPEC-4345 Signed-off-by: yanxk <yanxk.fnst@fujitsu.com> Change-Id: Ifb2ae53ca3aad941216dae94ac5e07d28eca55b7
2022-07-24Add precheck() funQiu Tingting7-2/+60
Checking env before run testsuite is necessary. 1. Add common check in agl_test_base.py. It is currently an empty function. If necessary, common check sould be added. 2. Add ptest common check in agl_test_ptest_base.py. For ptest testsuite, check whether ptest-runner and run-ptest script exist. 3. Add special check in run_tests.py. For aio-stress testsuite, check whether compiled aio-stress script exists For bzip2 testsuite, check whether bzip2 cmd exists. Bug-AGL: SPEC-4345 Signed-off-by: Qiu Tingting <qiutt@fujitsu.com> Change-Id: Id7841c0337465266dd607403ecb3e1c4377c6198
2022-07-23agl-test-framework: enable ptest of 'zlib'yanxk3-0/+79
Base on the agl-test-framework, the test results of OSS "zlib" can now be collected, analyzed and then reported. Bug-AGL: SPEC-4345 Signed-off-by: yanxk <yanxk.fnst@fujitsu.com> Change-Id: I023c080cc365facb7d647f4c1085b7c9739f0b0e
2022-07-21tests: solve the problem of duplicate nameduerpei3-2/+2
To avoid error caused by the same name between the test set and pyhton module, add an empty '__ init__. py' file in '/tests' Bug-AGL: SPEC-4345 Signed-off-by: duerpei <duep.fnst@fujitsu.com> Change-Id: I7748e6ae90b2eb12b59a39b3659b5910fc487810
2022-07-11agl-test-framework: enable Ptest of "bzip2"yanxk3-0/+2082
Base on the agl-test-framework, the test results of OSS "bzip2" can now be collected, analyzed and then reported. Bug-AGL: SPEC-4345 Signed-off-by: yanxk <yanxk.fnst@fujitsu.com> Change-Id: I28df7079cfc0bc4b95b3ca195119f435b9ba8b6b
2022-09-30agl-test-framework: add test set of aio-stressduerpei3-0/+53
Add the script file that calls the test set 'aiostress'. Bug-AGL: SPEC-4345 Signed-off-by: duerpei <duep.fnst@fujitsu.com> Change-Id: Ib3906fed1f85031d3554342a5555aa3a046b8c09
2022-07-02agl-test-framework: enable Ptest of "Expat"yanxk3-0/+3494
Base on the agl-test-framework, the test results of OSS "Expat" can now be collected, analyzed and then reported. Bug-AGL: SPEC-4345 Signed-off-by: yanxk <yanxk.fnst@fujitsu.com> Change-Id: Ibd1cccc6f78f4ee6a640b47eb74d2b35f40b6d31
2022-06-28agl-test-framework: fix a naming definition mistakeyanxk1-1/+1
Since no 'date' information was written into the JSON file, the name of the function should be 'write_data_to_json'. Bug-AGL: SPEC-4345 Signed-off-by: yanxk <yanxk.fnst@fujitsu.com> Change-Id: I61043c28d36a6c208b729b7c755da4b97970ae97
2022-06-16agl-test-framework: use PTESTBaseTest classduerpei3-7/+56
Move the "run_tests" script of glib2 under "qa-test-misc" to "agl-test-framework", this is more reasonable. And modify the script file run_tests under glib2, make it use PTESTBaseTest class. Later, I will delete the "run_tests" scripts under "qa-test-misc" Bug-AGL: SPEC-4345 Signed-off-by: duerpei <duep.fnst@fujitsu.com> Change-Id: Id055b829954bf8bdf75a90ff5b417baae0c4d5fb
2022-06-11agl-test-framework: add PTESTBaseTest classduerpei2-2/+30
Add PTESTBaseTest class for test sets called from ptest. It inherits from AGLBaseTest class. It contains two common functions used to run the ptest test set. In test suite, please create a child class based on PTESTBaseTest. Then it is easy to init, run tests, check results and make a repoter. By the way, delete the useless code in agl_test_report.py Bug-AGL: SPEC-4345 Signed-off-by: duerpei <duep.fnst@fujitsu.com> Change-Id: Ief99073e4f63a44bde387b8c42d029c47fbf0943
2022-05-26agl-test-framework: add AGLBaseTest classQiu Tingting5-12/+275
Add AGLBaseTest class for existing test suites which has own test scripts. It package the default operations, like as log processing and file saving. In test suite, please create a child class based onAGLBaseTest. Then it is easy to init, run tests, check results and make a repoter. Bug-AGL: SPEC-4345 Signed-off-by: Qiu Tingting <qiutt@fujitsu.com> Change-Id: I9dbc01db59df16eb5718b19b3223ad95da0afb11
2022-05-25agl-test-framework: fix a bug for get_html_filenamelamprey_12.1.6lamprey_12.1.5lamprey/12.1.6lamprey/12.1.512.1.612.1.5Qiu Tingting1-1/+1
fix a bug for get_html_filename methond in agl_test_report.py Bug-AGL: SPEC-4345 Signed-off-by: Qiu Tingting <qiutt@fujitsu.com> Change-Id: Iea7dd4d09ab70189f59325fc0200486ae142f584
2022-05-22agl-test-framework: Add get method for agl-test-conf.pyQiu Tingting1-5/+47
Add get method for agl-test-conf.py get_tmp_log_dir get_tpl_dir get_default_html_tpl get_json_filename get_html_filename get_log_dir Bug-AGL: SPEC-4345 Signed-off-by: Qiu Tingting <qiutt@fujitsu.com> Change-Id: I6f63447a32df752033945ff77ad577dd46f0733a
2022-05-21agl-test-framework: change order of parameter for write_date_to_json.Qiu Tingting1-1/+1
Signed-off-by: Qiu Tingting <qiutt@fujitsu.com> Change-Id: Ia5cb70ac5ada8be10c0be530a24fb5d6537f7cf6
2022-07-28agl-test-framework: demo code submissionneedlefish_13.93.0needlefish/13.93.013.93.0duerpei9-0/+698
Submit the demo code of agl-test-framework The "agl-test framework" encapsulates pytest,which aims to provide a unified test set execution entrance. It supports to run various test sets, even these test sets come from different test frameworks, processing these test logs uniformly, and generating complete test report. In this way, it is convenient to test as many targets as possible, in a wide range, so that the test covers a wider range of objects, and is more comprehensive. At present, we plan to support the following test sets in "agl-test": 1. Transplant test sets under Fuego and AGL-JTA 2. Retain the test set under pyagl and agl-ptest (so will depend on "agl-ptest") 3. Migrate new test sets (with upstream) 4. Append new test sets (without upstream) The output of test run is summarized by levels. The first level is the summary of all test sets, and the second level is the summary of a single test set. Now, they are displayed in HTML format, and other formats also can be considered later. Bug-AGL: SPEC-4345 Signed-off-by: duerpei <duep.fnst@fujitsu.com> Change-Id: I25dfedcf8cdd373544c4fae677330defb5d21840
2022-06-28Add .gitreview fileqiutt1-0/+5
Bug-AGL: SPEC-4345 Signed-off-by: qiutt <qiutt@fujitsu.com> Change-Id: I916ac11cbb908205f256d308ba6d496651880cee
2022-06-27Initial empty repositorylamprey_12.1.4lamprey/12.1.412.1.4Jan-Simon Moeller0-0/+0