diff options
| author |   Corentin LABBE <clabbe@baylibre.com> | 2020-12-02 10:10:34 +0000 |
|---|---|---|
| committer |   Jan-Simon Moeller <jsmoeller@linuxfoundation.org> | 2020-12-04 13:54:06 +0000 |
| commit | 36f5fc1c9ee8c69dfe97b199e669d4998b971d0c (patch) | |
| tree | f5515e3f51e1a4ddf5b285009973e2bcb67d3749 /jjb/common/include-agl-lava-jobs-submit.sh | |
| parent | afd2602515b95bd74822d75c5b2895eb93489464 (diff) | |
SPEC-3705: handle lavacli wait errors
Sometime lavacli wait exit with an error.
Until the LAVA/lavacli bug is fixed, let's try to handle the situation.
Since the error seems to happen when the job end, let's just check if
the job ended, in the other case retry to wait.
Change-Id: I7b7b1aee0dcc2ecc2ef971cc40c29374307f6b0f
Bug-AGL: SPEC-3705
Signed-off-by: Corentin LABBE <clabbe@baylibre.com>
Diffstat (limited to 'jjb/common/include-agl-lava-jobs-submit.sh')
| -rw-r--r-- | jjb/common/include-agl-lava-jobs-submit.sh | 24 |
1 files changed, 23 insertions, 1 deletions
diff --git a/jjb/common/include-agl-lava-jobs-submit.sh b/jjb/common/include-agl-lava-jobs-submit.sh index 08b2f4e8..b2769c97 100644 --- a/jjb/common/include-agl-lava-jobs-submit.sh +++ b/jjb/common/include-agl-lava-jobs-submit.sh @@ -57,9 +57,29 @@ sed -i -e "s/ /\\\ /g" .msg #eval ssh -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" -p 29418 agl-jobbuilder@gerrit.automotivelinux.org gerrit review ${GERRIT_CHANGE_NUMBER},${GERRIT_PATCHSET_NUMBER} -n NONE --message='"$(cat .msg)"' || true #set +x +lavacli --version + +set +e # use lavacli wait job instead of the polling. -lavacli -i $lab jobs wait $job_id --timeout 7200 || true +lavacli -i $lab jobs wait $job_id --timeout 7200 +if [ $? -ne 0 ];then + retry=1 + # retry a few times with a small delay + while [ $retry -le 5 ] ; do + sleep 5 + # Work aroung SPEC-3705 "Unable to call 'jobs.wait': [Errno 2] No such file or directory" + lavacli -i $lab jobs show $job_id > status + grep -q 'state.*Finished' status + if [ $? -ne 0 ];then + # retry + retry=$(( $retry + 1 )) + lavacli -i $lab jobs wait $job_id --timeout 7200 || true + else + break + fi + done +fi lavacli -i $lab jobs show $job_id --yaml > $JOB_STATUS state=$(grep ^state: $JOB_STATUS| cut -d' ' -f2) if [ "$state" == "Finished" ]; then @@ -67,6 +87,8 @@ if [ "$state" == "Finished" ]; then echo "LAVA job $job_id completed with status: $status" fi +set -e + echo "####" echo "#### Start: Output from LAVA job $job_id ####" echo "####" |