diff options
Diffstat (limited to 'external/meta-updater')
69 files changed, 935 insertions, 758 deletions
diff --git a/external/meta-updater/.gitignore b/external/meta-updater/.gitignore index 8d35cb32..147f1629 100644 --- a/external/meta-updater/.gitignore +++ b/external/meta-updater/.gitignore @@ -1,2 +1,3 @@ __pycache__ *.pyc +.idea/ diff --git a/external/meta-updater/.gitlab-ci.yml b/external/meta-updater/.gitlab-ci.yml index 6ad00ea3..f0c08db7 100644 --- a/external/meta-updater/.gitlab-ci.yml +++ b/external/meta-updater/.gitlab-ci.yml @@ -2,6 +2,7 @@ stages: - docker - checkout - test + - trigger variables: BITBAKE_IMAGE: ${CI_REGISTRY_IMAGE}:ci-master-bitbake @@ -45,6 +46,17 @@ Build core-image-minimal: except: - pushes +Build core-image-minimal (rpi): + extends: .bitbake + + stage: test + variables: + TEST_BUILD_DIR: 'build-core-image-minimal-rpi' + BITBAKE_TARGETS: 'core-image-minimal' + TEST_MACHINE: 'raspberrypi3' + except: + - pushes + Oe-selftest qemux86_64: extends: .oe-selftest @@ -89,3 +101,15 @@ Ptest qemux86_64: only: variables: - $OE_PTEST + +# -- otf + +trigger-otf-on-pr: + stage: trigger + when: always + except: + - pushes + - schedules + script: + - apk add --no-cache curl + - curl -X POST -F "token=$CI_JOB_TOKEN" -F "ref=master" -F "variables[BITBAKE_JOB_ONLY]=true" -F "variables[BITBAKE_ENV]=thud" -F "variables[PROJECT_NAME]=meta-updater" -F "variables[PROJECT_SHA]=$CI_COMMIT_SHA" https://main.gitlab.in.here.com/api/v4/projects/163/trigger/pipeline diff --git a/external/meta-updater/CONTRIBUTING.adoc b/external/meta-updater/CONTRIBUTING.adoc index 0b404382..24916ccd 100644 --- a/external/meta-updater/CONTRIBUTING.adoc +++ b/external/meta-updater/CONTRIBUTING.adoc @@ -1,17 +1,9 @@ = Contributing +:aktualizr-docsroot: https://github.com/advancedtelematic/aktualizr/tree/master/docs/ota-client-guide/modules/ROOT/pages/ -We welcome pull requests from anyone. The master branch is the primary branch for development, and if you wish to add new functionality, it probably belongs there. We attempt to maintain recent previous branches and welcome bug fixes and backports for those. Currently, the actively maintained branches are: +We welcome pull requests from anyone. The master branch is the primary branch for development, and if you wish to add new functionality, it probably belongs there. We attempt to maintain recent release branches and welcome bug fixes and backports for those. Please see the xref:{aktualizr-docsroot}yocto-release-branches.adoc[release branches] documentation for the current list of supported branches. -* thud -* sumo -* rocko - -Previously, some older branches were also regularly supported, and while they should still be stable, they have not been updated or actively maintained for a while. These branches include: - -* pyro -* morty - -If you are developing with meta-updater, it may be helpful to read the README and other documentation for link:README.adoc[this repo], https://github.com/advancedtelematic/aktualizr[aktualizr], and the https://github.com/advancedtelematic/updater-repo/[updater-repo], particularly the sections about development and debugging. +If you are developing with meta-updater, it may be helpful to read the README and other documentation for xref:README.adoc[this repo], https://github.com/advancedtelematic/aktualizr[aktualizr], and the link:https://github.com/advancedtelematic/updater-repo/[updater-repo], particularly the sections about development and debugging. == Developer Certificate of Origin (DCO) @@ -23,7 +15,7 @@ New pull requests will automatically be checked by the https://probot.github.io/ * OTA-enabled build succeeds for at least one platform, the resulting image boots, and an update can be installed. This check is absolutely necessary for every pull request unless it only touches documentation. * If your change touches platform code (like `classes/sota_<platform>.bbclass`), please check building and updating on this particular platform. -* oe-selftest succeeds. To test meta-updater, run `oe-selftest -r updater` from a build directory with `MACHINE` set to `qemux86-64`. See the link:README.adoc#qa-with-oe-selftest[relevant section of the README] for more details. +* oe-selftest succeeds. To test meta-updater, run `oe-selftest -r updater` from a build directory with `MACHINE` set to `qemux86-64`. See the link:{aktualizr-docsroot}meta-updater-testing.adoc#qa-with-oe-selftest[relevant section of the README] for more details. * Updates are forwards- and backwards-compatible. You should be able to update an OTA-enabled build before the change is applied to the version with change applied and vice versa. One should pay double attention to the compatibility when bootloader code is affected. * The patch/branch should be based on the latest version of the target branch. This may mean that rebasing is necessary if other PRs are merged before yours is approved. diff --git a/external/meta-updater/README.adoc b/external/meta-updater/README.adoc index b047f914..1f18af6b 100644 --- a/external/meta-updater/README.adoc +++ b/external/meta-updater/README.adoc @@ -1,25 +1,20 @@ = meta-updater :toc: macro :toc-title: +:devguide-docsroot: https://docs.ota.here.com/ota-client/latest/ +:getstarted-docsroot: https://docs.ota.here.com/getstarted/dev/ -This layer enables over-the-air updates (OTA) with https://github.com/ostreedev/ostree[OSTree] and https://github.com/advancedtelematic/aktualizr[Aktualizr]. +Meta-updater is a link:https://www.yoctoproject.org/software-overview/layers/[Yocto layer] that enables over-the-air updates (OTA) with https://github.com/ostreedev/ostree[OSTree] and https://github.com/advancedtelematic/aktualizr[Aktualizr] -- the default client for link:https://www.here.com/products/automotive/ota-technology[HERE OTA Connect]. https://github.com/ostreedev/ostree[OSTree] is a tool for atomic full file system upgrades with rollback capability. OSTree has several advantages over traditional dual-bank systems, but the most important one is that it minimizes network bandwidth and data storage footprint by sharing files with the same contents across file system deployments. -https://github.com/advancedtelematic/aktualizr[Aktualizr] (and https://github.com/advancedtelematic/rvi_sota_client[RVI SOTA client]) add authentication and provisioning capabilities to OTA and are integrated with OSTree. You can connect with these open-source applications or sign up for a free account at https://connect.ota.here.com/[HERE OTA Connect] to get started. +https://github.com/advancedtelematic/aktualizr[Aktualizr] implements https://uptane.github.io/uptane-standard/uptane-standard.html[Uptane], supports device authentication and provisioning, and is integrated with OSTree. You can connect aktualizr to your own server solution or sign up for a free account at https://connect.ota.here.com/[HERE OTA Connect] to get started. -[discrete] -== Table of Contents - -toc::[] - -== Build +== Quickstart -=== Quickstart +If you don't already have a Yocto project that you want to add OTA to, you can use the xref:{getstarted-docsroot}get-started.html[HERE OTA Connect Quickstart] project to rapidly get up and running on a xref:{getstarted-docsroot}raspberry-pi.html[Raspberry Pi] or with xref:{getstarted-docsroot}qemuvirtualbox.html[QEMU]. It takes a standard https://www.yoctoproject.org/tools-resources/projects/poky[poky] distribution, and adds OTA and OSTree capabilities. -If you don't already have a Yocto project that you want to add OTA to, you can use the https://docs.atsgarage.com/quickstarts/raspberry-pi.html[HERE OTA Connect Quickstart] project to rapidly get up and running on a Raspberry Pi. It takes a standard https://www.yoctoproject.org/tools-resources/projects/poky[poky] distribution, and adds OTA and OSTree capabilities. - -=== Dependencies +== Dependencies In addition to the link:https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#required-packages-for-the-build-host[standard Yocto dependencies], meta-updater generally requires a few additional dependencies, depending on your use case and target platform. To install these additional packages on Debian/Ubuntu, run this: @@ -33,247 +28,49 @@ To build for https://github.com/advancedtelematic/meta-updater-minnowboard[Minno sudo apt install ovmf .... -=== Adding meta-updater capabilities to your build - -If you already have a Yocto-based project and you want to add atomic filesystem updates to it, you just need to do three things: - -1. Clone the `meta-updater` layer and add it to your https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#structure-build-conf-bblayers.conf[bblayers.conf]. -2. Clone BSP integration layer (`meta-updater-$\{PLATFORM}`, e.g. https://github.com/advancedtelematic/meta-updater-raspberrypi[meta-updater-raspberrypi]) and add it to your `conf/bblayers.conf`. If your board isn't supported yet, you could write a BSP integration for it yourself. See the <<Adding support for your board>> section for the details. -3. Set up your https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#var-DISTRO[distro]. If you are using "poky", the default distro in Yocto, you can change it in your `conf/local.conf` to "poky-sota". Alternatively, if you are using your own or third party distro configuration, you can add `INHERIT += " sota"` to it, thus combining capabilities of your distro with meta-updater features. - -You can then build your image as usual, with bitbake. After building the root file system, bitbake will then create an https://ostree.readthedocs.io/en/latest/manual/adapting-existing/[OSTree-enabled version] of it, commit it to your local OSTree repo and (optionally) push it to a remote server. Additionally, a live disk image will be created (normally named `$\{IMAGE_NAME}.-sdimg-ota` e.g. `core-image-raspberrypi3.rpi-sdimg-ota`). You can control this behaviour through <<sota-related-variables-in-localconf,variables in your local.conf>>. - -=== Build in AGL - -With AGL you can just add agl-sota feature while configuring your build environment: - -.... -source meta-agl/scripts/aglsetup.sh -m porter agl-demo agl-appfw-smack agl-devel agl-sota -.... - -You can then run: - -.... -bitbake agl-demo-platform -.... - -and get as a result an `ostree_repo` folder in your images directory (`tmp/deploy/images/$\{MACHINE}/ostree_repo`). It will contain: - -* your OSTree repository, with the rootfs committed as an OSTree deployment, -* an `ota-ext4` bootstrap image, which is an OSTree physical sysroot as a burnable filesystem image, and optionally -* some machine-dependent live images (e.g. `.wic` for Raspberry Pi or `.porter-sdimg-ota` Renesas Porter board). - -Although `aglsetup.sh` hooks provide reasonable defaults for SOTA-related variables, you may want to tune some of them. - -=== Build problems - -Ubuntu users that encounter an error due to missing `Python.h` should install `libpython2.7-dev` on their host machine. - -== Supported boards - -Currently supported platforms are - -* https://github.com/advancedtelematic/meta-updater-raspberrypi[Raspberry Pi3] -* https://github.com/advancedtelematic/meta-updater-minnowboard[Minnowboard] -* https://github.com/advancedtelematic/meta-updater-qemux86-64[Native QEMU emulation] - -=== Adding support for your board - -If your board isn't supported yet, you can add board integration code yourself. The main purpose of this code is to provide a bootloader that will be able to use https://ostree.readthedocs.io/en/latest/manual/atomic-upgrades/[OSTree's boot directory]. In the meta-updater integration layers we have written so far, the basic steps are: - -1. Make the board boot into http://www.denx.de/wiki/U-Boot[U-Boot] -2. Make U-boot import variables from /boot/loader/uEnv.txt and load the kernel with initramfs and kernel command line arguments according to what is set in this file. - -You may take a look into https://github.com/advancedtelematic/meta-updater-minnowboard[Minnowboard] or https://github.com/advancedtelematic/meta-updater-raspberrypi[Raspberry Pi] integration layers for examples. - -Although we have focused on U-Boot and GRUB so far, other bootloaders can be configured to work with OSTree as well. - -Your images will also need network connectivity to be able to reach an actual OTA backend. Our 'poky-sota' distribution does not mandate or install a default network manager but our supported platforms use the `virtual/network-configuration` recipe, which can be used as a starting example. - -== SOTA-related variables in local.conf - -* `OSTREE_REPO` - path to your OSTree repository. Defaults to `$\{DEPLOY_DIR_IMAGE}/ostree_repo` -* `OSTREE_OSNAME` - OS deployment name on your target device. For more information about deployments and osnames see the https://ostree.readthedocs.io/en/latest/manual/deployment/[OSTree documentation]. Defaults to "poky". -* `OSTREE_COMMIT_BODY` - Message attached to OSTree commit. Empty by default. -* `OSTREE_COMMIT_SUBJECT` - Commit subject used by OSTree. Defaults to `Commit-id: ${IMAGE_NAME}` -* `OSTREE_UPDATE_SUMMARY` - Set this to '1' to update summary of OSTree repository on each commit. '0' by default. -* `OSTREE_DEPLOY_DEVICETREE` - Set this to '1' to include devicetree(s) to boot -* `GARAGE_SIGN_AUTOVERSION` - Set this to '1' to automatically fetch the last version of the garage tools installed by the aktualizr-native. Otherwise use the fixed version specified in the recipe. -* `INITRAMFS_IMAGE` - initramfs/initrd image that is used as a proxy while booting into OSTree deployment. Do not change this setting unless you are sure that your initramfs can serve as such a proxy. -* `SOTA_PACKED_CREDENTIALS` - when set, your ostree commit will be pushed to a remote repo as a bitbake step. This should be the path to a zipped credentials file in https://github.com/advancedtelematic/aktualizr/blob/master/docs/credentials.adoc[the format accepted by garage-push]. -* `SOTA_DEPLOY_CREDENTIALS` - when set to '1' (default value), deploys credentials to the built image. Override it in `local.conf` to built a generic image that can be provisioned manually after the build. -* `SOTA_CLIENT_PROV` - which provisioning method to use. Valid options are `aktualizr-shared-prov`, `aktualizr-device-prov`, and `aktualizr-device-prov-hsm`. For more information on these provisioning methods, see the https://docs.ota.here.com/client-config/client-provisioning-methods.html[OTA Connect documentation]. The default is `aktualizr-shared-prov`. This can also be set to an empty string to avoid using a provisioning recipe. -* `SOTA_CLIENT_FEATURES` - extensions to aktualizr. The only valid options are `hsm` (to build with HSM support) and `secondary-network` (to set up a simulated 'in-vehicle' network with support for a primary node with a DHCP server and a secondary node with a DHCP client). -* `SOTA_SECONDARY_CONFIG` - a file containing JSON configuration for secondaries. It will be installed into `/etc/sota/ecus` on the device and automatically provided to aktualizr. See link:https://github.com/advancedtelematic/aktualizr/blob/master/docs/posix-secondaries-bitbaking.adoc[here] for more details. -* `SOTA_HARDWARE_ID` - a custom hardware ID that will be written to the aktualizr config. Defaults to MACHINE if not set. -* `SOTA_MAIN_DTB` - base device tree to use with the kernel. Used together with FIT images. You can change it, and the device tree will also be changed after the update. -* `SOTA_DT_OVERLAYS` - whitespace-separated list of used device tree overlays for FIT image. This list is OSTree-updateable as well. -* `SOTA_EXTRA_CONF_FRAGS` - extra https://lxr.missinglinkelectronics.com/uboot/doc/uImage.FIT/overlay-fdt-boot.txt[configuration fragments] for FIT image. -* `RESOURCE_xxx_pn-aktualizr` - controls maximum resource usage of the aktualizr service, when `aktualizr-resource-control` is installed on the image. See <<aktualizr service resource control>> for details. -* `SOTA_POLLING_SEC` - sets polling interval for aktualizr to check for updates if aktualizr-polling-sec is included in the image. - -== Usage - -=== OSTree - -OSTree used to include a simple HTTP server as part of the ostree binary, but this has been removed in more recent versions. However, OSTree repositories are self-contained directories, and can be trivially served over the network using any HTTP server. For example, you could use Python's SimpleHTTPServer: - -.... -cd tmp/deploy/images/qemux86-64/ostree_repo -python -m SimpleHTTPServer <port> # port defaults to 8000 -.... - -You can then run ostree from inside your device by adding your repo: - -.... -# This behaves like adding a Git remote; you can name it anything -ostree remote add --no-gpg-verify my-remote http://<your-ip>:<port> - -# If OSTREE_BRANCHNAME is set in local.conf, that will be the name of the -# branch. If not set, it defaults to the value of MACHINE (e.g. qemux86-64). -ostree pull my-remote <branch> - -# poky is the OS name as set in OSTREE_OSNAME -ostree admin deploy --os=poky my-remote:<branch> -.... - -After restarting, you will boot into the newly deployed OS image. - -For example, on the raspberry pi you can try this sequence: - -.... -# add remote -ostree remote add --no-gpg-verify agl-snapshot https://download.automotivelinux.org/AGL/snapshots/master/latest/raspberrypi3/deploy/images/raspberrypi3/ostree_repo/ agl-ota - -# pull -ostree pull agl-snapshot agl-ota - -# deploy -ostree admin deploy --os=agl agl-snapshot:agl-ota -.... - -=== garage-push - -The https://github.com/advancedtelematic/aktualizr[aktualizr repo] contains a tool, garage-push, which lets you push the changes in OSTree repository generated by bitbake process. It communicates with an http server capable of querying files with HEAD requests and uploading them with POST requests. In particular, this can be used with https://connect.ota.here.com/[HERE OTA Connect]. garage-push is used as follows: - -.... -garage-push --repo=/path/to/ostree-repo --ref=mybranch --credentials=/path/to/credentials.zip -.... - -You can set `SOTA_PACKED_CREDENTIALS` in your `local.conf` to automatically synchronize your build results with a remote server. Credentials are stored in an archive as described in the https://github.com/advancedtelematic/aktualizr/blob/master/docs/credentials.adoc[aktualizr documentation]. - -=== aktualizr configuration - -https://github.com/advancedtelematic/aktualizr[Aktualizr] supports a variety of https://github.com/advancedtelematic/aktualizr/blob/master/docs/configuration.adoc[configuration options via a configuration file and the command line]. There are two primary ways to control aktualizr's configuration from meta-updater. - -First, you can set `SOTA_CLIENT_PROV` to control which provisioning recipe is used. Each recipe installs an appropriate `sota.toml` file from aktualizr according to the provisioning needs. See the <<sota-related-variables-in-localconf,SOTA-related variables in local.conf>> section for more information. - -Second, you can write recipes to install additional config files with customized options. A few recipes already exist to address common needs and provide an example: - -* link:recipes-sota/config/aktualizr-auto-reboot.bb[aktualizr-auto-reboot.bb] configures aktualizr to automatically reboot after new updates are installed in order to apply the updates immediately. This is only relevant for package managers (such as OSTree) that require a reboot to complete the installation process. If this is not enabled, you will need to reboot the system through other means. -* link:recipes-sota/config/aktualizr-disable-send-ip.bb[aktualizr-disable-send-ip.bb] disables the reporting of networking information to the server. This is enabled by default and supported by https://connect.ota.here.com/[HERE OTA Connect]. However, if you are using a different server that does not support this feature, you may want to disable it in aktualizr. -* link:recipes-sota/config/aktualizr-log-debug.bb[aktualizr-log-debug.bb] sets the log level of aktualizr to 0 (trace). The default is 2 (info). This recipe is intended for development and debugging purposes. - -To use these recipes, you will need to add them to your image with a line such as `IMAGE_INSTALL_append = " aktualizr-log-debug "` in your `local.conf`. - -=== aktualizr service resource control - -With systemd based images, it is possible to set resource policies for the aktualizr service. The main use case is to provide a safeguard against resource exhaustion during an unforeseen failure scenario. - -To enable it, install `aktualizr-resource-control` on the target image and optionally override the default resource limits set in link:recipes-sota/aktualizr/aktualizr_git.bb[aktualizr_git.bb], from your `local.conf`. - -For example: - -.... -IMAGE_INSTALL_append += " aktualizr-resource-control " -RESOURCE_CPU_WEIGHT_pn-aktualizr = "50" -.... - -== Development configuration - -There are a few settings that can be controlled in `local.conf` to simplify the development process: - -[options="header"] -|====================== -| Option | Effect -| `require classes/sota_bleeding.inc` | Build the latest head (by default, using the master branch) of Aktualizr -| `BRANCH_pn-aktualizr = "mybranch"` - -`BRANCH_pn-aktualizr-native = "mybranch"` | Build `mybranch` of Aktualizr. Note that both of these need to be set. This is normally used in conjunction with `require classes/sota_bleeding.inc` -| `SRCREV_pn-aktualizr = "1004efa3f86cef90c012b34620992b5762b741e3"` - -`SRCREV_pn-aktualizr-native = "1004efa3f86cef90c012b34620992b5762b741e3"` | Build the specified revision of Aktualizr. Note that both of these need to be set. This can be used in conjunction with `BRANCH_pn-aktualizr` and `BRANCH_pn-aktualizr-native` but will conflict with `require classes/sota_bleeding.inc` -| `TOOLCHAIN_HOST_TASK_append = " nativesdk-cmake "` | Use with `bitbake -c populate_sdk core-image-minimal` to build an SDK. See the https://github.com/advancedtelematic/aktualizr#developing-against-an-openembedded-system[aktualizr repo] for more information. -|====================== - -=== Overriding target version -*Warning: overriding target version is a dangerous operation, make sure you understand this section completely before doing it.* - -Every time you build an image with `SOTA_PACKED_CREDENTIALS` set, a new entry in your Uptane metadata is created and you can see it in the OTA Garage UI if you're using one. Normally this version will be equal to OSTree hash of your root file system. If you want it to be different though you can override is using one of two methods: - -1. Set `GARAGE_TARGET_VERSION` variable in your `local.conf`. -2. Write a recipe or a bbclass to write the desired version to `${STAGING_DATADIR_NATIVE}/target_version`. An example of such bbclass can be found in `classes/target_version_example.bbclass`. - -Please note that [target name, target version] pairs are expected to be unique in the system. If you build a new target with the same target version as a previously built one, the old package will be overwritten on the update server. It can have unpredictable effect on devices that have this version installed, and it is not guaranteed that information will be reported correctly for such devices or that you will be able to update them (we're doing our best though). The easiest way to avoid problems is to make sure that your overriding version is as unique as an OSTree commit hash. - -== QA with oe-selftest +[discrete] +== Table of Contents -This layer relies on the test framework oe-selftest for quality assurance. Currently, you will need to run this in a build directory with `MACHINE` set to `qemux86-64`. Follow the steps below to run the tests: +The following documentation focuses on tasks that involve the meta-updater layer. If you want to get an idea of the overall developer workflow in OTA Connect, see the link:{devguide-docsroot}index.html[OTA Connect Developer Guide]. -1. Append the line below to `conf/local.conf` to disable the warning about supported operating systems: +* xref:{devguide-docsroot}supported-boards.html[Supported boards] + -``` -SANITY_TESTED_DISTROS = "" -``` - -2. If your image does not already include an ssh daemon such as dropbear or openssh, add this line to `conf/local.conf` as well: +Find out if your board is supported and learn about the minimum hardware requirements. + -``` -IMAGE_INSTALL_append = " dropbear " -``` - -3. Some tests require that `SOTA_PACKED_CREDENTIALS` is set in your `conf/local.conf`. See the <<sota-related-variables-in-localconf,SOTA-related variables in local.conf>> section. - -4. To be able to build an image for the GRUB tests, you will need to install the ovmf package as described in the <<Dependencies,dependencies>>. - -5. Run oe-selftest: +* xref:{devguide-docsroot}build-agl.html[Build an Automotive Grade Linux image] + -``` -oe-selftest -r updater_native updater_qemux86_64 updater_minnowboard updater_raspberrypi updater_qemux86_64_ptest -``` - -For more information about oe-selftest, including details about how to run individual test modules or classes, please refer to the https://wiki.yoctoproject.org/wiki/Oe-selftest[Yocto Project wiki]. - -== Aktualizr test suite with ptest - -The meta-updater layer includes support for running parts of the aktualizr test suite on deployed devices through link:https://wiki.yoctoproject.org/wiki/Ptest[Yocto's ptest functionality]. Since it adds significant build time cost, it is currently disabled by default. To enable it, add the following to your `conf/local.conf`: - -``` -PTEST_ENABLED_pn-aktualizr = "1" -IMAGE_INSTALL_append += " aktualizr-ptest ptest-runner " -``` - -Be aware that it will add several hundreds of MB to the generated file system. - -The aktualizr tests will now be part of the deployed ptest suite, which can be run by calling `ptest-runner`. Alternatively, the required files and run script can be found in `/usr/lib/aktualizr/ptest`. - -== Manual provisoning - -As described in <<sota-related-variables-in-localconf,SOTA-related variables in local.conf>> section you can set `SOTA_DEPLOY_CREDENTIALS` to `0` to prevent deploying credentials to the built `wic` image. In this case you get a generic image that you can use e.g. on a production line to flash a series of devices. The cost of this approach is that this image is half-baked and should be provisioned before it can connect to the backend. - -Provisioning procedure depends on your provisioning recipe, i.e. the value of `SOTA_CLIENT_PROV` (equal to `aktualizr-shared-prov` by default): - -* For `aktualizr-shared-prov` put your `credentials.zip` to `/var/sota/sota_provisioning_credentials.zip` on the filesystem of a running device. If you have the filesystem of our device mounted to your build machine, prefix all paths with `/ostree/deploy/poky` as in `/ostree/deploy/poky/var/sota/sota_provisioning_credentials.zip`. -* For `aktualizr-device-prov` -** put URL to the backend server (together with protocol prefix and port number) at `/var/sota/gateway.url`. If you're using HERE OTA Connect, you can find the URL in the `autoprov.url` file in your credentials archive. -** put client certificate, private key and root CA certificate (for the *server*, not for the *device*) at `/var/sota/import/client.pem`, `/var/sota/import/pkey.pem` and `/var/sota/import/root.crt` respectively. -* For `aktualizr-device-prov-hsm` -** put URL to the server backend (together with protocol prefix and port number) at `/var/sota/gateway.url`. If you're using HERE OTA Connect, you can find the URL in the `autoprov.url` file in your credentials archive. -** put root CA certificate (for the *server*, not for the *device*) at `/var/sota/import/root.crt`. -** put client certificate and private key to slots 1 and 2 of the PKCS#11-compatible device. +Learn how to use this layer as part of AGL. ++ +* xref:{devguide-docsroot}add-ota-functonality-existing-yocto-project.html[Add OTA functionality to an existing Yocto project] ++ +Learn how to add this layer to your own Yocto project. ++ +* xref:{devguide-docsroot}build-configuration.html[SOTA-related variables in local.conf] ++ +Learn how to configure OTA-related functionality when building images, including how to install custom versions of aktualizr. ++ +* xref:{devguide-docsroot}recommended-clientconfig.html[Recommended configuration] ++ +Learn how to optimize your build for development or production. ++ +* xref:{devguide-docsroot}client-provisioning-methods.html[Provisoning methods] ++ +Learn more about the methods for provisioning devices. For more detail, you may also want to read about how to xref:{devguide-docsroot}enable-device-cred-provisioning.html[enable device credential provisioning] or how to xref:{devguide-docsroot}simulate-device-cred-provtest.html[simulate it for testing]. ++ +* xref:{devguide-docsroot}meta-updater-usage.html[Advanced usage] ++ +Learn about the `garage-push` and `garage-sign` utilities, aktualizr configuration recipes, and service resource control. ++ +* xref:{devguide-docsroot}meta-updater-testing.html[Testing with oe-selftest and ptest] ++ +Learn how to use the `oe-selftest` framework for quality assurance and how to run the aktualizr test suite via ptest. ++ +* xref:{devguide-docsroot}troubleshooting.html[Troubleshooting] ++ +Get help on common problems. == License -This code is licensed under the link:COPYING.MIT[MIT license], a copy of which can be found in this repository. All code is copyright HERE Europe B.V., 2016-2019. +This code is licensed under the link:COPYING.MIT[MIT license], a copy of which can be found in this repository. All code is copyright HERE Europe B.V., 2016-2020. We require that contributors accept the terms of Linux Foundation's link:https://developercertificate.org/[Developer Certificate of Origin]. Please see the https://github.com/advancedtelematic/aktualizr/blob/master/CONTRIBUTING.md[contribution instructions of aktualizr] for more information. diff --git a/external/meta-updater/classes/image_repo_manifest.bbclass b/external/meta-updater/classes/image_repo_manifest.bbclass index c2e7056d..72dc28fe 100644 --- a/external/meta-updater/classes/image_repo_manifest.bbclass +++ b/external/meta-updater/classes/image_repo_manifest.bbclass @@ -9,8 +9,6 @@ # For more information, see: # https://web.archive.org/web/20161224194009/https://wiki.cyanogenmod.org/w/Doc:_Using_manifests -HOSTTOOLS_NONFATAL += " repo " - # Write build information to target filesystem buildinfo_manifest () { if [ $(which repo) ]; then diff --git a/external/meta-updater/classes/image_types_ostree.bbclass b/external/meta-updater/classes/image_types_ostree.bbclass index 56d4d76c..533d338b 100644 --- a/external/meta-updater/classes/image_types_ostree.bbclass +++ b/external/meta-updater/classes/image_types_ostree.bbclass @@ -5,6 +5,7 @@ OSTREE_KERNEL ??= "${KERNEL_IMAGETYPE}" OSTREE_ROOTFS ??= "${WORKDIR}/ostree-rootfs" OSTREE_COMMIT_SUBJECT ??= "Commit-id: ${IMAGE_NAME}" OSTREE_COMMIT_BODY ??= "" +OSTREE_COMMIT_VERSION ??= "${DISTRO_VERSION}" OSTREE_UPDATE_SUMMARY ??= "0" OSTREE_DEPLOY_DEVICETREE ??= "0" @@ -13,11 +14,11 @@ BUILD_OSTREE_TARBALL ??= "1" SYSTEMD_USED = "${@oe.utils.ifelse(d.getVar('VIRTUAL-RUNTIME_init_manager') == 'systemd', 'true', '')}" IMAGE_CMD_TAR = "tar --xattrs --xattrs-include=*" -CONVERSION_CMD_tar = "touch ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}; ${IMAGE_CMD_TAR} --numeric-owner -cf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.tar -C ${OTA_IMAGE_ROOTFS} . || [ $? -eq 1 ]" +CONVERSION_CMD_tar = "touch ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}; ${IMAGE_CMD_TAR} --numeric-owner -cf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.tar -C ${TAR_IMAGE_ROOTFS} . || [ $? -eq 1 ]" CONVERSIONTYPES_append = " tar" REQUIRED_DISTRO_FEATURES = "usrmerge" -OTA_IMAGE_ROOTFS_task-image-ostree = "${OSTREE_ROOTFS}" +TAR_IMAGE_ROOTFS_task-image-ostree = "${OSTREE_ROOTFS}" do_image_ostree[dirs] = "${OSTREE_ROOTFS}" do_image_ostree[cleandirs] = "${OSTREE_ROOTFS}" do_image_ostree[depends] = "coreutils-native:do_populate_sysroot virtual/kernel:do_deploy ${INITRAMFS_IMAGE}:do_image_complete" @@ -78,15 +79,15 @@ IMAGE_CMD_ostree () { if [ "$(ls -A $dir)" ]; then bbwarn "Data in /$dir directory is not preserved by OSTree. Consider moving it under /usr" fi - - if [ -n "${SYSTEMD_USED}" ]; then - echo "d /var/rootdirs/${dir} 0755 root root -" >>${tmpfiles_conf} - else - echo "mkdir -p /var/rootdirs/${dir}; chown 755 /var/rootdirs/${dir}" >>${tmpfiles_conf} - fi rm -rf ${dir} - ln -sf var/rootdirs/${dir} ${dir} fi + + if [ -n "${SYSTEMD_USED}" ]; then + echo "d /var/rootdirs/${dir} 0755 root root -" >>${tmpfiles_conf} + else + echo "mkdir -p /var/rootdirs/${dir}; chown 755 /var/rootdirs/${dir}" >>${tmpfiles_conf} + fi + ln -sf var/rootdirs/${dir} ${dir} done if [ -d root ] && [ ! -L root ]; then @@ -95,15 +96,40 @@ IMAGE_CMD_ostree () { fi if [ -n "${SYSTEMD_USED}" ]; then - echo "d /var/roothome 0755 root root -" >>${tmpfiles_conf} + echo "d /var/roothome 0700 root root -" >>${tmpfiles_conf} else - echo "mkdir -p /var/roothome; chown 755 /var/roothome" >>${tmpfiles_conf} + echo "mkdir -p /var/roothome; chown 700 /var/roothome" >>${tmpfiles_conf} fi rm -rf root ln -sf var/roothome root fi + if [ -d usr/local ] && [ ! -L usr/local ]; then + if [ "$(ls -A usr/local)" ]; then + bbfatal "Data in /usr/local directory is not preserved by OSTree." + fi + rm -rf usr/local + fi + + if [ -n "${SYSTEMD_USED}" ]; then + echo "d /var/usrlocal 0755 root root -" >>${tmpfiles_conf} + else + echo "mkdir -p /var/usrlocal; chown 755 /var/usrlocal" >>${tmpfiles_conf} + fi + + dirs="bin etc games include lib man sbin share src" + + for dir in ${dirs}; do + if [ -n "${SYSTEMD_USED}" ]; then + echo "d /var/usrlocal/${dir} 0755 root root -" >>${tmpfiles_conf} + else + echo "mkdir -p /var/usrlocal/${dir}; chown 755 /var/usrlocal/${dir}" >>${tmpfiles_conf} + fi + done + + ln -sf ../var/usrlocal usr/local + if [ "${KERNEL_IMAGETYPE}" = "fitImage" ]; then # this is a hack for ostree not to override init= in kernel cmdline - # make it think that the initramfs is present (while it is in FIT image) @@ -143,7 +169,9 @@ IMAGE_CMD_ostreecommit () { --skip-if-unchanged \ --branch=${OSTREE_BRANCHNAME} \ --subject="${OSTREE_COMMIT_SUBJECT}" \ - --body="${OSTREE_COMMIT_BODY}" + --body="${OSTREE_COMMIT_BODY}" \ + --add-metadata-string=version="${OSTREE_COMMIT_VERSION}" \ + --bind-ref="${OSTREE_BRANCHNAME}-${IMAGE_BASENAME}" if [ "${OSTREE_UPDATE_SUMMARY}" = "1" ]; then ostree --repo=${OSTREE_REPO} summary -u @@ -160,13 +188,20 @@ IMAGE_CMD_ostreecommit () { IMAGE_TYPEDEP_ostreepush = "ostreecommit" do_image_ostreepush[depends] += "aktualizr-native:do_populate_sysroot ca-certificates-native:do_populate_sysroot" IMAGE_CMD_ostreepush () { - # Print warnings if credetials are not set or if the file has not been found. + # send a copy of the repo manifest to backend if available + local SEND_MANIFEST="" + # check if garage-push supports the --repo-manifest option before trying + if $(garage-push --help | grep -q '^\s*--repo-manifest') && [ -f ${IMAGE_ROOTFS}${sysconfdir}/manifest.xml ]; then + SEND_MANIFEST="--repo-manifest ${IMAGE_ROOTFS}${sysconfdir}/manifest.xml" + fi + if [ -n "${SOTA_PACKED_CREDENTIALS}" ]; then if [ -e ${SOTA_PACKED_CREDENTIALS} ]; then garage-push -vv --repo=${OSTREE_REPO} \ --ref=${OSTREE_BRANCHNAME} \ --credentials=${SOTA_PACKED_CREDENTIALS} \ - --cacert=${STAGING_ETCDIR_NATIVE}/ssl/certs/ca-certificates.crt + --cacert=${STAGING_ETCDIR_NATIVE}/ssl/certs/ca-certificates.crt \ + $SEND_MANIFEST else bbwarn "SOTA_PACKED_CREDENTIALS file does not exist." fi @@ -203,19 +238,29 @@ IMAGE_CMD_garagesign () { target_version=${ostree_target_hash} if [ -n "${GARAGE_TARGET_VERSION}" ]; then target_version=${GARAGE_TARGET_VERSION} - bbwarn "Target version is overriden with GARAGE_TARGET_VERSION variable. It is a dangerous operation, make sure you've read the respective secion in meta-updater/README.adoc" + bbwarn "Target version is overriden with GARAGE_TARGET_VERSION variable. This is a dangerous operation! See https://docs.ota.here.com/ota-client/latest/build-configuration.html#_overriding_target_version" elif [ -e "${STAGING_DATADIR_NATIVE}/target_version" ]; then target_version=$(cat "${STAGING_DATADIR_NATIVE}/target_version") - bbwarn "Target version is overriden with target_version file. It is a dangerous operation, make sure you've read the respective secion in meta-updater/README.adoc" + bbwarn "Target version is overriden with target_version file. This is a dangerous operation! See https://docs.ota.here.com/ota-client/latest/build-configuration.html#_overriding_target_version" fi # Push may fail due to race condition when multiple build machines try to push simultaneously # in which case targets.json should be pulled again and the whole procedure repeated push_success=0 - target_url="" - if [ -n "${GARAGE_TARGET_URL}" ]; then - target_url='--url ${GARAGE_TARGET_URL}' - fi + target_url="" + if [ -n "${GARAGE_TARGET_URL}" ]; then + target_url="--url ${GARAGE_TARGET_URL}" + fi + target_expiry="" + if [ -n "${GARAGE_TARGET_EXPIRES}" ] && [ -n "${GARAGE_TARGET_EXPIRE_AFTER}" ]; then + bbfatal "Both GARAGE_TARGET_EXPIRES and GARAGE_TARGET_EXPIRE_AFTER are set. Only one can be set at a time." + elif [ -n "${GARAGE_TARGET_EXPIRES}" ]; then + target_expiry="--expires ${GARAGE_TARGET_EXPIRES}" + elif [ -n "${GARAGE_TARGET_EXPIRE_AFTER}" ]; then + target_expiry="--expire-after ${GARAGE_TARGET_EXPIRE_AFTER}" + else + target_expiry="--expire-after 1M" + fi for push_retries in $( seq 3 ); do garage-sign targets pull --repo tufrepo \ @@ -229,8 +274,15 @@ IMAGE_CMD_garagesign () { ${target_url} \ --sha256 ${ostree_target_hash} \ --hardwareids ${SOTA_HARDWARE_ID} + if [ -n "${GARAGE_CUSTOMIZE_TARGET}" ]; then + bbplain "Running command(${GARAGE_CUSTOMIZE_TARGET}) to customize target" + ${GARAGE_CUSTOMIZE_TARGET} \ + ${GARAGE_SIGN_REPO}/tufrepo/roles/unsigned/targets.json \ + ${GARAGE_TARGET_NAME}-${target_version} + fi garage-sign targets sign --repo tufrepo \ --home-dir ${GARAGE_SIGN_REPO} \ + ${target_expiry} \ --key-name=targets errcode=0 garage-sign targets push --repo tufrepo \ diff --git a/external/meta-updater/classes/image_types_ota.bbclass b/external/meta-updater/classes/image_types_ota.bbclass index 12375ec1..857161af 100644 --- a/external/meta-updater/classes/image_types_ota.bbclass +++ b/external/meta-updater/classes/image_types_ota.bbclass @@ -38,13 +38,14 @@ calculate_size () { } OTA_SYSROOT = "${WORKDIR}/ota-sysroot" -OTA_IMAGE_ROOTFS_task-image-ota = "${OTA_SYSROOT}" +TAR_IMAGE_ROOTFS_task-image-ota = "${OTA_SYSROOT}" IMAGE_TYPEDEP_ota = "ostreecommit" do_image_ota[dirs] = "${OTA_SYSROOT}" do_image_ota[cleandirs] = "${OTA_SYSROOT}" do_image_ota[depends] = "${@'grub:do_populate_sysroot' if d.getVar('OSTREE_BOOTLOADER') == 'grub' else ''} \ ${@'virtual/bootloader:do_deploy' if d.getVar('OSTREE_BOOTLOADER') == 'u-boot' else ''}" IMAGE_CMD_ota () { + export OSTREE_BOOT_PARTITION=${OSTREE_BOOT_PARTITION} ostree admin --sysroot=${OTA_SYSROOT} init-fs ${OTA_SYSROOT} ostree admin --sysroot=${OTA_SYSROOT} os-init ${OSTREE_OSNAME} mkdir -p ${OTA_SYSROOT}/boot/loader.0 diff --git a/external/meta-updater/classes/sota.bbclass b/external/meta-updater/classes/sota.bbclass index bf27b6dd..c248cfc0 100644 --- a/external/meta-updater/classes/sota.bbclass +++ b/external/meta-updater/classes/sota.bbclass @@ -1,28 +1,12 @@ DISTROOVERRIDES .= "${@bb.utils.contains('DISTRO_FEATURES', 'sota', ':sota', '', d)}" -HOSTTOOLS_NONFATAL += "java" - SOTA_CLIENT ??= "aktualizr" SOTA_CLIENT_PROV ??= "aktualizr-shared-prov" SOTA_DEPLOY_CREDENTIALS ?= "1" SOTA_HARDWARE_ID ??= "${MACHINE}" -# Translate old provisioning recipe names into the new versions. -python () { - prov = d.getVar("SOTA_CLIENT_PROV") - if prov == "aktualizr-auto-prov": - bb.warn('aktualizr-auto-prov is deprecated. Please use aktualizr-shared-prov instead.') - d.setVar("SOTA_CLIENT_PROV", "aktualizr-shared-prov") - elif prov == "aktualizr-ca-implicit-prov": - bb.warn('aktualizr-ca-implicit-prov is deprecated. Please use aktualizr-device-prov instead.') - d.setVar("SOTA_CLIENT_PROV", "aktualizr-device-prov") - elif prov == "aktualizr-hsm-prov": - bb.warn('aktualizr-hsm-prov is deprecated. Please use aktualizr-device-prov-hsm instead.') - d.setVar("SOTA_CLIENT_PROV", "aktualizr-device-prov-hsm") -} - IMAGE_INSTALL_append_sota = " ostree os-release ${SOTA_CLIENT} ${SOTA_CLIENT_PROV}" -IMAGE_CLASSES += " image_types_ostree image_types_ota" +IMAGE_CLASSES += " image_types_ostree image_types_ota image_repo_manifest" IMAGE_FSTYPES += "${@bb.utils.contains('DISTRO_FEATURES', 'sota', 'ostreepush garagesign garagecheck ota-ext4 wic', ' ', d)}" IMAGE_FSTYPES += "${@bb.utils.contains('BUILD_OSTREE_TARBALL', '1', 'ostree.tar.bz2', ' ', d)}" @@ -38,11 +22,11 @@ EXTRA_IMAGEDEPENDS_append_sota = " parted-native mtools-native dosfstools-native INITRAMFS_FSTYPES ?= "${@oe.utils.ifelse(d.getVar('OSTREE_BOOTLOADER') == 'u-boot', 'cpio.gz.u-boot', 'cpio.gz')}" # Please redefine OSTREE_REPO in order to have a persistent OSTree repo -export OSTREE_REPO ?= "${DEPLOY_DIR_IMAGE}/ostree_repo" -export OSTREE_BRANCHNAME ?= "${SOTA_HARDWARE_ID}" -export OSTREE_OSNAME ?= "poky" -export OSTREE_BOOTLOADER ??= 'u-boot' -export OSTREE_BOOT_PARTITION ??= "/boot" +OSTREE_REPO ?= "${DEPLOY_DIR_IMAGE}/ostree_repo" +OSTREE_BRANCHNAME ?= "${SOTA_HARDWARE_ID}" +OSTREE_OSNAME ?= "poky" +OSTREE_BOOTLOADER ??= 'u-boot' +OSTREE_BOOT_PARTITION ??= "/boot" INITRAMFS_IMAGE ?= "initramfs-ostree-image" @@ -51,6 +35,9 @@ GARAGE_SIGN_KEYNAME ?= "garage-key" GARAGE_TARGET_NAME ?= "${OSTREE_BRANCHNAME}" GARAGE_TARGET_VERSION ?= "" GARAGE_TARGET_URL ?= "" +GARAGE_TARGET_EXPIRES ?= "" +GARAGE_TARGET_EXPIRE_AFTER ?= "" +GARAGE_CUSTOMIZE_TARGET ?= "" SOTA_MACHINE ??="none" SOTA_MACHINE_rpi ?= "raspberrypi" @@ -63,4 +50,4 @@ SOTA_MACHINE_am335x-evm ?= "am335x-evm-wifi" SOTA_OVERRIDES_BLACKLIST = "ostree ota" SOTA_REQUIRED_VARIABLES = "OSTREE_REPO OSTREE_BRANCHNAME OSTREE_OSNAME OSTREE_BOOTLOADER OSTREE_BOOT_PARTITION GARAGE_SIGN_REPO GARAGE_TARGET_NAME" -inherit sota_sanity sota_${SOTA_MACHINE} image_repo_manifest +inherit sota_sanity sota_${SOTA_MACHINE} diff --git a/external/meta-updater/classes/sota_bleeding.inc b/external/meta-updater/classes/sota_bleeding.inc index fc5947de..77d004bf 100644 --- a/external/meta-updater/classes/sota_bleeding.inc +++ b/external/meta-updater/classes/sota_bleeding.inc @@ -1 +1,2 @@ SRCREV_pn-aktualizr ?= "${AUTOREV}" +SRCREV_pn-aktualizr-native ?= "${AUTOREV}" diff --git a/external/meta-updater/classes/sota_m3ulcb.bbclass b/external/meta-updater/classes/sota_m3ulcb.bbclass index b93cc407..e7fa9c2f 100644 --- a/external/meta-updater/classes/sota_m3ulcb.bbclass +++ b/external/meta-updater/classes/sota_m3ulcb.bbclass @@ -1,11 +1,12 @@ # Commit united image to OSTree, not just uImage OSTREE_KERNEL = "Image" -EXTRA_IMAGEDEPENDS_append_sota = " m3ulcb-ota-bootfiles" -IMAGE_BOOT_FILES_sota += "m3ulcb-ota-bootfiles/*" +EXTRA_IMAGEDEPENDS_append_sota = " renesas-ota-bootfiles" +IMAGE_BOOT_FILES_sota += "renesas-ota-bootfiles/*" OSTREE_BOOTLOADER ?= "u-boot" -UBOOT_MACHINE_sota = "m3ulcb_defconfig" + +UBOOT_MACHINE_sota = "${@d.getVar('SOC_FAMILY').split(':')[1]}_ulcb_defconfig" PREFERRED_RPROVIDER_virtual/network-configuration ?= "connman" IMAGE_INSTALL_append_sota = " virtual/network-configuration " diff --git a/external/meta-updater/classes/sota_raspberrypi.bbclass b/external/meta-updater/classes/sota_raspberrypi.bbclass index 69f09fd5..c901a70e 100644 --- a/external/meta-updater/classes/sota_raspberrypi.bbclass +++ b/external/meta-updater/classes/sota_raspberrypi.bbclass @@ -16,25 +16,55 @@ DEV_MATCH_DIRECTIVE_pn-networkd-dhcp-conf = "Driver=smsc95xx lan78xx" IMAGE_INSTALL_append_sota = " virtual/network-configuration " PREFERRED_PROVIDER_virtual/bootloader_sota ?= "u-boot" -UBOOT_ENTRYPOINT_sota ?= "0x00008000" +UBOOT_ENTRYPOINT_sota ?= "0x00080000" IMAGE_FSTYPES_remove_sota = "rpi-sdimg" OSTREE_BOOTLOADER ?= "u-boot" +def make_dtb_boot_files(d): + # Generate IMAGE_BOOT_FILES entries for device tree files listed in + # KERNEL_DEVICETREE. + # + # This function was taken from conf/machine/include/rpi-base.inc in + # meta-raspberrypi + alldtbs = d.getVar('KERNEL_DEVICETREE') + imgtyp = d.getVar('KERNEL_IMAGETYPE') + + def transform(dtb): + base = os.path.basename(dtb) + if dtb.endswith('dtb'): + return base + elif dtb.endswith('dtbo'): + return '{};{}'.format(base, dtb) + + return ' '.join([transform(dtb) for dtb in alldtbs.split(' ') if dtb]) + +IMAGE_BOOT_FILES_sota = "bcm2835-bootfiles/* \ + u-boot.bin;${SDIMG_KERNELIMAGE} \ + " + # OSTree puts its own boot.scr to bcm2835-bootfiles -IMAGE_BOOT_FILES_sota = "bcm2835-bootfiles/* u-boot.bin;${SDIMG_KERNELIMAGE}" +# raspberrypi4 needs dtb in /boot partition so that they can be read by the +# firmware +IMAGE_BOOT_FILES_append_sota_raspberrypi4 = "${@make_dtb_boot_files(d)}" # Just the overlays that will be used should be listed KERNEL_DEVICETREE_raspberrypi2_sota ?= " bcm2709-rpi-2-b.dtb " KERNEL_DEVICETREE_raspberrypi3_sota ?= " bcm2710-rpi-3-b.dtb overlays/vc4-kms-v3d.dtbo overlays/rpi-ft5406.dtbo" KERNEL_DEVICETREE_raspberrypi3-64_sota ?= " broadcom/bcm2710-rpi-3-b.dtb overlays/vc4-kms-v3d.dtbo overlays/vc4-fkms-v3d.dtbo overlays/rpi-ft5406.dtbo" +KERNEL_DEVICETREE_raspberrypi4_sota ?= " bcm2711-rpi-4-b.dtb overlays/vc4-fkms-v3d.dtbo overlays/uart0-rpi4.dtbo" +KERNEL_DEVICETREE_raspberrypi4-64_sota ?= " broadcom/bcm2711-rpi-4-b.dtb overlays/vc4-fkms-v3d.dtbo overlays/uart0-rpi4.dtbo" SOTA_MAIN_DTB_raspberrypi2 ?= "bcm2709-rpi-2-b.dtb" SOTA_MAIN_DTB_raspberrypi3 ?= "bcm2710-rpi-3-b.dtb" SOTA_MAIN_DTB_raspberrypi3-64 ?= "broadcom_bcm2710-rpi-3-b.dtb" +SOTA_MAIN_DTB_raspberrypi4_sota ?= "bcm2711-rpi-4-b.dtb" +SOTA_MAIN_DTB_raspberrypi4-64_sota ?= "broadcom_bcm2711-rpi-4-b.dtb" SOTA_DT_OVERLAYS_raspberrypi3 ?= "vc4-kms-v3d.dtbo rpi-ft5406.dtbo" SOTA_DT_OVERLAYS_raspberrypi3-64 ?= "vc4-kms-v3d.dtbo vc4-fkms-v3d.dtbo rpi-ft5406.dtbo" +SOTA_DT_OVERLAYS_raspberrypi4 ?= "vc4-fkms-v3d.dtbo uart0-rpi4.dtbo" +SOTA_DT_OVERLAYS_raspberrypi4-64 ?= "vc4-fkms-v3d.dtbo uart0-rpi4.dtbo" # Kernel args normally provided by RPi's internal bootloader. Non-updateable OSTREE_KERNEL_ARGS_sota ?= " 8250.nr_uarts=1 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=614 bcm2708_fb.fbswap=1 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 dwc_otg.lpm_enable=0 console=ttyS0,115200 usbhid.mousepoll=0 " diff --git a/external/meta-updater/classes/sota_sanity.bbclass b/external/meta-updater/classes/sota_sanity.bbclass index 8e80acbf..74973eb5 100644 --- a/external/meta-updater/classes/sota_sanity.bbclass +++ b/external/meta-updater/classes/sota_sanity.bbclass @@ -10,6 +10,52 @@ def sota_check_required_variables(status, d): if not d.getVar(var): status.addresult("%s should be set in your local.conf.\n" % var) +def sota_check_variables_validity(status, d): + import re + import os.path + + if d.getVar("OSTREE_BRANCHNAME") and re.match("^[a-zA-Z0-9._-]*$", d.getVar("OSTREE_BRANCHNAME")) is None: + status.addresult("OSTREE_BRANCHNAME Should only contain characters from the character set [a-zA-Z0-9._-].\n") + if d.getVar("SOTA_HARDWARE_ID") and re.match("^[a-zA-Z0-9._-]*$", d.getVar("SOTA_HARDWARE_ID")) is None: + status.addresult("SOTA_HARDWARE_ID Should only contain characters from the character set [a-zA-Z0-9._-].\n") + if d.getVar("SOTA_CLIENT_FEATURES") is not None: + for feat in d.getVar("SOTA_CLIENT_FEATURES").split(' '): + if feat not in ("hsm", "serialcan", "ubootenv", ""): + status.addresult("SOTA_CLIENT_FEATURES should only include hsm, serialcan and bootenv.\n") + break + if d.getVar("SOTA_CLIENT_PROV") is not None: + prov = d.getVar("SOTA_CLIENT_PROV").strip() + if prov not in ("aktualizr-shared-prov", "aktualizr-device-prov", "aktualizr-device-prov-hsm", ""): + status.addresult("Valid options for SOTA_CLIENT_PROV are aktualizr-shared-prov, aktualizr-device-prov and aktualizr-device-prov-hsm.\n") + if prov == "aktualizr-auto-prov": + bb.warn('aktualizr-auto-prov is deprecated. Please use aktualizr-shared-prov instead.') + elif prov == "aktualizr-ca-implicit-prov": + bb.warn('aktualizr-ca-implicit-prov is deprecated. Please use aktualizr-device-prov instead.') + elif prov == "aktualizr-hsm-prov": + bb.warn('aktualizr-hsm-prov is deprecated. Please use aktualizr-device-prov-hsm instead.') + if d.getVar("GARAGE_TARGET_URL") and re.match("^(https?|ftp|file)://.+$", d.getVar("GARAGE_TARGET_URL")) is None: + status.addresult("GARAGE_TARGET_URL is set to a bad url.\n") + if d.getVar("SOTA_POLLING_SEC") and re.match("^[1-9]\d*|0$", d.getVar("SOTA_POLLING_SEC")) is None: + status.addresult("SOTA_POLLING_SEC should be an integer.\n") + config = d.getVar("SOTA_SECONDARY_CONFIG") + if config is not None and config != "": + path = os.path.abspath(config) + if not os.path.exists(path): + status.addresult("SOTA_SECONDARY_CONFIG is not set correctly. The file containing JSON configuration for secondaries does not exist.\n") + credentials = d.getVar("SOTA_PACKED_CREDENTIALS") + if credentials is not None and credentials != "": + path = os.path.abspath(credentials) + if not os.path.exists(path): + status.addresult("SOTA_PACKED_CREDENTIALS is not set correctly. The zipped credentials file does not exist.\n") + if d.getVar("OSTREE_UPDATE_SUMMARY") and d.getVar("OSTREE_UPDATE_SUMMARY") not in ("0", "1", ""): + status.addresult("OSTREE_UPDATE_SUMMARY should be set to 0 or 1.\n") + if d.getVar("OSTREE_DEPLOY_DEVICETREE") and d.getVar("OSTREE_DEPLOY_DEVICETREE") not in ("0", "1", ""): + status.addresult("OSTREE_DEPLOY_DEVICETREE should be set to 0 or 1.\n") + if d.getVar("GARAGE_SIGN_AUTOVERSION") and d.getVar("GARAGE_SIGN_AUTOVERSION") not in ("0", "1", ""): + status.addresult("GARAGE_SIGN_AUTOVERSION should be set to 0 or 1.\n") + if d.getVar("SOTA_DEPLOY_CREDENTIALS") and d.getVar("SOTA_DEPLOY_CREDENTIALS") not in ("0", "1", ""): + status.addresult("SOTA_DEPLOY_CREDENTIALS should be set to 0 or 1.\n") + def sota_raise_sanity_error(msg, d): if d.getVar("SANITY_USE_EVENTS") == "1": bb.event.fire(bb.event.SanityCheckFailed(msg), d) @@ -34,6 +80,7 @@ def sota_check_sanity(sanity_data): sota_check_overrides(status, sanity_data) sota_check_required_variables(status, sanity_data) + sota_check_variables_validity(status, sanity_data) if status.messages != "": sota_raise_sanity_error(sanity_data.expand(status.messages), sanity_data) diff --git a/external/meta-updater/classes/target_version_example.bbclass b/external/meta-updater/classes/target_version_example.bbclass index ef119fb2..c0b5aec3 100644 --- a/external/meta-updater/classes/target_version_example.bbclass +++ b/external/meta-updater/classes/target_version_example.bbclass @@ -1,7 +1,5 @@ # Writes target version to be used by garage-sign -HOSTTOOLS += " git " - deploy_target_version () { version=$(git --git-dir=${METADIR}/.repo/manifests/.git/ rev-parse HEAD) echo -n ${version} > ${STAGING_DATADIR_NATIVE}/target_version diff --git a/external/meta-updater/conf/distro/poky-sota-systemd.conf b/external/meta-updater/conf/distro/poky-sota-systemd.conf index b30b322b..0dc50b0e 100644 --- a/external/meta-updater/conf/distro/poky-sota-systemd.conf +++ b/external/meta-updater/conf/distro/poky-sota-systemd.conf @@ -2,7 +2,7 @@ require conf/distro/poky.conf require conf/distro/sota.conf.inc -DISTRO = "poky-sota" +DISTRO = "poky-sota-systemd" DISTRO_NAME = "OTA-enabled Linux" DISTRO_VERSION = "1.0" DISTRO_CODENAME = "sota" diff --git a/external/meta-updater/conf/distro/poky-sota.conf b/external/meta-updater/conf/distro/poky-sota.conf index 3fb1d204..bfac90f8 100644 --- a/external/meta-updater/conf/distro/poky-sota.conf +++ b/external/meta-updater/conf/distro/poky-sota.conf @@ -5,3 +5,5 @@ DISTRO = "poky-sota" DISTRO_NAME = "OTA-enabled Linux" DISTRO_VERSION = "1.0" DISTRO_CODENAME = "sota" + +IMAGE_INSTALL_append_sota = " ostree-booted" diff --git a/external/meta-updater/conf/distro/sota.conf.inc b/external/meta-updater/conf/distro/sota.conf.inc index f6111bfc..1d5f8df1 100644 --- a/external/meta-updater/conf/distro/sota.conf.inc +++ b/external/meta-updater/conf/distro/sota.conf.inc @@ -16,4 +16,5 @@ INHERIT += "reproducible_build_simple" export SOURCE_DATE_EPOCH ?= "0" REPRODUCIBLE_TIMESTAMP_ROOTFS ?= "0" -HOSTTOOLS_append = " sync sha256sum" +HOSTTOOLS += "git sync sha256sum" +HOSTTOOLS_NONFATAL += "java repo python" diff --git a/external/meta-updater/conf/include/bblayers/sota_raspberrypi3-64.inc b/external/meta-updater/conf/include/bblayers/sota_raspberrypi3-64.inc index ea420bad..03f8f44b 100644 --- a/external/meta-updater/conf/include/bblayers/sota_raspberrypi3-64.inc +++ b/external/meta-updater/conf/include/bblayers/sota_raspberrypi3-64.inc @@ -1,2 +1,3 @@ +BBLAYERS += "${METADIR}/meta-openembedded/meta-python" BBLAYERS += "${METADIR}/meta-updater-raspberrypi" BBLAYERS += "${METADIR}/meta-raspberrypi" diff --git a/external/meta-updater/conf/include/bblayers/sota_raspberrypi4-64.inc b/external/meta-updater/conf/include/bblayers/sota_raspberrypi4-64.inc new file mode 100644 index 00000000..7e320af2 --- /dev/null +++ b/external/meta-updater/conf/include/bblayers/sota_raspberrypi4-64.inc @@ -0,0 +1,3 @@ +BBLAYERS += "${METADIR}/meta-updater-raspberrypi" +BBLAYERS += "${METADIR}/meta-raspberrypi" +BBLAYERS += "${METADIR}/meta-openembedded/meta-networking" diff --git a/external/meta-updater/conf/include/bblayers/sota_raspberrypi4.inc b/external/meta-updater/conf/include/bblayers/sota_raspberrypi4.inc new file mode 100644 index 00000000..7e320af2 --- /dev/null +++ b/external/meta-updater/conf/include/bblayers/sota_raspberrypi4.inc @@ -0,0 +1,3 @@ +BBLAYERS += "${METADIR}/meta-updater-raspberrypi" +BBLAYERS += "${METADIR}/meta-raspberrypi" +BBLAYERS += "${METADIR}/meta-openembedded/meta-networking" diff --git a/external/meta-updater/conf/layer.conf b/external/meta-updater/conf/layer.conf index 627a1b8a..39ea749f 100644 --- a/external/meta-updater/conf/layer.conf +++ b/external/meta-updater/conf/layer.conf @@ -9,5 +9,14 @@ BBFILE_COLLECTIONS += "sota" BBFILE_PATTERN_sota = "^${LAYERDIR}/" BBFILE_PRIORITY_sota = "7" -LAYERDEPENDS_sota = "filesystems-layer" +LAYERDEPENDS_sota = "openembedded-layer" +LAYERDEPENDS_sota += "filesystems-layer" LAYERSERIES_COMPAT_sota = "thud" + +SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS += " \ + aktualizr-device-prov->aktualizr \ + aktualizr-device-prov-hsm->aktualizr \ + aktualizr-shared-prov->aktualizr \ + aktualizr-shared-prov-creds->aktualizr \ + aktualizr-uboot-env-rollback->aktualizr \ +" diff --git a/external/meta-updater/conf/local.conf.base.append b/external/meta-updater/conf/local.conf.base.append new file mode 100644 index 00000000..36b2f59f --- /dev/null +++ b/external/meta-updater/conf/local.conf.base.append @@ -0,0 +1,18 @@ +# +# meta-updater configuration, see README.adoc and aktualizr's +# documentation for more options and detailed documentation +# + +MACHINE = "##MACHINE##" +DISTRO = "##DISTRO##" + +# General SOTA setup +#SOTA_CLIENT_PROV = "aktualizr-shared-prov" +#SOTA_PACKED_CREDENTIALS = "/path/to/credentials.zip" + +# Uncomment this line to start an ssh server at boot automatically +#IMAGE_FEATURES += "ssh-server-dropbear" + +# Uncomment this line to set the log level of aktualizr to 'debug' (from 'info' +# by default) +#IMAGE_INSTALL_append += " aktualizr-log-debug" diff --git a/external/meta-updater/conf/local.conf.nonostree.append b/external/meta-updater/conf/local.conf.nonostree.append new file mode 100644 index 00000000..0e63e988 --- /dev/null +++ b/external/meta-updater/conf/local.conf.nonostree.append @@ -0,0 +1,11 @@ + +DISTRO_FEATURES_append = " systemd" +VIRTUAL-RUNTIME_init_manager = "systemd" + +PREFERRED_RPROVIDER_virtual/network-configuration ??= "networkd-dhcp-conf" + +SOTA_DEPLOY_CREDENTIALS ?= "1" +PACKAGECONFIG_pn-aktualizr = "" + +IMAGE_INSTALL_append += "aktualizr" +IMAGE_INSTALL_append += "aktualizr-shared-prov" diff --git a/external/meta-updater/conf/local.conf.sample.append b/external/meta-updater/conf/local.conf.systemd.append index 4588ec3a..12e0182a 100644 --- a/external/meta-updater/conf/local.conf.sample.append +++ b/external/meta-updater/conf/local.conf.systemd.append @@ -1,23 +1,3 @@ - -# -# meta-updater configuration, see README.adoc and aktualizr's -# documentation for more options and detailed documentation -# - -MACHINE = "##MACHINE##" -DISTRO = "poky-sota-systemd" - -# General SOTA setup -#SOTA_CLIENT_PROV = "aktualizr-auto-prov" -#SOTA_PACKED_CREDENTIALS = "/path/to/credentials.zip" - -# Uncomment this line to start an ssh server at boot automatically -#IMAGE_FEATURES += "ssh-server-dropbear" - -# Uncomment this line to set the log level of aktualizr to 'debug' (from 'info' -# by default) -#IMAGE_INSTALL_append += " aktualizr-log-debug" - # Store systemd logs in persistent storage # # It greatly helps diagnosing issues on testing devices but should be diff --git a/external/meta-updater/lib/oeqa/selftest/cases/testutils.py b/external/meta-updater/lib/oeqa/selftest/cases/testutils.py index 8d618a68..3abfa5eb 100644 --- a/external/meta-updater/lib/oeqa/selftest/cases/testutils.py +++ b/external/meta-updater/lib/oeqa/selftest/cases/testutils.py @@ -33,6 +33,7 @@ def qemu_boot_image(imagename, **kwargs): # subdirectory. args.dir = 'tmp/deploy/images' args.efi = kwargs.get('efi', False) + args.bootloader = kwargs.get('bootloader', None) args.machine = kwargs.get('machine', None) args.mem = kwargs.get('mem', '128M') qemu_use_kvm = get_bb_var("QEMU_USE_KVM") @@ -48,6 +49,7 @@ def qemu_boot_image(imagename, **kwargs): args.overlay = kwargs.get('overlay', None) args.dry_run = kwargs.get('dry_run', False) args.secondary_network = kwargs.get('secondary_network', False) + args.uboot_enable = kwargs.get('uboot_enable', 'yes') qemu = QemuCommand(args) cmdline = qemu.command_line() @@ -119,20 +121,29 @@ def verifyNotProvisioned(testInst, machine): 'Device already provisioned!? ' + stderr.decode() + stdout.decode()) -def verifyProvisioned(testInst, machine): +def verifyProvisioned(testInst, machine, hwid=''): # Verify that device HAS provisioned. + # First loop while waiting for the device to boot. ran_ok = False for delay in [5, 5, 5, 5, 10, 10, 10, 10]: stdout, stderr, retcode = testInst.qemu_command('aktualizr-info') - if retcode == 0 and stderr == b'' and stdout.decode().find('Fetched metadata: yes') >= 0: + if retcode == 0 and stderr == b'': ran_ok = True break sleep(delay) testInst.assertTrue(ran_ok, 'aktualizr-info failed: ' + stderr.decode() + stdout.decode()) - + # Then wait for aktualizr to provision. + if stdout.decode().find('Fetched metadata: yes') < 0: + stdout, stderr, retcode = testInst.qemu_command('aktualizr-info --wait-until-provisioned') + testInst.assertFalse(retcode, 'aktualizr-info failed: ' + stderr.decode() + stdout.decode()) + testInst.assertEqual(stderr, b'', 'aktualizr-info failed: ' + stderr.decode() + stdout.decode()) testInst.assertIn(b'Device ID: ', stdout, 'Provisioning failed: ' + stderr.decode() + stdout.decode()) - testInst.assertIn(b'Primary ecu hardware ID: ' + machine.encode(), stdout, - 'Provisioning failed: ' + stderr.decode() + stdout.decode()) + if hwid == '': + testInst.assertIn(b'Primary ECU hardware ID: ' + machine.encode(), stdout, + 'Provisioning failed: ' + stderr.decode() + stdout.decode()) + else: + testInst.assertIn(b'Primary ECU hardware ID: ' + hwid.encode(), stdout, + 'Provisioning failed: ' + stderr.decode() + stdout.decode()) testInst.assertIn(b'Fetched metadata: yes', stdout, 'Provisioning failed: ' + stderr.decode() + stdout.decode()) p = re.compile(r'Device ID: ([a-z0-9-]*)\n') m = p.search(stdout.decode()) diff --git a/external/meta-updater/lib/oeqa/selftest/cases/updater_qemux86_64.py b/external/meta-updater/lib/oeqa/selftest/cases/updater_qemux86_64.py index 2b4726cb..08220f4e 100644 --- a/external/meta-updater/lib/oeqa/selftest/cases/updater_qemux86_64.py +++ b/external/meta-updater/lib/oeqa/selftest/cases/updater_qemux86_64.py @@ -85,6 +85,7 @@ class SharedCredProvTests(OESelftestTestCase): self.append_config('MACHINE = "qemux86-64"') self.append_config('SOTA_CLIENT_PROV = " aktualizr-shared-prov "') self.append_config('IMAGE_FSTYPES_remove = "ostreepush garagesign garagecheck"') + self.append_config('SOTA_HARDWARE_ID = "plain_reibekuchen_314"') self.qemu, self.s = qemu_launch(machine='qemux86-64') def tearDownLocal(self): @@ -107,7 +108,34 @@ class SharedCredProvTests(OESelftestTestCase): self.assertEqual(value, machine, 'MACHINE does not match hostname: ' + machine + ', ' + value) - verifyProvisioned(self, machine) + hwid = get_bb_var('SOTA_HARDWARE_ID') + verifyProvisioned(self, machine, hwid) + + +class SharedCredProvTestsNonOSTree(SharedCredProvTests): + + def setUpLocal(self): + layer = "meta-updater-qemux86-64" + result = runCmd('bitbake-layers show-layers') + if re.search(layer, result.output) is None: + self.meta_qemu = metadir() + layer + runCmd('bitbake-layers add-layer "%s"' % self.meta_qemu) + else: + self.meta_qemu = None + self.append_config('MACHINE = "qemux86-64"') + self.append_config('SOTA_CLIENT_PROV = ""') + self.append_config('IMAGE_FSTYPES_remove = "ostreepush garagesign garagecheck"') + self.append_config('SOTA_HARDWARE_ID = "plain_reibekuchen_314"') + + self.append_config('DISTRO = "poky"') + self.append_config('DISTRO_FEATURES_append = " systemd"') + self.append_config('VIRTUAL-RUNTIME_init_manager = "systemd"') + self.append_config('PREFERRED_RPROVIDER_virtual/network-configuration ??= "networkd-dhcp-conf"') + self.append_config('PACKAGECONFIG_pn-aktualizr = ""') + self.append_config('SOTA_DEPLOY_CREDENTIALS = "1"') + self.append_config('IMAGE_INSTALL_append += "aktualizr"') + self.append_config('IMAGE_INSTALL_append += " aktualizr-shared-prov"') + self.qemu, self.s = qemu_launch(machine='qemux86-64', uboot_enable='no') class ManualControlTests(OESelftestTestCase): @@ -358,17 +386,7 @@ class IpSecondaryTests(OESelftestTestCase): self._test_ctx.append_config('SOTA_CLIENT_PROV = " aktualizr-shared-prov "') def is_ecu_registered(self, ecu_id): - max_number_of_tries = 40 - try_counter = 0 - - # aktualizr-info is not always able to load ECU serials from DB - # so, let's run it a few times until it actually succeeds - while try_counter < max_number_of_tries: - device_status = self.get_info() - try_counter += 1 - if device_status.find("load ECU serials") == -1: - break - sleep(1) + device_status = self.get_info() if not ((device_status.find(ecu_id[0]) != -1) and (device_status.find(ecu_id[1]) != -1)): return False @@ -377,7 +395,7 @@ class IpSecondaryTests(OESelftestTestCase): return not_reg_start == -1 or (device_status.find(ecu_id[1], not_reg_start) == -1) def get_info(self): - stdout, stderr, retcode = self.send_command('aktualizr-info') + stdout, stderr, retcode = self.send_command('aktualizr-info --wait-until-provisioned', timeout=620) self._test_ctx.assertEqual(retcode, 0, 'Unable to run aktualizr-info: {}'.format(stderr)) return stdout @@ -473,4 +491,40 @@ class ResourceControlTests(OESelftestTestCase): stdout, stderr, retcode = self.qemu_command('systemctl --no-pager show --property=ExecMainStatus aktualizr') self.assertIn(b'ExecMainStatus=0', stdout, 'Aktualizr did not restart') + +class NonSystemdTests(OESelftestTestCase): + def setUpLocal(self): + layer = "meta-updater-qemux86-64" + result = runCmd('bitbake-layers show-layers') + if re.search(layer, result.output) is None: + self.meta_qemu = metadir() + layer + runCmd('bitbake-layers add-layer "%s"' % self.meta_qemu) + else: + self.meta_qemu = None + self.append_config('MACHINE = "qemux86-64"') + self.append_config('SOTA_CLIENT_PROV = " aktualizr-shared-prov "') + self.append_config('IMAGE_FSTYPES_remove = "ostreepush garagesign garagecheck"') + self.append_config('DISTRO = "poky-sota"') + self.append_config('IMAGE_INSTALL_remove += " aktualizr-resource-control"') + self.qemu, self.s = qemu_launch(machine='qemux86-64') + + def tearDownLocal(self): + qemu_terminate(self.s) + if self.meta_qemu: + runCmd('bitbake-layers remove-layer "%s"' % self.meta_qemu, ignore_status=True) + + def qemu_command(self, command): + return qemu_send_command(self.qemu.ssh_port, command) + + def test_provisioning(self): + print('Checking if systemd is not installed...') + stdout, stderr, retcode = self.qemu_command('systemctl') + self.assertTrue(retcode != 0, 'systemd is installed while it is not supposed to: ' + str(stdout)) + + stdout, stderr, retcode = self.qemu_command('aktualizr --run-mode once') + self.assertEqual(retcode, 0, 'Failed to run aktualizr: ' + str(stdout) + str(stderr)) + + machine = get_bb_var('MACHINE', 'core-image-minimal') + verifyProvisioned(self, machine) + # vim:set ts=4 sw=4 sts=4 expandtab: diff --git a/external/meta-updater/lib/oeqa/selftest/cases/updater_raspberrypi.py b/external/meta-updater/lib/oeqa/selftest/cases/updater_raspberrypi.py index 26d5c4c6..25c5f12e 100644 --- a/external/meta-updater/lib/oeqa/selftest/cases/updater_raspberrypi.py +++ b/external/meta-updater/lib/oeqa/selftest/cases/updater_raspberrypi.py @@ -35,26 +35,10 @@ class RpiTests(OESelftestTestCase): else: self.meta_upd_rpi = None - # This is trickier that I would've thought. The fundamental problem is - # that the qemu layer changes the u-boot file extension to .rom, but - # raspberrypi still expects .bin. To prevent this, the qemu layer must - # be temporarily removed if it is present. It has to be removed by name - # without the complete path, but to add it back when we are done, we - # need the full path. - p = re.compile(r'meta-updater-qemux86-64\s*(\S*meta-updater-qemux86-64)\s') - m = p.search(result.output) - if m and m.lastindex > 0: - self.meta_qemu = m.group(1) - runCmd('bitbake-layers remove-layer meta-updater-qemux86-64') - else: - self.meta_qemu = None - self.append_config('MACHINE = "raspberrypi3"') self.append_config('SOTA_CLIENT_PROV = " aktualizr-shared-prov "') def tearDownLocal(self): - if self.meta_qemu: - runCmd('bitbake-layers add-layer "%s"' % self.meta_qemu, ignore_status=True) if self.meta_upd_rpi: runCmd('bitbake-layers remove-layer "%s"' % self.meta_upd_rpi, ignore_status=True) if self.meta_rpi: diff --git a/external/meta-updater/recipes-connectivity/connman/connman_1.35.bbappend b/external/meta-updater/recipes-connectivity/connman/connman_1.35.bbappend new file mode 100644 index 00000000..08ec8326 --- /dev/null +++ b/external/meta-updater/recipes-connectivity/connman/connman_1.35.bbappend @@ -0,0 +1,6 @@ +RPROVIDES_${PN} += "virtual/network-configuration" + +# patch to not create the resolv.conf symlink at run-time, as it's already +# handled in the recipe and messes up with ostree +FILESEXTRAPATHS_prepend := "${THISDIR}/files:" +SRC_URI += "file://0001-tmpfiles-script-do-not-create-the-resolv.conf-symlin.patch" diff --git a/external/meta-updater/recipes-connectivity/connman/files/0001-tmpfiles-script-do-not-create-the-resolv.conf-symlin.patch b/external/meta-updater/recipes-connectivity/connman/files/0001-tmpfiles-script-do-not-create-the-resolv.conf-symlin.patch new file mode 100644 index 00000000..9b4a78c9 --- /dev/null +++ b/external/meta-updater/recipes-connectivity/connman/files/0001-tmpfiles-script-do-not-create-the-resolv.conf-symlin.patch @@ -0,0 +1,22 @@ +From 9e724a61f015304c9d72d829a66178d20e3fa980 Mon Sep 17 00:00:00 2001 +From: Laurent Bonnans <laurent.bonnans@here.com> +Date: Wed, 31 Jul 2019 18:15:47 +0200 +Subject: [PATCH] tmpfiles script: do not create the resolv.conf symlink + +It's handled by yocto in our case + +Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com> +--- + scripts/connman_resolvconf.conf.in | 1 - + 1 file changed, 1 deletion(-) + +diff --git a/scripts/connman_resolvconf.conf.in b/scripts/connman_resolvconf.conf.in +index 2d61dfe1..8a7d3071 100644 +--- a/scripts/connman_resolvconf.conf.in ++++ b/scripts/connman_resolvconf.conf.in +@@ -1,2 +1 @@ + d @runstatedir@/connman - - - - +-L+ /etc/resolv.conf - - - - @runstatedir@/connman/resolv.conf +-- +2.20.1 + diff --git a/external/meta-updater/recipes-connectivity/networkd-dhcp-conf/files/clean-connman-symlink.service b/external/meta-updater/recipes-connectivity/networkd-dhcp-conf/files/clean-connman-symlink.service new file mode 100644 index 00000000..8af82637 --- /dev/null +++ b/external/meta-updater/recipes-connectivity/networkd-dhcp-conf/files/clean-connman-symlink.service @@ -0,0 +1,11 @@ +[Unit] +Description=Clean up bogus symlinked resolv.conf +Before=network-pre.target +Wants=network-pre.target + +[Service] +Type=oneshot +ExecStart=/usr/sbin/resolvconf-clean + +[Install] +WantedBy=multi-user.target diff --git a/external/meta-updater/recipes-connectivity/networkd-dhcp-conf/files/resolvconf-clean b/external/meta-updater/recipes-connectivity/networkd-dhcp-conf/files/resolvconf-clean new file mode 100644 index 00000000..89c7e905 --- /dev/null +++ b/external/meta-updater/recipes-connectivity/networkd-dhcp-conf/files/resolvconf-clean @@ -0,0 +1,14 @@ +#!/bin/sh + +set -e + +if [ ! -L /etc/resolv.conf ]; then + exit 0 +fi + +# 'readlink -f' will fail if the symlink doesn't resolve to an existing path +if readlink /etc/resolv.conf | grep -q connman; then + echo "Replacing resolv.conf symlink: $(readlink /etc/resolv.conf) to /etc/resolv-conf.systemd" + rm /etc/resolv.conf + ln -s /etc/resolv-conf.systemd /etc/resolv.conf +fi diff --git a/external/meta-updater/recipes-connectivity/networkd-dhcp-conf/networkd-dhcp-conf.bb b/external/meta-updater/recipes-connectivity/networkd-dhcp-conf/networkd-dhcp-conf.bb index 0700ac6e..40a39582 100644 --- a/external/meta-updater/recipes-connectivity/networkd-dhcp-conf/networkd-dhcp-conf.bb +++ b/external/meta-updater/recipes-connectivity/networkd-dhcp-conf/networkd-dhcp-conf.bb @@ -4,25 +4,41 @@ interfaces through systemd-networkd" LICENSE = "MPL-2.0" LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MPL-2.0;md5=815ca599c9df247a0c7f619bab123dad" -inherit allarch systemd +inherit systemd RPROVIDES_${PN} = "virtual/network-configuration" -SRC_URI_append = " file://20-wired-dhcp.network" +SRC_URI = " \ + file://20-wired-dhcp.network \ + file://resolvconf-clean \ + file://clean-connman-symlink.service \ + " PR = "r1" -RDEPENDS_${PN} = "systemd" +REQUIRED_DISTRO_FEATURES_${PN} = "systemd" +RCONFLICTS_${PN} = "connman" S = "${WORKDIR}" PACKAGE_ARCH = "${MACHINE_ARCH}" -FILES_${PN} = "${systemd_unitdir}/network/*" +FILES_${PN} = " \ + ${systemd_unitdir}/network/* \ + ${sbindir}/resolvconf-clean \ + ${systemd_unitdir}/system/clean-connman-symlink.service \ + " + +SYSTEMD_SERVICE_${PN} = "clean-connman-symlink.service" DEV_MATCH_DIRECTIVE ?= "Name=en*" do_install() { install -d ${D}/${systemd_unitdir}/network - install -m 0644 ${WORKDIR}/20-wired-dhcp.network ${D}/${systemd_unitdir}/network + install -m 0644 ${WORKDIR}/20-wired-dhcp.network ${D}${systemd_unitdir}/network sed -i -e 's|@MATCH_DIRECTIVE@|${DEV_MATCH_DIRECTIVE}|g' ${D}${systemd_unitdir}/network/20-wired-dhcp.network + + install -d ${D}${sbindir} + install -m 0755 ${WORKDIR}/resolvconf-clean ${D}${sbindir}/resolvconf-clean + install -d ${D}${systemd_unitdir}/system + install -m 0644 ${WORKDIR}/clean-connman-symlink.service ${D}${systemd_unitdir}/system/clean-connman-symlink.service } diff --git a/external/meta-updater/recipes-connectivity/zabbix/zabbix_%.bbappend b/external/meta-updater/recipes-connectivity/zabbix/zabbix_%.bbappend new file mode 100644 index 00000000..c6a69b5b --- /dev/null +++ b/external/meta-updater/recipes-connectivity/zabbix/zabbix_%.bbappend @@ -0,0 +1,20 @@ + +do_install_append() { + + # Set the zabbix Server + if [ ! -z ${SOTA_COMM_CONF_ZABBIX_SERVER} ]; then + sed -i "s/Server=\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}/Server=${SOTA_COMM_CONF_ZABBIX_SERVER}/g" ${D}${sysconfdir}/zabbix_agentd.conf + if ! grep -Fxq "Server=${SOTA_COMM_CONF_ZABBIX_SERVER}" ${D}${sysconfdir}/zabbix_agentd.conf; then + echo -e '\nServer='${SOTA_COMM_CONF_ZABBIX_SERVER} >> ${D}${sysconfdir}/zabbix_agentd.conf + fi + fi + + # Set ServerActive + if [ ! -z ${SOTA_COMM_CONF_ZABBIX_SERVERACTIVE} ]; then + sed -i "s/ServerActive=\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}/ServerActive=${SOTA_COMM_CONF_ZABBIX_SERVERACTIVE}/g" ${D}${sysconfdir}/zabbix_agentd.conf + if ! grep -Fxq "ServerActive=${SOTA_COMM_CONF_ZABBIX_SERVERACTIVE}" ${D}${sysconfdir}/zabbix_agentd.conf; then + echo -e '\nServerActive='${SOTA_COMM_CONF_ZABBIX_SERVERACTIVE} >> ${D}${sysconfdir}/zabbix_agentd.conf + fi + + fi +} diff --git a/external/meta-updater/recipes-core/images/initramfs-ostree-image.bb b/external/meta-updater/recipes-core/images/initramfs-ostree-image.bb index e77499e2..936c59a0 100644 --- a/external/meta-updater/recipes-core/images/initramfs-ostree-image.bb +++ b/external/meta-updater/recipes-core/images/initramfs-ostree-image.bb @@ -13,8 +13,13 @@ IMAGE_LINGUAS = "" LICENSE = "MIT" +IMAGE_CLASSES_remove = "image_repo_manifest" + IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}" +# Avoid circular dependencies +EXTRA_IMAGEDEPENDS = "" + inherit core-image IMAGE_ROOTFS_SIZE = "8192" @@ -25,5 +30,3 @@ IMAGE_ROOTFS_EXTRA_SPACE = "0" IMAGE_OVERHEAD_FACTOR = "1.0" BAD_RECOMMENDATIONS += "busybox-syslog" - - diff --git a/external/meta-updater/recipes-sota/aktualizr/aktualizr-device-prov-creds.bb b/external/meta-updater/recipes-sota/aktualizr/aktualizr-device-prov-creds.bb deleted file mode 100644 index 6e02a501..00000000 --- a/external/meta-updater/recipes-sota/aktualizr/aktualizr-device-prov-creds.bb +++ /dev/null @@ -1,60 +0,0 @@ -SUMMARY = "Credentials for device provisioning with fleet CA certificate" -HOMEPAGE = "https://github.com/advancedtelematic/aktualizr" -SECTION = "base" -LICENSE = "MPL-2.0" -LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MPL-2.0;md5=815ca599c9df247a0c7f619bab123dad" - -inherit allarch - -# WARNING: it is NOT a production solution. The secure way to provision devices -# is to create certificate request directly on the device (either with HSM/TPM -# or with software) and then sign it with a CA stored on a disconnected machine. - -DEPENDS = "aktualizr aktualizr-native" -ALLOW_EMPTY_${PN} = "1" - -SRC_URI = " \ - file://ca.cnf \ - " - -require credentials.inc - -export SOTA_CACERT_PATH -export SOTA_CAKEY_PATH - -do_install() { - if [ -n "${SOTA_PACKED_CREDENTIALS}" ]; then - if [ -z ${SOTA_CACERT_PATH} ]; then - SOTA_CACERT_PATH=${DEPLOY_DIR_IMAGE}/CA/cacert.pem - SOTA_CAKEY_PATH=${DEPLOY_DIR_IMAGE}/CA/ca.private.pem - mkdir -p ${DEPLOY_DIR_IMAGE}/CA - bbwarn "SOTA_CACERT_PATH is not specified, use default one at ${SOTA_CACERT_PATH}" - - if [ ! -f ${SOTA_CACERT_PATH} ]; then - bbwarn "${SOTA_CACERT_PATH} does not exist, generate a new CA" - SOTA_CACERT_DIR_PATH="$(dirname "${SOTA_CACERT_PATH}")" - openssl genrsa -out ${SOTA_CACERT_DIR_PATH}/ca.private.pem 4096 - openssl req -key ${SOTA_CACERT_DIR_PATH}/ca.private.pem -new -x509 -days 7300 -out ${SOTA_CACERT_PATH} -subj "/C=DE/ST=Berlin/O=Reis und Kichererbsen e.V/commonName=meta-updater" -batch -config ${WORKDIR}/ca.cnf -extensions cacert - bbwarn "${SOTA_CACERT_PATH} has been created, you'll need to upload it to the server" - fi - fi - - if [ -z ${SOTA_CAKEY_PATH} ]; then - bbfatal "SOTA_CAKEY_PATH should be set when using device credential provisioning" - fi - - install -m 0700 -d ${D}${localstatedir}/sota - aktualizr-cert-provider --credentials ${SOTA_PACKED_CREDENTIALS} \ - --fleet-ca ${SOTA_CACERT_PATH} \ - --fleet-ca-key ${SOTA_CAKEY_PATH} \ - --root-ca \ - --server-url \ - --local ${D} \ - --config ${STAGING_DIR_HOST}${libdir}/sota/sota-device-cred.toml - fi -} - -FILES_${PN} = " \ - ${localstatedir}/sota/*" - -# vim:set ts=4 sw=4 sts=4 expandtab: diff --git a/external/meta-updater/recipes-sota/aktualizr/aktualizr-device-prov-hsm.bb b/external/meta-updater/recipes-sota/aktualizr/aktualizr-device-prov-hsm.bb index c3cd593b..8f28c03b 100644 --- a/external/meta-updater/recipes-sota/aktualizr/aktualizr-device-prov-hsm.bb +++ b/external/meta-updater/recipes-sota/aktualizr/aktualizr-device-prov-hsm.bb @@ -7,14 +7,16 @@ LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MPL-2.0;md5=815ca599c9df247a0c7 inherit allarch -DEPENDS = "aktualizr aktualizr-native" -RDEPENDS_${PN}_append = "${@' aktualizr-device-prov-creds softhsm-testtoken' if d.getVar('SOTA_DEPLOY_CREDENTIALS') == '1' else ''}" +# We need to get the config files from the aktualizr-host-tools package built by +# the aktualizr (target) recipe. +DEPENDS = "aktualizr" -SRC_URI = "" +# If the config file from aktualizr used here is changed, you will need to bump +# the version here because of SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS! PV = "1.0" -PR = "6" +PR = "7" -require credentials.inc +SRC_URI = "" do_install() { install -m 0700 -d ${D}${libdir}/sota/conf.d diff --git a/external/meta-updater/recipes-sota/aktualizr/aktualizr-device-prov.bb b/external/meta-updater/recipes-sota/aktualizr/aktualizr-device-prov.bb index d5795324..55f398d6 100644 --- a/external/meta-updater/recipes-sota/aktualizr/aktualizr-device-prov.bb +++ b/external/meta-updater/recipes-sota/aktualizr/aktualizr-device-prov.bb @@ -7,13 +7,16 @@ LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MPL-2.0;md5=815ca599c9df247a0c7 inherit allarch -DEPENDS = "aktualizr aktualizr-native openssl-native" -RDEPENDS_${PN}_append = "${@' aktualizr-device-prov-creds' if d.getVar('SOTA_DEPLOY_CREDENTIALS') == '1' else ''}" +# We need to get the config files from the aktualizr-host-tools package built by +# the aktualizr (target) recipe. +DEPENDS = "aktualizr" +# If the config file from aktualizr used here is changed, you will need to bump +# the version here because of SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS! PV = "1.0" PR = "1" -require credentials.inc +SRC_URI = "" do_install() { install -m 0700 -d ${D}${libdir}/sota/conf.d diff --git a/external/meta-updater/recipes-sota/aktualizr/aktualizr-hwid.bb b/external/meta-updater/recipes-sota/aktualizr/aktualizr-hwid.bb new file mode 100644 index 00000000..fd3e3953 --- /dev/null +++ b/external/meta-updater/recipes-sota/aktualizr/aktualizr-hwid.bb @@ -0,0 +1,24 @@ +SUMMARY = "Aktualizr hwid configuration" +HOMEPAGE = "https://github.com/advancedtelematic/aktualizr" +SECTION = "base" +LICENSE = "MPL-2.0" +LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MPL-2.0;md5=815ca599c9df247a0c7f619bab123dad" + +# Because of the dependency on MACHINE. +PACKAGE_ARCH = "${MACHINE_ARCH}" + +SRC_URI = "" + +do_install() { + install -m 0700 -d ${D}${libdir}/sota/conf.d + if [ -n "${SOTA_HARDWARE_ID}" ]; then + printf "[provision]\nprimary_ecu_hardware_id = ${SOTA_HARDWARE_ID}\n" > ${D}${libdir}/sota/conf.d/40-hardware-id.toml + fi +} + +FILES_${PN} = " \ + ${libdir}/sota/conf.d \ + ${libdir}/sota/conf.d/40-hardware-id.toml \ + " + +# vim:set ts=4 sw=4 sts=4 expandtab: diff --git a/external/meta-updater/recipes-sota/aktualizr/aktualizr-shared-prov-creds.bb b/external/meta-updater/recipes-sota/aktualizr/aktualizr-shared-prov-creds.bb index dbb5fde5..9c6f0dd4 100644 --- a/external/meta-updater/recipes-sota/aktualizr/aktualizr-shared-prov-creds.bb +++ b/external/meta-updater/recipes-sota/aktualizr/aktualizr-shared-prov-creds.bb @@ -6,22 +6,32 @@ LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MPL-2.0;md5=815ca599c9df247a0c7 inherit allarch -DEPENDS = "aktualizr-native zip-native" +DEPENDS = "zip-native" ALLOW_EMPTY_${PN} = "1" +# If the config file from aktualizr used here is changed, you will need to bump +# the version here because of SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS! +PV = "1.0" +PR = "1" + +SRC_URI = "" + require credentials.inc do_install() { if [ -n "${SOTA_PACKED_CREDENTIALS}" ]; then install -m 0700 -d ${D}${localstatedir}/sota - cp "${SOTA_PACKED_CREDENTIALS}" ${D}${localstatedir}/sota/sota_provisioning_credentials.zip - # Device should not be able to push data to treehub - zip -d ${D}${localstatedir}/sota/sota_provisioning_credentials.zip treehub.json - # Device has no use for the API Gateway. Remove if present. See: - # https://github.com/advancedtelematic/ota-plus-server/pull/1913/ - if unzip -l ${D}${localstatedir}/sota/sota_provisioning_credentials.zip api_gateway.url > /dev/null; then - zip -d ${D}${localstatedir}/sota/sota_provisioning_credentials.zip api_gateway.url - fi + # root.json contains the root metadata for bootstrapping the Uptane metadata verification process. + # autoprov.url has the URL to the device gateway on the server, which is where we send most of our requests. + # autoprov_credentials.p12 contains the shared provisioning credentials. + for var in root.json autoprov.url autoprov_credentials.p12; do + if unzip -l "${SOTA_PACKED_CREDENTIALS}" $var > /dev/null; then + unzip "${SOTA_PACKED_CREDENTIALS}" $var -d ${T} + zip -mj -q ${D}${localstatedir}/sota/sota_provisioning_credentials.zip ${T}/$var + else + bbwarn "$var is missing from credentials.zip" + fi + done fi } diff --git a/external/meta-updater/recipes-sota/aktualizr/aktualizr-shared-prov.bb b/external/meta-updater/recipes-sota/aktualizr/aktualizr-shared-prov.bb index d3d6f165..2ee47a16 100644 --- a/external/meta-updater/recipes-sota/aktualizr/aktualizr-shared-prov.bb +++ b/external/meta-updater/recipes-sota/aktualizr/aktualizr-shared-prov.bb @@ -7,15 +7,18 @@ LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MPL-2.0;md5=815ca599c9df247a0c7 inherit allarch -DEPENDS = "aktualizr-native zip-native" +# We need to get the config files from the aktualizr-host-tools package built by +# the aktualizr (target) recipe. +DEPENDS = "aktualizr" RDEPENDS_${PN}_append = "${@' aktualizr-shared-prov-creds' if d.getVar('SOTA_DEPLOY_CREDENTIALS') == '1' else ''}" + +# If the config file from aktualizr used here is changed, you will need to bump +# the version here because of SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS! PV = "1.0" PR = "6" SRC_URI = "" -require credentials.inc - do_install() { if [ -n "${SOTA_AUTOPROVISION_CREDENTIALS}" ]; then bbwarn "SOTA_AUTOPROVISION_CREDENTIALS are ignored. Please use SOTA_PACKED_CREDENTIALS" @@ -31,7 +34,7 @@ do_install() { fi install -m 0700 -d ${D}${libdir}/sota/conf.d - install -m 0644 ${STAGING_DIR_NATIVE}${libdir}/sota/sota-shared-cred.toml \ + install -m 0644 ${STAGING_DIR_HOST}${libdir}/sota/sota-shared-cred.toml \ ${D}${libdir}/sota/conf.d/20-sota-shared-cred.toml } diff --git a/external/meta-updater/recipes-sota/aktualizr/aktualizr-uboot-env-rollback.bb b/external/meta-updater/recipes-sota/aktualizr/aktualizr-uboot-env-rollback.bb index 860f225c..2895e5c4 100644 --- a/external/meta-updater/recipes-sota/aktualizr/aktualizr-uboot-env-rollback.bb +++ b/external/meta-updater/recipes-sota/aktualizr/aktualizr-uboot-env-rollback.bb @@ -6,14 +6,18 @@ LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MPL-2.0;md5=815ca599c9df247a0c7 inherit allarch -DEPENDS = "aktualizr-native" -RDEPENDS_${PN} = "aktualizr" +DEPENDS = "aktualizr" + +# If the config file from aktualizr used here is changed, you will need to bump +# the version here because of SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS! +PV = "1.0" +PR = "1" SRC_URI = "" do_install() { install -m 0700 -d ${D}${libdir}/sota/conf.d - install -m 0644 ${STAGING_DIR_NATIVE}${libdir}/sota/sota-uboot-env.toml ${D}${libdir}/sota/conf.d/30-rollback.toml + install -m 0644 ${STAGING_DIR_HOST}${libdir}/sota/sota-uboot-env.toml ${D}${libdir}/sota/conf.d/30-rollback.toml } FILES_${PN} = " \ diff --git a/external/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb b/external/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb index 5de341e4..20dd4237 100644 --- a/external/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb +++ b/external/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb @@ -3,35 +3,36 @@ DESCRIPTION = "SOTA Client application written in C++" HOMEPAGE = "https://github.com/advancedtelematic/aktualizr" SECTION = "base" LICENSE = "MPL-2.0" -LIC_FILES_CHKSUM = "file://${S}/LICENSE;md5=9741c346eef56131163e13b9db1241b3" +LIC_FILES_CHKSUM = "file://${S}/LICENSE;md5=815ca599c9df247a0c7f619bab123dad" DEPENDS = "boost curl openssl libarchive libsodium sqlite3 asn1c-native" DEPENDS_append = "${@bb.utils.contains('PTEST_ENABLED', '1', ' coreutils-native net-tools-native ostree-native aktualizr-native ', '', d)}" -RDEPENDS_${PN}_class-target = "aktualizr-configs lshw" -RDEPENDS_${PN}-host-tools = "aktualizr aktualizr-repo aktualizr-cert-provider ${@bb.utils.contains('PACKAGECONFIG', 'sota-tools', 'garage-deploy garage-push', '', d)}" +RDEPENDS_${PN}_class-target = "${PN}-configs ${PN}-hwid lshw" +RDEPENDS_${PN}-host-tools = "aktualizr aktualizr-cert-provider ${@bb.utils.contains('PACKAGECONFIG', 'sota-tools', 'garage-deploy garage-push', '', d)}" RDEPENDS_${PN}-ptest += "bash cmake curl python3-misc python3-modules openssl-bin sqlite3 valgrind" +PRIVATE_LIBS_${PN}-ptest = "libaktualizr.so libaktualizr_secondary.so" + PV = "1.0+git${SRCPV}" PR = "7" -GARAGE_SIGN_PV = "0.7.0-3-gf5ba640" +GARAGE_SIGN_PV = "0.7.0-87-g905dc3c" SRC_URI = " \ - gitsm://github.com/advancedtelematic/aktualizr;branch=${BRANCH} \ + gitsm://github.com/advancedtelematic/aktualizr;branch=${BRANCH};name=aktualizr \ file://run-ptest \ file://aktualizr.service \ file://aktualizr-secondary.service \ file://aktualizr-serialcan.service \ file://10-resource-control.conf \ - ${@ d.expand("https://ats-tuf-cli-releases.s3-eu-central-1.amazonaws.com/cli-${GARAGE_SIGN_PV}.tgz;unpack=0") if d.getVar('GARAGE_SIGN_AUTOVERSION') != '1' else ''} \ + ${@ d.expand("https://ats-tuf-cli-releases.s3-eu-central-1.amazonaws.com/cli-${GARAGE_SIGN_PV}.tgz;unpack=0;name=garagesign") if d.getVar('GARAGE_SIGN_AUTOVERSION') != '1' else ''} \ " -# for garage-sign archive -SRC_URI[md5sum] = "e104ccd4f32e52571a5fc0e5042db050" -SRC_URI[sha256sum] = "c590be1a57523bfe097af82279eda5c97cf40ae47fb27162cf33c469702c8a9b" +SRC_URI[garagesign.md5sum] = "064b408c60676dcf282aa9209bff7dac" +SRC_URI[garagesign.sha256sum] = "75c9b3cf24eb31dacb127d3b3d073359082e2b4ee4eeb27d75e792664800ba82" -SRCREV = "9c592cf9d8dfcd995d47753f2be7bd1a2b56c7da" +SRCREV = "f90e8996e826d130976a7b7f1835947b7e631025" BRANCH ?= "master" S = "${WORKDIR}/git" @@ -46,17 +47,17 @@ SYSTEMD_PACKAGES = "${PN} ${PN}-secondary" SYSTEMD_SERVICE_${PN} = "aktualizr.service" SYSTEMD_SERVICE_${PN}-secondary = "aktualizr-secondary.service" -EXTRA_OECMAKE = "-DCMAKE_BUILD_TYPE=Release -DAKTUALIZR_VERSION=${PV} ${@bb.utils.contains('PTEST_ENABLED', '1', '-DTESTSUITE_VALGRIND=on', '', d)}" +EXTRA_OECMAKE = "-DCMAKE_BUILD_TYPE=Release ${@bb.utils.contains('PTEST_ENABLED', '1', '-DTESTSUITE_VALGRIND=on', '', d)}" GARAGE_SIGN_OPS = "${@ d.expand('-DGARAGE_SIGN_ARCHIVE=${WORKDIR}/cli-${GARAGE_SIGN_PV}.tgz') if d.getVar('GARAGE_SIGN_AUTOVERSION') != '1' else ''}" +PKCS11_ENGINE_PATH = "${libdir}/engines-1.1/pkcs11.so" -PACKAGECONFIG ?= "ostree ${@bb.utils.filter('DISTRO_FEATURES', 'systemd', d)} ${@bb.utils.filter('SOTA_CLIENT_FEATURES', 'hsm serialcan ubootenv', d)}" +PACKAGECONFIG ?= "ostree ${@bb.utils.filter('SOTA_CLIENT_FEATURES', 'hsm serialcan ubootenv', d)}" PACKAGECONFIG_class-native = "sota-tools" PACKAGECONFIG[warning-as-error] = "-DWARNING_AS_ERROR=ON,-DWARNING_AS_ERROR=OFF," PACKAGECONFIG[ostree] = "-DBUILD_OSTREE=ON,-DBUILD_OSTREE=OFF,ostree," -PACKAGECONFIG[hsm] = "-DBUILD_P11=ON,-DBUILD_P11=OFF,libp11," +PACKAGECONFIG[hsm] = "-DBUILD_P11=ON -DPKCS11_ENGINE_PATH=${PKCS11_ENGINE_PATH},-DBUILD_P11=OFF,libp11," PACKAGECONFIG[sota-tools] = "-DBUILD_SOTA_TOOLS=ON ${GARAGE_SIGN_OPS},-DBUILD_SOTA_TOOLS=OFF,glib-2.0," -PACKAGECONFIG[systemd] = "-DBUILD_SYSTEMD=ON,-DBUILD_SYSTEMD=OFF,systemd," PACKAGECONFIG[load-tests] = "-DBUILD_LOAD_TESTS=ON,-DBUILD_LOAD_TESTS=OFF," PACKAGECONFIG[serialcan] = ",,,slcand-start" PACKAGECONFIG[ubootenv] = ",,,u-boot-fw-utils aktualizr-uboot-env-rollback" @@ -70,6 +71,14 @@ RESOURCE_CPU_WEIGHT = "100" RESOURCE_MEMORY_HIGH = "100M" RESOURCE_MEMORY_MAX = "80%" +do_configure_prepend() { + # CMake has trouble finding yocto's git when cross-compiling, let's do this step manually + cd ${S} + if [ ! -f VERSION ]; then + ./scripts/get_version.sh > VERSION + fi +} + do_compile_ptest() { cmake_runcmake_build --target build_tests "${PARALLEL_MAKE}" } @@ -79,9 +88,6 @@ do_install_ptest() { cp -r ${B}/ ${D}/${PTEST_PATH}/build cp -r ${S}/ ${D}/${PTEST_PATH}/src - # remove huge external unused repository - rm -rf ${D}/${PTEST_PATH}/src/partial/extern/RIOT - # remove huge build artifacts find ${D}/${PTEST_PATH}/build/src -name "*.a" -delete @@ -102,10 +108,6 @@ do_install_append () { install -m 0700 -d ${D}${libdir}/sota/conf.d install -m 0700 -d ${D}${sysconfdir}/sota/conf.d - if [ -n "${SOTA_HARDWARE_ID}" ]; then - printf "[provision]\nprimary_ecu_hardware_id = ${SOTA_HARDWARE_ID}\n" > ${D}${libdir}/sota/conf.d/40-hardware-id.toml - fi - install -m 0755 -d ${D}${systemd_unitdir}/system aktualizr_service=${@bb.utils.contains('SOTA_CLIENT_FEATURES', 'serialcan', '${WORKDIR}/aktualizr-serialcan.service', '${WORKDIR}/aktualizr.service', d)} install -m 0644 ${aktualizr_service} ${D}${systemd_unitdir}/system/aktualizr.service @@ -137,7 +139,7 @@ python split_hosttools_packages () { PACKAGES_DYNAMIC = "^aktualizr-.* ^garage-.*" -PACKAGES =+ "${PN}-resource-control ${PN}-examples ${PN}-secondary ${PN}-configs ${PN}-host-tools" +PACKAGES =+ "${PN}-host-tools ${PN}-lib ${PN}-resource-control ${PN}-configs ${PN}-secondary ${PN}-secondary-lib ${PN}-sotatools-lib" ALLOW_EMPTY_${PN}-host-tools = "1" @@ -147,6 +149,10 @@ FILES_${PN} = " \ ${systemd_unitdir}/system/aktualizr.service \ " +FILES_${PN}-lib = " \ + ${libdir}/libaktualizr.so \ + " + FILES_${PN}-resource-control = " \ ${systemd_system_unitdir}/aktualizr.service.d/10-resource-control.conf \ " @@ -156,16 +162,22 @@ FILES_${PN}-configs = " \ ${libdir}/sota/* \ " -FILES_${PN}-examples = " \ - ${bindir}/hmi-stub \ - " - FILES_${PN}-secondary = " \ ${bindir}/aktualizr-secondary \ ${libdir}/sota/sota-secondary.toml \ ${systemd_unitdir}/system/aktualizr-secondary.service \ " +FILES_${PN}-secondary-lib = " \ + ${libdir}/libaktualizr_secondary.so \ + " + +FILES_${PN}-sotatools-lib = " \ + ${libdir}/libsota_tools.so \ + " + +FILES_${PN}-dev = "" + BBCLASSEXTEND = "native" # vim:set ts=4 sw=4 sts=4 expandtab: diff --git a/external/meta-updater/recipes-sota/aktualizr/files/aktualizr-secondary.service b/external/meta-updater/recipes-sota/aktualizr/files/aktualizr-secondary.service index b577ae8b..fb610f9b 100644 --- a/external/meta-updater/recipes-sota/aktualizr/files/aktualizr-secondary.service +++ b/external/meta-updater/recipes-sota/aktualizr/files/aktualizr-secondary.service @@ -1,6 +1,7 @@ [Unit] Description=Aktualizr SOTA Client (UPTANE Secondary) -After=network.target +After=network-online.target +Wants=network-online.target [Service] RestartSec=10 diff --git a/external/meta-updater/recipes-sota/aktualizr/files/aktualizr.service b/external/meta-updater/recipes-sota/aktualizr/files/aktualizr.service index 726809e8..3d807a1f 100644 --- a/external/meta-updater/recipes-sota/aktualizr/files/aktualizr.service +++ b/external/meta-updater/recipes-sota/aktualizr/files/aktualizr.service @@ -1,6 +1,7 @@ [Unit] Description=Aktualizr SOTA Client -After=network.target +After=network-online.target nss-lookup.target +Wants=network-online.target [Service] RestartSec=10 diff --git a/external/meta-updater/recipes-sota/config/aktualizr-virtualsec.bb b/external/meta-updater/recipes-sota/config/aktualizr-virtualsec.bb new file mode 100644 index 00000000..b7d55aaa --- /dev/null +++ b/external/meta-updater/recipes-sota/config/aktualizr-virtualsec.bb @@ -0,0 +1,27 @@ +SUMMARY = "Example virtual secondary in aktualizr" +DESCRIPTION = "Creates an example virtual secondary to be used to update an arbitrary file on the primary" +HOMEPAGE = "https://github.com/advancedtelematic/aktualizr" +SECTION = "base" +LICENSE = "MPL-2.0" +LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MPL-2.0;md5=815ca599c9df247a0c7f619bab123dad" + +inherit allarch + +SRC_URI = " \ + file://30-virtualsec.toml \ + file://virtualsec.json \ + " + +do_install_append () { + install -m 0700 -d ${D}${libdir}/sota/conf.d + install -m 0644 ${WORKDIR}/30-virtualsec.toml ${D}${libdir}/sota/conf.d/30-virtualsec.toml + install -m 0644 ${WORKDIR}/virtualsec.json ${D}${libdir}/sota/virtualsec.json +} + +FILES_${PN} = " \ + ${libdir}/sota/conf.d/30-virtualsec.toml \ + ${libdir}/sota/virtualsec.json \ + " + +# vim:set ts=4 sw=4 sts=4 expandtab: + diff --git a/external/meta-updater/recipes-sota/config/files/30-virtualsec.toml b/external/meta-updater/recipes-sota/config/files/30-virtualsec.toml new file mode 100644 index 00000000..987f692d --- /dev/null +++ b/external/meta-updater/recipes-sota/config/files/30-virtualsec.toml @@ -0,0 +1,3 @@ +[uptane] +secondary_config_file = "/usr/lib/sota/virtualsec.json" + diff --git a/external/meta-updater/recipes-sota/config/files/virtualsec.json b/external/meta-updater/recipes-sota/config/files/virtualsec.json new file mode 100644 index 00000000..dcdcdba7 --- /dev/null +++ b/external/meta-updater/recipes-sota/config/files/virtualsec.json @@ -0,0 +1,14 @@ +{ + "virtual": [ + { + "partial_verifying": "false", + "ecu_hardware_id": "external-config", + "full_client_dir": "/var/sota/external-config", + "ecu_private_key": "sec.private", + "ecu_public_key": "sec.public", + "firmware_path": "/var/sota/external-config/config.txt", + "target_name_path": "/var/sota/external-config/target_name", + "metadata_path": "/var/sota/external-config/metadata" + } + ] +} diff --git a/external/meta-updater/recipes-sota/ostree/files/touch-ostree b/external/meta-updater/recipes-sota/ostree/files/touch-ostree new file mode 100755 index 00000000..28cb6723 --- /dev/null +++ b/external/meta-updater/recipes-sota/ostree/files/touch-ostree @@ -0,0 +1,21 @@ +#!/bin/sh +### BEGIN INIT INFO +# Provides: touch-ostree +# Required-Start: $network $remote_fs +# Required-Stop: $network $remote_fs +# Default-Start: 2 3 4 5 +# Default-Stop: 0 1 6 +# Short-Description: Indicate OSTree boot +### END INIT INFO + +case "$1" in + start) + touch /run/ostree-booted + ;; + stop) + ;; + *) + echo "Usage: /etc/init.d/touch-ostree {start|stop}" + exit 1 + ;; +esac diff --git a/external/meta-updater/recipes-sota/ostree/ostree-booted_1.0.bb b/external/meta-updater/recipes-sota/ostree/ostree-booted_1.0.bb new file mode 100644 index 00000000..d74cf247 --- /dev/null +++ b/external/meta-updater/recipes-sota/ostree/ostree-booted_1.0.bb @@ -0,0 +1,15 @@ +SUMMARY = "Indicate an OSTree boot" +DESCRIPTION = "Indicate an OSTree boot" +LICENSE = "MPL-2.0" +LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MPL-2.0;md5=815ca599c9df247a0c7f619bab123dad" +SRC_URI = "file://touch-ostree" + +inherit allarch update-rc.d + +INITSCRIPT_NAME = "touch-ostree" +INITSCRIPT_PARAMS = "start 8 2 3 4 5 . stop 20 0 1 6 ." + +do_install() { + install -d ${D}${sysconfdir}/init.d + install -m 0755 ${WORKDIR}/touch-ostree ${D}${sysconfdir}/init.d/touch-ostree +} diff --git a/external/meta-updater/recipes-test/demo-config/files/30-fake-pacman.toml b/external/meta-updater/recipes-test/demo-config/files/30-fake-pacman.toml deleted file mode 100644 index 3fb5cf2c..00000000 --- a/external/meta-updater/recipes-test/demo-config/files/30-fake-pacman.toml +++ /dev/null @@ -1,2 +0,0 @@ -[pacman] -type = "fake" diff --git a/external/meta-updater/recipes-test/demo-config/files/30-pacman-config.toml b/external/meta-updater/recipes-test/demo-config/files/30-pacman-config.toml new file mode 100644 index 00000000..750cf5c7 --- /dev/null +++ b/external/meta-updater/recipes-test/demo-config/files/30-pacman-config.toml @@ -0,0 +1,2 @@ +[pacman] +type = @UPDATE_TYPE@ diff --git a/external/meta-updater/recipes-test/demo-config/primary-config.bb b/external/meta-updater/recipes-test/demo-config/primary-config.bb index 27cb553e..5c8abb54 100644 --- a/external/meta-updater/recipes-test/demo-config/primary-config.bb +++ b/external/meta-updater/recipes-test/demo-config/primary-config.bb @@ -4,11 +4,14 @@ LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MPL-2.0;md5=815ca599c9df247a0c7 require shared-conf.inc +inherit allarch + PRIMARY_SECONDARIES ?= "${SECONDARY_IP}:${SECONDARY_PORT}" SRC_URI = "\ file://30-secondary-config.toml \ file://ip_secondary_config.json \ + ${@('file://' + d.getVar('SOTA_SECONDARY_CONFIG')) if d.getVar('SOTA_SECONDARY_CONFIG') else ''} \ " def get_secondary_addrs(d): diff --git a/external/meta-updater/recipes-test/demo-config/secondary-config.bb b/external/meta-updater/recipes-test/demo-config/secondary-config.bb index 9411646b..638f0aea 100644 --- a/external/meta-updater/recipes-test/demo-config/secondary-config.bb +++ b/external/meta-updater/recipes-test/demo-config/secondary-config.bb @@ -4,36 +4,55 @@ LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MPL-2.0;md5=815ca599c9df247a0c7 require shared-conf.inc +# Because of the dependency on MACHINE. +PACKAGE_ARCH = "${MACHINE_ARCH}" + SECONDARY_SERIAL_ID ?= "" SOTA_HARDWARE_ID ?= "${MACHINE}-sndry" SECONDARY_HARDWARE_ID ?= "${SOTA_HARDWARE_ID}" +SECONDARY_UPDATE_TYPE ?= "ostree" + +UPDATE_TYPE = "${SECONDARY_UPDATE_TYPE}" +python () { + update_type = d.getVar('UPDATE_TYPE') + if update_type not in [ 'ostree', 'file']: + bb.fatal('Unsupported type of an update specified for secondary: SECONDARY_UPDATE_TYPE = {}\n' + 'Supported update types are: ostree and file' + .format(update_type)) + + if update_type == 'file': + d.setVar('UPDATE_TYPE', 'none') +} SRC_URI = "\ - file://30-fake-pacman.toml \ + file://30-pacman-config.toml \ file://35-network-config.toml \ file://45-id-config.toml \ " + do_install () { install -m 0700 -d ${D}${libdir}/sota/conf.d - install -m 0644 ${WORKDIR}/30-fake-pacman.toml ${D}/${libdir}/sota/conf.d/30-fake-pacman.toml - install -m 0644 ${WORKDIR}/35-network-config.toml ${D}/${libdir}/sota/conf.d/35-network-config.toml + install -m 0644 ${WORKDIR}/30-pacman-config.toml ${D}${libdir}/sota/conf.d/30-pacman-config.toml + sed -i -e 's|@UPDATE_TYPE@|${UPDATE_TYPE}|g' ${D}${libdir}/sota/conf.d/30-pacman-config.toml + + install -m 0644 ${WORKDIR}/35-network-config.toml ${D}${libdir}/sota/conf.d/35-network-config.toml sed -i -e 's|@PORT@|${SECONDARY_PORT}|g' \ -e 's|@PRIMARY_IP@|${PRIMARY_IP}|g' \ -e 's|@PRIMARY_PORT@|${PRIMARY_PORT}|g' \ - ${D}/${libdir}/sota/conf.d/35-network-config.toml + ${D}${libdir}/sota/conf.d/35-network-config.toml - install -m 0644 ${WORKDIR}/45-id-config.toml ${D}/${libdir}/sota/conf.d/45-id-config.toml + install -m 0644 ${WORKDIR}/45-id-config.toml ${D}${libdir}/sota/conf.d/45-id-config.toml sed -i -e 's|@SERIAL@|${SECONDARY_SERIAL_ID}|g' \ -e 's|@HWID@|${SECONDARY_HARDWARE_ID}|g' \ - ${D}/${libdir}/sota/conf.d/45-id-config.toml + ${D}${libdir}/sota/conf.d/45-id-config.toml } FILES_${PN} = " \ ${libdir}/sota/conf.d \ - ${libdir}/sota/conf.d/30-fake-pacman.toml \ + ${libdir}/sota/conf.d/30-pacman-config.toml \ ${libdir}/sota/conf.d/35-network-config.toml \ ${libdir}/sota/conf.d/45-id-config.toml \ " diff --git a/external/meta-updater/recipes-test/demo-config/shared-conf.inc b/external/meta-updater/recipes-test/demo-config/shared-conf.inc index c5ab5987..55234068 100644 --- a/external/meta-updater/recipes-test/demo-config/shared-conf.inc +++ b/external/meta-updater/recipes-test/demo-config/shared-conf.inc @@ -1,5 +1,5 @@ -SECONDARY_IP ?= "10.0.3.2" +SECONDARY_IP ?= "192.168.254.2" SECONDARY_PORT ?= "9050" -PRIMARY_IP ?= "10.0.3.1" +PRIMARY_IP ?= "192.168.254.1" PRIMARY_PORT ?= "9040" PRIMARY_WAIT_TIMEOUT ?= "240" diff --git a/external/meta-updater/recipes-test/demo-network-config/files/25-dhcp-server.network b/external/meta-updater/recipes-test/demo-network-config/files/25-dhcp-server.network index 4766f9ae..03bb3023 100644 --- a/external/meta-updater/recipes-test/demo-network-config/files/25-dhcp-server.network +++ b/external/meta-updater/recipes-test/demo-network-config/files/25-dhcp-server.network @@ -4,9 +4,9 @@ Name=enp0s4 [Network] Description=Private internal network between aktualizr Primary and Secondary nodes DHCPServer=yes -Address=10.0.3.1/24 +Address=192.168.254.1/24 IPForward=yes IPMasquerade=yes [DHCPServer] -PoolOffset=10
\ No newline at end of file +PoolOffset=10 diff --git a/external/meta-updater/recipes-test/demo-network-config/network-config.inc b/external/meta-updater/recipes-test/demo-network-config/network-config.inc index ed623d46..b023f514 100644 --- a/external/meta-updater/recipes-test/demo-network-config/network-config.inc +++ b/external/meta-updater/recipes-test/demo-network-config/network-config.inc @@ -2,15 +2,18 @@ SRC_URI_append = "\ file://26-${CONF_TYPE}-client.network \ " +# Because of the dependency on MACHINE. +PACKAGE_ARCH = "${MACHINE_ARCH}" + SECONDARY_INTERFACE ?= "${@ 'eth0' if d.getVar('MACHINE') == 'raspberrypi3' else 'enp0s5'}" do_install_append() { bbnote "Network configuration type to be applied: ${CONF_TYPE}" - install -d ${D}/usr/lib/systemd/network - install -m 0644 ${WORKDIR}/26-${CONF_TYPE}-client.network ${D}/usr/lib/systemd/network/ + install -d ${D}${libdir}/systemd/network + install -m 0644 ${WORKDIR}/26-${CONF_TYPE}-client.network ${D}${libdir}/systemd/network/ sed -i -e 's|@ADDR@|${IP_ADDR}|g' \ -e 's|@IFNAME@|${SECONDARY_INTERFACE}|g' \ - ${D}/usr/lib/systemd/network/26-${CONF_TYPE}-client.network + ${D}${libdir}/systemd/network/26-${CONF_TYPE}-client.network } diff --git a/external/meta-updater/recipes-test/demo-network-config/primary-network-config.bb b/external/meta-updater/recipes-test/demo-network-config/primary-network-config.bb index d840a951..7ee873f4 100644 --- a/external/meta-updater/recipes-test/demo-network-config/primary-network-config.bb +++ b/external/meta-updater/recipes-test/demo-network-config/primary-network-config.bb @@ -2,22 +2,20 @@ DESCRIPTION = "Sample network configuration for an Uptane Primary" LICENSE = "MPL-2.0" LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MPL-2.0;md5=815ca599c9df247a0c7f619bab123dad" -inherit allarch - SRC_URI = "\ file://27-dhcp-client-external.network \ " -FILES_${PN} = "/usr/lib/systemd/network" +FILES_${PN} = "${libdir}/systemd/network" PR = "1" do_install() { - install -d ${D}/usr/lib/systemd/network - install -m 0644 ${WORKDIR}/27-dhcp-client-external.network ${D}/usr/lib/systemd/network/ + install -d ${D}${libdir}/systemd/network + install -m 0644 ${WORKDIR}/27-dhcp-client-external.network ${D}${libdir}/systemd/network/ } -PRIMARY_IP ?= "10.0.3.1" +PRIMARY_IP ?= "192.168.254.1" IP_ADDR = "${PRIMARY_IP}" CONF_TYPE ?= "${@ 'multihomed' if d.getVar('MACHINE') == 'raspberrypi3' and d.getVar('RPI_WIFI_ENABLE') != '1' else 'static'}" diff --git a/external/meta-updater/recipes-test/demo-network-config/secondary-network-config.bb b/external/meta-updater/recipes-test/demo-network-config/secondary-network-config.bb index b1d70f1f..b268cd38 100644 --- a/external/meta-updater/recipes-test/demo-network-config/secondary-network-config.bb +++ b/external/meta-updater/recipes-test/demo-network-config/secondary-network-config.bb @@ -2,8 +2,6 @@ DESCRIPTION = "Sample network configuration for an Uptane Secondary" LICENSE = "MPL-2.0" LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MPL-2.0;md5=815ca599c9df247a0c7f619bab123dad" -inherit allarch - # TODO: It configures the 'user' interface in NAT mode and provides an access to public Inet via it # which is not desired for Secondary. It cannot be just removed since we get SSH access to Secondary # VM via this interface. So, the task is to configure the interface in such way that it does provide access @@ -12,18 +10,18 @@ SRC_URI = "\ file://27-dhcp-client-external.network \ " -FILES_${PN} = "/usr/lib/systemd/network" +FILES_${PN} = "${libdir}/systemd/network" PR = "1" do_install() { - install -d ${D}/usr/lib/systemd/network - install -m 0644 ${WORKDIR}/27-dhcp-client-external.network ${D}/usr/lib/systemd/network/ + install -d ${D}${libdir}/systemd/network + install -m 0644 ${WORKDIR}/27-dhcp-client-external.network ${D}${libdir}/systemd/network/ } -SECONDARY_IP ?= "10.0.3.2" +SECONDARY_IP ?= "192.168.254.2" IP_ADDR = "${SECONDARY_IP}" -CONF_TYPE = "static" +CONF_TYPE ?= "${@ 'multihomed' if d.getVar('MACHINE') == 'raspberrypi3' and d.getVar('RPI_WIFI_ENABLE') != '1' else 'static'}" require network-config.inc diff --git a/external/meta-updater/recipes-test/images/secondary-image.bb b/external/meta-updater/recipes-test/images/secondary-image.bb index 27d1e3f9..7db2c684 100644 --- a/external/meta-updater/recipes-test/images/secondary-image.bb +++ b/external/meta-updater/recipes-test/images/secondary-image.bb @@ -14,7 +14,6 @@ IMAGE_INSTALL_remove = " \ aktualizr-shared-prov \ aktualizr-shared-prov-creds \ aktualizr-device-prov \ - aktualizr-device-prov-creds \ aktualizr-device-prov-hsm \ aktualizr-uboot-env-rollback \ virtual/network-configuration \ diff --git a/external/meta-updater/scripts/ci/Dockerfile.bitbake b/external/meta-updater/scripts/ci/Dockerfile.bitbake index c91f94c3..51eaa570 100644 --- a/external/meta-updater/scripts/ci/Dockerfile.bitbake +++ b/external/meta-updater/scripts/ci/Dockerfile.bitbake @@ -1,15 +1,17 @@ -FROM debian:stable +FROM debian:stretch LABEL Description="Image for bitbaking" -RUN sed -i 's#deb http://deb.debian.org/debian stable main#deb http://deb.debian.org/debian stable main contrib#g' /etc/apt/sources.list -RUN sed -i 's#deb http://deb.debian.org/debian stable-updates main#deb http://deb.debian.org/debian stable-updates main contrib#g' /etc/apt/sources.list -RUN apt-get update -q && apt-get install -qy \ +RUN sed -i 's#deb http://deb.debian.org/debian stretch main#deb http://deb.debian.org/debian stretch main contrib#g' /etc/apt/sources.list +RUN sed -i 's#deb http://deb.debian.org/debian stretch-updates main#deb http://deb.debian.org/debian stretch-updates main contrib#g' /etc/apt/sources.list +RUN apt-get update -q && apt-get install --no-install-suggests --no-install-recommends -qy \ + awscli \ build-essential \ bzip2 \ chrpath \ cpio \ default-jre \ diffstat \ + file \ gawk \ gcc-multilib \ git-core \ @@ -17,26 +19,40 @@ RUN apt-get update -q && apt-get install -qy \ iproute \ libpython-dev \ libsdl1.2-dev \ + libvirt-clients \ + libvirt-daemon-system \ locales \ ovmf \ + openssh-client \ procps \ python \ python3 \ python3-pexpect \ - qemu \ + qemu-kvm \ socat \ + sudo \ texinfo \ unzip \ wget \ xterm \ xz-utils -ARG uid=1000 -ARG gid=1000 +ARG uid=4321 +ARG gid=4321 RUN groupadd -g $gid bitbake -RUN useradd -m -u $uid -g $gid bitbake +RUN useradd -m -u $uid -g $gid -s /bin/bash bitbake RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && locale-gen ENV LC_ALL="en_US.UTF-8" ENV LANG="en_US.UTF-8" ENV LANGUAGE="en_US.UTF-8" + +# script to mirror kvm group id with host +RUN echo "bitbake ALL=NOPASSWD: /usr/local/bin/setup_kvm.sh" >> /etc/sudoers +COPY ./docker/setup_kvm.sh /usr/local/bin/setup_kvm.sh + +# other ci scripts +RUN mkdir /scripts +COPY configure.sh build.sh oe-selftest.sh /scripts/ + +USER "bitbake" diff --git a/external/meta-updater/scripts/ci/Jenkinsfile.bleeding b/external/meta-updater/scripts/ci/Jenkinsfile.bleeding deleted file mode 100644 index 6d340fde..00000000 --- a/external/meta-updater/scripts/ci/Jenkinsfile.bleeding +++ /dev/null @@ -1,87 +0,0 @@ -// This CI setup checks out aktualizr, meta-updater and updater-repo and builds -// master branches whenever a change is pushed to any of these - -// define these for docker image creation -node { - // might cause some problems: - // https://stackoverflow.com/questions/44805076/setting-build-args-for-dockerfile-agent-using-a-jenkins-declarative-pipeline - JENKINS_UID = sh(returnStdout: true, script: 'id -u').trim() - JENKINS_GID = sh(returnStdout: true, script: 'id -g').trim() -} - -pipeline { - agent any - environment { - TEST_AKTUALIZR_REMOTE = 'aktualizr' - TEST_AKTUALIZR_DIR = 'aktualizr' - TEST_AKTUALIZR_BRANCH = 'master' - TEST_BITBAKE_COMMON_DIR = "/opt/jenkins/bitbake-common" - } - stages { - stage('checkout') { - steps { - - checkout([$class: 'GitSCM', - userRemoteConfigs: [ - [url: 'https://github.com/advancedtelematic/aktualizr', name: 'aktualizr'] - ], - branches: [[name: 'refs/heads/master']], - extensions: [ - [$class: 'DisableRemotePoll'], - [$class: 'PruneStaleBranch'], - [$class: 'RelativeTargetDirectory', - relativeTargetDir: 'aktualizr' - ] - ], - ]) - - checkout([$class: 'RepoScm', - manifestRepositoryUrl: 'https://github.com/advancedtelematic/updater-repo', - manifestBranch: null, - manifestFile: 'master.xml', - manifestGroup: null, - mirrorDir: null, - jobs: 0, - depth: 0, - localManifest: null, - destinationDir: 'updater-repo', - repoUrl: null, - currentBranch: false, - resetFirst: true, - quiet: false, - trace: false, - showAllChanges: false, - ]) - - // ignore bitbake build directories in docker - sh 'echo \'build*\' > .dockerignore' - - // override meta-updater commit with currently tested branch - sh ''' - META_UPDATER_COMMIT=$(git rev-parse HEAD) - cd updater-repo/meta-updater - git checkout $META_UPDATER_COMMIT - ''' - } - } - stage('build-core-image-minimal') { - agent { - dockerfile { - filename 'scripts/ci/Dockerfile.bitbake' - args '-v /opt/jenkins/bitbake-common:/opt/jenkins/bitbake-common' - additionalBuildArgs "--build-arg uid=${JENKINS_UID} --build-arg gid=${JENKINS_GID}" - reuseNode true - } - } - environment { - TEST_AKTUALIZR_CREDENTIALS = credentials('garage-credentials') - } - steps { - sh 'scripts/ci/configure.sh' - - sh 'scripts/ci/build.sh core-image-minimal' - } - } - } -} -// vim: set ft=groovy tabstop=2 shiftwidth=2 expandtab: diff --git a/external/meta-updater/scripts/ci/Jenkinsfile.bleeding-selftest b/external/meta-updater/scripts/ci/Jenkinsfile.bleeding-selftest deleted file mode 100644 index 8c2d1de6..00000000 --- a/external/meta-updater/scripts/ci/Jenkinsfile.bleeding-selftest +++ /dev/null @@ -1,91 +0,0 @@ -// This CI setup checks out aktualizr, meta-updater and updater-repo and builds -// master branches whenever a change is pushed to any of these - -// define these for docker image creation -node { - // might cause some problems: - // https://stackoverflow.com/questions/44805076/setting-build-args-for-dockerfile-agent-using-a-jenkins-declarative-pipeline - JENKINS_UID = sh(returnStdout: true, script: 'id -u').trim() - JENKINS_GID = sh(returnStdout: true, script: 'id -g').trim() -} - -pipeline { - agent { - node { label 'bitbake' } - } - environment { - TEST_AKTUALIZR_REMOTE = 'aktualizr' - TEST_AKTUALIZR_DIR = 'aktualizr' - TEST_AKTUALIZR_BRANCH = 'master' - TEST_BITBAKE_COMMON_DIR = "/opt/jenkins/bitbake-common" - } - stages { - stage('checkout') { - steps { - - checkout([$class: 'GitSCM', - userRemoteConfigs: [ - [url: 'https://github.com/advancedtelematic/aktualizr', name: 'aktualizr'] - ], - branches: [[name: 'refs/heads/master']], - extensions: [ - [$class: 'DisableRemotePoll'], - [$class: 'PruneStaleBranch'], - [$class: 'RelativeTargetDirectory', - relativeTargetDir: 'aktualizr' - ] - ], - ]) - - checkout([$class: 'RepoScm', - manifestRepositoryUrl: 'https://github.com/advancedtelematic/updater-repo', - manifestBranch: null, - manifestFile: 'master.xml', - manifestGroup: null, - mirrorDir: null, - jobs: 0, - depth: 0, - localManifest: null, - destinationDir: 'updater-repo', - repoUrl: null, - currentBranch: false, - resetFirst: true, - quiet: false, - trace: false, - showAllChanges: false, - ]) - - // ignore bitbake build directories in docker - sh 'echo \'build*\' > .dockerignore' - - // override meta-updater commit with currently tested branch - sh ''' - META_UPDATER_COMMIT=$(git rev-parse HEAD) - cd updater-repo/meta-updater - git checkout $META_UPDATER_COMMIT - ''' - } - } - stage('build-core-image-minimal+oe-selftest') { - agent { - dockerfile { - filename 'scripts/ci/Dockerfile.bitbake' - args '-v /opt/jenkins/bitbake-common:/opt/jenkins/bitbake-common' - additionalBuildArgs "--build-arg uid=${JENKINS_UID} --build-arg gid=${JENKINS_GID}" - reuseNode true - } - } - environment { - TEST_AKTUALIZR_CREDENTIALS = credentials('garage-credentials') - } - steps { - sh 'scripts/ci/configure.sh' - - sh 'scripts/ci/build.sh core-image-minimal' - - sh 'scripts/ci/oe-selftest.sh' - } - } - } -} -// vim: set ft=groovy tabstop=2 shiftwidth=2 expandtab: diff --git a/external/meta-updater/scripts/ci/README.adoc b/external/meta-updater/scripts/ci/README.adoc deleted file mode 100644 index 222982b1..00000000 --- a/external/meta-updater/scripts/ci/README.adoc +++ /dev/null @@ -1,14 +0,0 @@ -= Jenkins setup for running meta-updater CI - -As bitbake is quite resource-hungry, there are some special steps that are -needed to run Jenkins CI tasks: - -- docker should be installed and the `jenkins` unix user should belong to - the `docker` group -- `/opt/jenkins` should exist and have `jenkins:jenkins` permissions, it - will be mapped as a volume on the same location in the docker build - container - -Note that for nodes running Jenkins slaves as a docker container, the -`/opt/jenkins` directory must exist on the host system as well, with -permissions matching the user and groupd ids in Jenkins' docker diff --git a/external/meta-updater/scripts/ci/build.sh b/external/meta-updater/scripts/ci/build.sh index 62354289..9fbae989 100755 --- a/external/meta-updater/scripts/ci/build.sh +++ b/external/meta-updater/scripts/ci/build.sh @@ -12,7 +12,10 @@ IMAGE_NAME=${1:-core-image-minimal} ( set +euo pipefail set +x +METADIR=$(realpath "$TEST_REPO_DIR") +export METADIR . "${TEST_REPO_DIR}/meta-updater/scripts/envsetup.sh" "${TEST_MACHINE}" "${TEST_BUILD_DIR}" +set -x bitbake "${IMAGE_NAME}" ) diff --git a/external/meta-updater/scripts/ci/configure.sh b/external/meta-updater/scripts/ci/configure.sh index 960a0cc9..ae78f066 100755 --- a/external/meta-updater/scripts/ci/configure.sh +++ b/external/meta-updater/scripts/ci/configure.sh @@ -8,9 +8,19 @@ TEST_BUILD_DIR=${TEST_BUILD_DIR:-build} TEST_REPO_DIR=${TEST_REPO_DIR:-updater-repo} TEST_BITBAKE_COMMON_DIR=${TEST_BITBAKE_COMMON_DIR:-} -TEST_AKTUALIZR_DIR=${TEST_AKTUALIZR_DIR:-.} -TEST_AKTUALIZR_BRANCH=${TEST_AKTUALIZR_BRANCH:-master} -TEST_AKTUALIZR_REV=${TEST_AKTUALIZR_REV:-$(GIT_DIR="$TEST_AKTUALIZR_DIR/.git" git rev-parse "$TEST_AKTUALIZR_REMOTE/$TEST_AKTUALIZR_BRANCH")} +TEST_AKTUALIZR_REMOTE=${TEST_AKTUALIZR_REMOTE:-} +TEST_AKTUALIZR_TAG=${TEST_AKTUALIZR_TAG:-} +if [ -n "$TEST_AKTUALIZR_REMOTE" ]; then + if [ -n "$TEST_AKTUALIZR_TAG" ]; then + TEST_AKTUALIZR_BRANCH="" + TEST_AKTUALIZR_REV="" + else + TEST_AKTUALIZR_DIR=${TEST_AKTUALIZR_DIR:-.} + TEST_AKTUALIZR_BRANCH=${TEST_AKTUALIZR_BRANCH:-master} + TEST_AKTUALIZR_REV=${TEST_AKTUALIZR_REV:-$(GIT_DIR="$TEST_AKTUALIZR_DIR/.git" git rev-parse "$TEST_AKTUALIZR_REMOTE/$TEST_AKTUALIZR_BRANCH")} + fi +fi + TEST_AKTUALIZR_CREDENTIALS=${TEST_AKTUALIZR_CREDENTIALS:-} # move existing conf directory to backup, before generating a new one @@ -21,6 +31,8 @@ mv "$TEST_BUILD_DIR/conf" "$TEST_BUILD_DIR/conf.old" || true set +euo pipefail set +x echo ">> Running envsetup.sh" +METADIR=$(realpath "$TEST_REPO_DIR") +export METADIR . "$TEST_REPO_DIR/meta-updater/scripts/envsetup.sh" "$TEST_MACHINE" "$TEST_BUILD_DIR" ) @@ -31,19 +43,30 @@ SITE_CONF="$TEST_BUILD_DIR/conf/site.conf" echo ">> Set common bitbake config options" cat << EOF > "$SITE_CONF" SANITY_TESTED_DISTROS = "" -SSTATE_MIRRORS ?= "file://.* https://bitbake-cache.atsgarage.com/PATH;downloadfilename=PATH" IMAGE_FEATURES += "ssh-server-openssh" EOF -echo ">> Set aktualizr branch in bitbake's config" -cat << EOF >> "$SITE_CONF" +if [ -n "$TEST_AKTUALIZR_REMOTE" ]; then + echo ">> Set aktualizr branch in bitbake's config" + if [ -n "$TEST_AKTUALIZR_TAG" ]; then + # tag case + cat << EOF >> "$SITE_CONF" +SRCREV_pn-aktualizr = "" +SRCREV_pn-aktualizr-native = "" +BRANCH_pn-aktualizr = ";nobranch=1;tag=$TEST_AKTUALIZR_TAG" +BRANCH_pn-aktualizr-native = "\${BRANCH_pn-aktualizr}" +EOF + else + # branch case + cat << EOF >> "$SITE_CONF" SRCREV_pn-aktualizr = "$TEST_AKTUALIZR_REV" SRCREV_pn-aktualizr-native = "\${SRCREV_pn-aktualizr}" BRANCH_pn-aktualizr = "$TEST_AKTUALIZR_BRANCH" BRANCH_pn-aktualizr-native = "\${BRANCH_pn-aktualizr}" - EOF + fi +fi if [[ -n $TEST_AKTUALIZR_CREDENTIALS ]]; then echo ">> Set aktualizr credentials" @@ -63,3 +86,6 @@ SSTATE_DIR = "$SSTATE_DIR" DL_DIR = "$DL_DIR" EOF fi + +echo -e ">> Final configuration (site.conf):\\n" +cat "$SITE_CONF" diff --git a/external/meta-updater/scripts/ci/oe-selftest.sh b/external/meta-updater/scripts/ci/oe-selftest.sh index 3124cce1..d441d027 100755 --- a/external/meta-updater/scripts/ci/oe-selftest.sh +++ b/external/meta-updater/scripts/ci/oe-selftest.sh @@ -12,7 +12,19 @@ TEST_REPO_DIR=${TEST_REPO_DIR:-updater-repo} ( set +euo pipefail set +x +METADIR=$(realpath "$TEST_REPO_DIR") +export METADIR . "${TEST_REPO_DIR}/meta-updater/scripts/envsetup.sh" "${TEST_MACHINE}" "${TEST_BUILD_DIR}" -oe-selftest -r updater +set -x + +# work poky around bug on sumo and thud +# see https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=d3a94e5b9b3c107cf54d5639071cc6609c002f67 +mkdir -p "tmp/log" + +# This is apparently required here now as well. +git config --global user.email "meta-updater-ci@example.org" +git config --global user.name "meta-updater-ci" + +oe-selftest -r "$@" ) diff --git a/external/meta-updater/scripts/envsetup.sh b/external/meta-updater/scripts/envsetup.sh index 5827bc2a..a7ee877d 100755 --- a/external/meta-updater/scripts/envsetup.sh +++ b/external/meta-updater/scripts/envsetup.sh @@ -3,15 +3,26 @@ SCRIPT="envsetup.sh" MACHINE="$1" BUILDDIR="build" +DISTRO="poky-sota-systemd" +BASE_CONF="local.conf.base.append" -[[ "$#" -lt 1 ]] && { echo "Usage: ${SCRIPT} <machine> [builddir]"; return 1; } -[[ "$#" -eq 2 ]] && { BUILDDIR="$2"; } +# A definition of a dictionary with a list of configuration files that must be appended +# to resulting conf/local.conf file for each particular distribution. +declare -A supported_distros=( + ["poky-sota-systemd"]="local.conf.systemd.append" + ["poky-sota"]="local.conf.base.append" + ["poky"]="local.conf.systemd.append local.conf.nonostree.append" +) + +[[ "$#" -lt 1 ]] && { echo "Usage: ${SCRIPT} <machine> [builddir] [distro=< poky-sota-systemd | poky-sota | poky >]"; return 1; } +[[ "$#" -ge 2 ]] && { BUILDDIR="$2"; } +[[ "$#" -eq 3 ]] && { DISTRO="$3"; } # detect if this script is sourced: see http://stackoverflow.com/a/38128348/6255594 SOURCED=0 -if [ -n "$ZSH_EVAL_CONTEXT" ]; then +if [[ -n "$ZSH_EVAL_CONTEXT" ]]; then [[ "$ZSH_EVAL_CONTEXT" =~ :file$ ]] && { SOURCED=1; SOURCEDIR=$(cd "$(dirname -- "$0")" && pwd -P); } -elif [ -n "$BASH_VERSION" ]; then +elif [[ -n "$BASH_VERSION" ]]; then [[ "$0" != "${BASH_SOURCE[0]}" ]] && { SOURCED=1; SOURCEDIR=$(cd "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P); } fi @@ -24,13 +35,23 @@ fi METADIR=${METADIR:-${SOURCEDIR}/../..} if [[ ! -f "${BUILDDIR}/conf/local.conf" ]]; then + declare -a DISTRO_CONFIGS=${supported_distros[$DISTRO]} + [[ -n ${DISTRO_CONFIGS[@]} ]] && { echo "Using (${DISTRO_CONFIGS[*]}) for the specified distro '$DISTRO'"; } || { echo "The specified distro $DISTRO is not supported"; return 1; } + source "$METADIR/poky/oe-init-build-env" "$BUILDDIR" echo "METADIR := \"\${@os.path.abspath('${METADIR}')}\"" >> conf/bblayers.conf cat "${METADIR}/meta-updater/conf/include/bblayers/sota.inc" >> conf/bblayers.conf cat "${METADIR}/meta-updater/conf/include/bblayers/sota_${MACHINE}.inc" >> conf/bblayers.conf - - sed "s/##MACHINE##/$MACHINE/g" "${METADIR}/meta-updater/conf/local.conf.sample.append" >> conf/local.conf + sed -e "s/##MACHINE##/$MACHINE/g" \ + -e "s/##DISTRO##/$DISTRO/g" \ + "${METADIR}/meta-updater/conf/$BASE_CONF" >> conf/local.conf + + for config in ${DISTRO_CONFIGS[@]}; do + if [[ "$BASE_CONF" != "$config" ]]; then + cat "${METADIR}/meta-updater/conf/$config" >> conf/local.conf + fi + done else source "$METADIR/poky/oe-init-build-env" "$BUILDDIR" -fi +fi
\ No newline at end of file diff --git a/external/meta-updater/scripts/find_aktualizr_dependencies.sh b/external/meta-updater/scripts/find_aktualizr_dependencies.sh index 493df800..fcb2f97e 100755 --- a/external/meta-updater/scripts/find_aktualizr_dependencies.sh +++ b/external/meta-updater/scripts/find_aktualizr_dependencies.sh @@ -13,7 +13,6 @@ ${parentdir}/find_dependencies.py aktualizr ${parentdir}/find_dependencies.py aktualizr-shared-prov ${parentdir}/find_dependencies.py aktualizr-shared-prov-creds ${parentdir}/find_dependencies.py aktualizr-device-prov -${parentdir}/find_dependencies.py aktualizr-device-prov-creds ${parentdir}/find_dependencies.py aktualizr-device-prov-hsm ${parentdir}/find_dependencies.py aktualizr-auto-reboot ${parentdir}/find_dependencies.py aktualizr-disable-send-ip diff --git a/external/meta-updater/scripts/qemucommand.py b/external/meta-updater/scripts/qemucommand.py index 3045b454..30929acc 100644 --- a/external/meta-updater/scripts/qemucommand.py +++ b/external/meta-updater/scripts/qemucommand.py @@ -1,7 +1,8 @@ -from os.path import exists, join, realpath, abspath +from os.path import exists, isdir, join, realpath, abspath from os import listdir import random import socket +from shutil import copyfile from subprocess import check_output EXTENSIONS = { @@ -39,29 +40,84 @@ def random_mac(): class QemuCommand(object): def __init__(self, args): + self.enable_u_boot = True + self.dry_run = args.dry_run + self.overlay = args.overlay + self.host_fwd = None + self.kernel = None + self.drive_interface = "ide" + + if hasattr(args, 'uboot_enable'): + self.enable_u_boot = args.uboot_enable.lower() in ("yes", "true", "1") + + # Rise an exception if U-Boot is disabled and overlay option is used + if not self.enable_u_boot and self.overlay: + raise EnvironmentError("An overlay option is currently supported only with U-Boot loader!") + + # If booting with u-boot is disabled we use "ext4" root fs instead of custom one "ota-ext4" + if not self.enable_u_boot: + self.drive_interface = "virtio" + EXTENSIONS['qemux86-64'] = 'ext4' + if args.machine: self.machine = args.machine else: + if not isdir(args.dir): + raise ValueError("Directory %s does not exist, please specify a --machine or a valid images directory" % args.dir) machines = listdir(args.dir) if len(machines) == 1: self.machine = machines[0] else: raise ValueError("Could not autodetect machine type. More than one entry in %s. Maybe --machine qemux86-64?" % args.dir) + + # If using an overlay with U-Boot, copy the rom when we create the + # overlay so that we can keep it around just in case. if args.efi: self.bios = 'OVMF.fd' + elif self.enable_u_boot: + uboot_path = abspath(join(args.dir, self.machine, 'u-boot-qemux86-64.rom')) + if self.overlay: + new_uboot_path = self.overlay + '.u-boot.rom' + if not exists(self.overlay): + if not exists(uboot_path): + raise ValueError("U-Boot image %s does not exist" % uboot_path) + if not exists(new_uboot_path): + if self.dry_run: + print("cp %s %s" % (uboot_path, new_uboot_path)) + else: + copyfile(uboot_path, new_uboot_path) + uboot_path = new_uboot_path + if not exists(uboot_path) and not (self.dry_run and not exists(self.overlay)): + raise ValueError("U-Boot image %s does not exist" % uboot_path) + self.bios = uboot_path else: - uboot = abspath(join(args.dir, self.machine, 'u-boot-qemux86-64.rom')) - if not exists(uboot): - raise ValueError("U-Boot image %s does not exist" % uboot) - self.bios = uboot + self.kernel = abspath(join(args.dir, self.machine, 'bzImage-qemux86-64.bin')) + + # If using an overlay, we need to keep the "backing" image around, as + # bitbake will often clean it up, and the overlay silently depends on + # the hardcoded path. The easiest solution is to keep the file and use + # a relative path to it. if exists(args.imagename): - image = args.imagename + image = realpath(args.imagename) else: ext = EXTENSIONS.get(self.machine, 'wic') image = join(args.dir, self.machine, '%s-%s.%s' % (args.imagename, self.machine, ext)) - self.image = realpath(image) - if not exists(self.image): + if self.overlay: + new_image_path = self.overlay + '.img' + if not exists(self.overlay): + if not exists(image): + raise ValueError("OS image %s does not exist" % image) + if not exists(new_image_path): + if self.dry_run: + print("cp %s %s" % (image, new_image_path)) + else: + copyfile(image, new_image_path) + self.image = new_image_path + else: + self.image = realpath(image) + if not exists(self.image) and not (self.dry_run and not exists(self.overlay)): raise ValueError("OS image %s does not exist" % self.image) + if args.mac: self.mac_address = args.mac else: @@ -84,28 +140,34 @@ class QemuCommand(object): self.gui = not args.no_gui self.gdb = args.gdb self.pcap = args.pcap - self.overlay = args.overlay self.secondary_network = args.secondary_network + # Append additional port forwarding to QEMU command line. + if hasattr(args, 'host_forward'): + self.host_fwd = args.host_forward + def command_line(self): netuser = 'user,hostfwd=tcp:0.0.0.0:%d-:22,restrict=off' % self.ssh_port if self.gdb: netuser += ',hostfwd=tcp:0.0.0.0:2159-:2159' + if self.host_fwd: + netuser += ",hostfwd=" + self.host_fwd + cmdline = [ "qemu-system-x86_64", - "-bios", self.bios ] + if self.enable_u_boot: + cmdline += ["-bios", self.bios] + else: + cmdline += ["-kernel", self.kernel] + if not self.overlay: - cmdline += ["-drive", "file=%s,if=ide,format=raw,snapshot=on" % self.image] + cmdline += ["-drive", "file=%s,if=%s,format=raw,snapshot=on" % (self.image, self.drive_interface)] cmdline += [ "-serial", "tcp:127.0.0.1:%d,server,nowait" % self.serial_port, "-m", self.mem, - "-usb", "-object", "rng-random,id=rng0,filename=/dev/urandom", "-device", "virtio-rng-pci,rng=rng0", - "-device", "usb-tablet", - "-show-cursor", - "-vga", "std", "-net", netuser, "-net", "nic,macaddr=%s" % self.mac_address ] @@ -117,15 +179,27 @@ class QemuCommand(object): '-device', 'e1000,netdev=vlan1,mac='+random_mac(), ] if self.gui: - cmdline += ["-serial", "stdio"] + cmdline += [ + "-usb", + "-device", "usb-tablet", + "-show-cursor", + "-vga", "std" + ] else: - cmdline.append('-nographic') + cmdline += [ + "-nographic", + "-monitor", "null", + ] if self.kvm: cmdline += ['-enable-kvm', '-cpu', 'host'] else: cmdline += ['-cpu', 'Haswell'] if self.overlay: cmdline.append(self.overlay) + + # If booting with u-boot is disabled, add kernel command line arguments through qemu -append option + if not self.enable_u_boot: + cmdline += ["-append", "root=/dev/vda rw highres=off console=ttyS0 ip=dhcp"] return cmdline def img_command_line(self): diff --git a/external/meta-updater/scripts/run-qemu-ota b/external/meta-updater/scripts/run-qemu-ota index de632970..59301a43 100755 --- a/external/meta-updater/scripts/run-qemu-ota +++ b/external/meta-updater/scripts/run-qemu-ota @@ -2,7 +2,7 @@ from argparse import ArgumentParser from subprocess import Popen -from os.path import exists +from os.path import exists, dirname import sys from qemucommand import QemuCommand @@ -13,6 +13,9 @@ def main(): parser = ArgumentParser(description='Run meta-updater image in qemu') parser.add_argument('imagename', default='core-image-minimal', nargs='?', help="Either the name of the bitbake image target, or a path to the image to run") + parser.add_argument('--uboot-enable', default='yes', + help='(yes/no). Determines whether or not to use U-Boot loader for running image, ' + 'if yes then u-boot binary file will be passed as -bios option into QEMU cmd line.') parser.add_argument('mac', default=None, nargs='?') parser.add_argument('--dir', default=DEFAULT_DIR, help='Path to build directory containing the image and u-boot-qemux86-64.rom') @@ -20,6 +23,7 @@ def main(): help='Boot using UEFI rather than U-Boot. This requires the image to be built with ' + 'OSTREE_BOOTLOADER = "grub" and OVMF.fd firmware to be installed (try "apt install ovmf")', action='store_true') + parser.add_argument('--bootloader', default=None, help="Path to bootloader, e.g. a u-boot ROM") parser.add_argument('--machine', default=None, help="Target MACHINE") kvm_group = parser.add_argument_group() kvm_group.add_argument('--force-kvm', help='Force use of KVM (default is to autodetect)', @@ -38,28 +42,44 @@ def main(): help='Give the image a second network card connected to a virtual network. ' + 'This can be used to test Uptane Primary/Secondary communication.') parser.add_argument('-n', '--dry-run', help='Print qemu command line rather then run it', action='store_true') + parser.add_argument('--host-forward', + help='Redirect incoming TCP or UDP connections to the host port. ' + 'Example forwarding guest port 10050 to the host port 10555:' + '--host-forward="tcp:0.0.0.0:10556-:10050". ' + 'For more details please refer to QEMU man page, option <hostfwd>. ' + 'https://manpages.debian.org/testing/qemu-system-x86/qemu-system-x86_64.1.en.html') args = parser.parse_args() + + if args.overlay and not exists(args.overlay) and dirname(args.overlay) and not dirname(args.overlay) == '.': + print('Error: please provide a file name in the current working directory. ' + + 'Overlays do not work properly with other directories.') + sys.exit(1) + if args.overlay and exists(args.overlay) and args.imagename != parser.get_default('imagename'): + # qemu-img amend -o <filename> might work, but it has not yet been done + # successfully. + print('Warning: cannot change backing image of overlay after it has been created.') + try: qemu_command = QemuCommand(args) except ValueError as e: print(e.message) sys.exit(1) - print("Launching %s with mac address %s" % (args.imagename, qemu_command.mac_address)) - print("To connect via SSH:") - print(" ssh -o StrictHostKeyChecking=no root@localhost -p %d" % qemu_command.ssh_port) - print("To connect to the serial console:") - print(" nc localhost %d" % qemu_command.serial_port) - cmdline = qemu_command.command_line() if args.overlay and not exists(args.overlay): - print("Image file %s does not yet exist, creating." % args.overlay) + print("Overlay file %s does not yet exist, creating." % args.overlay) img_cmdline = qemu_command.img_command_line() if args.dry_run: print(" ".join(img_cmdline)) else: Popen(img_cmdline).wait() + print("Launching %s with mac address %s" % (args.imagename, qemu_command.mac_address)) + print("To connect via SSH:") + print(" ssh -o StrictHostKeyChecking=no root@localhost -p %d" % qemu_command.ssh_port) + print("To connect to the serial console:") + print(" nc localhost %d" % qemu_command.serial_port) + if args.dry_run: print(" ".join(cmdline)) else: |