SPEC-3723: restructure meta-agl

Goal is to reach a minimal meta-agl-core as base for IVI and IC work at the same time.
Trim dependencies and move most 'demo' related recipes to meta-agl-demo.

v2: changed to bbapend + .inc , added description
v3: testbuild of all images
v4: restore -test packagegroup and -qa images, compare manifests and adapt packagegroups.
v5: rebased
v6: merged meta-agl-distro into meta-agl-core,
    due to dependency on meta-oe, moved -test packagegroup and -qa images
      to own layer meta-agl-core-test
v7: Fixed comments from Paul Barker
v8: Update the markdown files
v9: restore wayland/weston/agl-compositor recipes/appends, reworked to
    move app f/w specific changes to bbappends in meta-app-framework and
    only demo specific weston-init changes to meta-agl-demo
v10: fix s/agldemo/aglcore/ missed in weston-init.bbappend

Description:

This patch is part 1 out of 2 large patches that implement the layer rework
discussed during the previous workshop. Essentially meta-agl-core is the
small but versatile new core layer of AGL serving as basis for
the work done by the IC and IVI EGs.
All demo related work is moved to meta-agl-demo in the 2nd patchset.
This should be applied together as atomic change.

The resulting meta-agl/* follows these guidelines:
- only bsp adaptations in meta-agl-bsp
- remove the agl-profile-* layers for simplicity
-- the packagegroup-agl(-profile)-graphical and so on
   have been kept in meta-agl-demo
- meta-agl-profile-core is now meta-agl-core
- meta-agl-core does pass yocto-check-layer
-- therefore use the bbappend + conditional + .inc file
   construct found in meta-virtualization
- meta-agl/meta-security has been merged into meta-agl/meta-app-framework
- meta-netboot does pass yocto-check-layer
- meta-pipewire does pass yocto-check-layer

Migration:
All packagegroups are preserved but they're now enabled by 'agl-demo'.

Bug-AGL: SPEC-3723

Signed-off-by: Jan-Simon Moeller <jsmoeller@linuxfoundation.org>
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Change-Id: Ia6c6e5e6ce2b4ffa69ea94959cdc57c310ba7c53
Reviewed-on: https://gerrit.automotivelinux.org/gerrit/c/AGL/meta-agl/+/25769

3 files changed, 219 insertions, 0 deletions

diff --git a/meta-agl-core/scripts/oe-depends-dot b/meta-agl-core/scripts/oe-depends-dot
new file mode 100755
index 000000000..5cec23bf0
--- /dev/null
+++ b/meta-agl-core/scripts/oe-depends-dot
@@ -0,0 +1,121 @@
+#!/usr/bin/env python3
+#
+# Copyright (C) 2018 Wind River Systems, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import os
+import sys
+import argparse
+import logging
+import re
+
+class Dot(object):
+    def __init__(self):
+        parser = argparse.ArgumentParser(
+            description="Analyse recipe-depends.dot generated by bitbake -g",
+            epilog="Use %(prog)s --help to get help")
+        parser.add_argument("dotfile",
+            help = "Specify the dotfile", nargs = 1, action='store', default='')
+        parser.add_argument("-k", "--key",
+            help = "Specify the key, e.g., recipe name",
+            action="store", default='')
+        parser.add_argument("-d", "--depends",
+            help = "Print the key's dependencies",
+            action="store_true", default=False)
+        parser.add_argument("-w", "--why",
+            help = "Print why the key is built",
+            action="store_true", default=False)
+        parser.add_argument("-r", "--remove",
+            help = "Remove duplicated dependencies to reduce the size of the dot files."
+                    " For example, A->B, B->C, A->C, then A->C can be removed.",
+            action="store_true", default=False)
+
+        self.args = parser.parse_args()
+
+        if len(sys.argv) != 3 and len(sys.argv) < 5:
+            print('ERROR: Not enough args, see --help for usage')
+
+    def main(self):
+        #print(self.args.dotfile[0])
+        # The format is {key: depends}
+        depends = {}
+        with open(self.args.dotfile[0], 'r') as f:
+            for line in f.readlines():
+                if ' -> ' not in line:
+                    continue
+                line_no_quotes = line.replace('"', '')
+                m = re.match("(.*) -> (.*)", line_no_quotes)
+                if not m:
+                    print('WARNING: Found unexpected line: %s' % line)
+                    continue
+                key = m.group(1)
+                if key == "meta-world-pkgdata":
+                    continue
+                dep = m.group(2)
+                if key in depends:
+                    if not key in depends[key]:
+                        depends[key].add(dep)
+                    else:
+                        print('WARNING: Fonud duplicated line: %s' % line)
+                else:
+                    depends[key] = set()
+                    depends[key].add(dep)
+
+        if self.args.remove:
+            reduced_depends = {}
+            for k, deps in depends.items():
+                child_deps = set()
+                added = set()
+                # Both direct and indirect depends are already in the dict, so
+                # we don't have to do this recursively.
+                for dep in deps:
+                    if dep in depends:
+                        child_deps |= depends[dep]
+
+                reduced_depends[k] = deps - child_deps
+                outfile= '%s-reduced%s' % (self.args.dotfile[0][:-4], self.args.dotfile[0][-4:])
+            with open(outfile, 'w') as f:
+                print('Saving reduced dot file to %s' % outfile)
+                f.write('digraph depends {\n')
+                for k, v in reduced_depends.items():
+                    for dep in v:
+                        f.write('"%s" -> "%s"\n' % (k, dep))
+                f.write('}\n')
+            sys.exit(0)
+
+        if self.args.key not in depends:
+            print("ERROR: Can't find key %s in %s" % (self.args.key, self.args.dotfile[0]))
+            sys.exit(1)
+
+        if self.args.depends:
+            if self.args.key in depends:
+                print('Depends: %s' % ' '.join(depends[self.args.key]))
+
+        reverse_deps = []
+        if self.args.why:
+            for k, v in depends.items():
+                if self.args.key in v and not k in reverse_deps:
+                    reverse_deps.append(k)
+            print('Because: %s' % ' '.join(reverse_deps))
+
+if __name__ == "__main__":
+    try:
+        dot = Dot()
+        ret = dot.main()
+    except Exception as esc:
+        ret = 1
+        import traceback
+        traceback.print_exc()
+    sys.exit(ret)
diff --git a/meta-agl-core/scripts/run-yocto-check-layer-feature-enabled.sh b/meta-agl-core/scripts/run-yocto-check-layer-feature-enabled.sh
new file mode 100755
index 000000000..93403b974
--- /dev/null
+++ b/meta-agl-core/scripts/run-yocto-check-layer-feature-enabled.sh
@@ -0,0 +1,50 @@
+#!/bin/bash
+set -x
+
+SCRIPTPATH="$( cd $(dirname $0) >/dev/null 2>&1 ; pwd -P )"
+echo $SCRIPTPATH
+AGLROOT="$SCRIPTPATH/../../.."
+POKYDIR="$AGLROOT/external/poky"
+TMPROOT=`mktemp -d`
+
+rm -rf ${TMPROOT}/testbuild-ycl || true
+mkdir -p ${TMPROOT}/testbuild-ycl
+cd ${TMPROOT}/testbuild-ycl
+
+source $POKYDIR/oe-init-build-env .
+
+cat << EOF >> conf/local.conf
+# just define defaults
+AGL_FEATURES ?= ""
+AGL_EXTRA_IMAGE_FSTYPES ?= ""
+
+# important settings imported from poky-agl.conf
+# we do not import 
+DISTRO_FEATURES_append = " systemd smack"
+DISTRO_FEATURES_BACKFILL_CONSIDERED_append = " sysvinit"
+VIRTUAL-RUNTIME_init_manager = "systemd"
+
+# workaround:
+# ERROR: Nothing PROVIDES 'smack' (but /home/dl9pf/AGL/master-newlayout/external/meta-openembedded/meta-oe/recipes-extended/ostree/ostree_2020.3.bb DEPENDS on or otherwise requires it)
+BBMASK += "meta-openembedded/meta-oe/recipes-extended/ostree/ostree_2020.3.bb"
+
+AGL_FEATURES += "aglcore"
+
+EOF
+
+
+yocto-check-layer \
+	--dependency \
+	$AGLROOT/external/meta-openembedded/meta-oe \
+	-- \
+	$AGLROOT/meta-agl/meta-agl-core
+
+
+[ $? = 0 ] && rm -rf ${TMPROOT}/testbuild-ycl
+
+exit 0
+
+	--dependency \
+	    $AGLROOT/external/meta-openembedded/meta-oe \
+	    $AGLROOT/external/meta-openembedded/meta-python \
+	    $AGLROOT/external/meta-openembedded/meta-networking \
diff --git a/meta-agl-core/scripts/run-yocto-check-layer.sh b/meta-agl-core/scripts/run-yocto-check-layer.sh
new file mode 100755
index 000000000..9082be920
--- /dev/null
+++ b/meta-agl-core/scripts/run-yocto-check-layer.sh
@@ -0,0 +1,48 @@
+#!/bin/bash
+set -x
+
+SCRIPTPATH="$( cd $(dirname $0) >/dev/null 2>&1 ; pwd -P )"
+echo $SCRIPTPATH
+AGLROOT="$SCRIPTPATH/../../.."
+POKYDIR="$AGLROOT/external/poky"
+TMPROOT=`mktemp -d`
+
+rm -rf ${TMPROOT}/testbuild-ycl || true
+mkdir -p ${TMPROOT}/testbuild-ycl
+cd ${TMPROOT}/testbuild-ycl
+
+source $POKYDIR/oe-init-build-env .
+
+cat << EOF >> conf/local.conf
+# just define defaults
+AGL_FEATURES ?= ""
+AGL_EXTRA_IMAGE_FSTYPES ?= ""
+
+# important settings imported from poky-agl.conf
+# we do not import 
+DISTRO_FEATURES_append = " systemd smack"
+DISTRO_FEATURES_BACKFILL_CONSIDERED_append = " sysvinit"
+VIRTUAL-RUNTIME_init_manager = "systemd"
+
+# workaround:
+# ERROR: Nothing PROVIDES 'smack' (but /home/dl9pf/AGL/master-newlayout/external/meta-openembedded/meta-oe/recipes-extended/ostree/ostree_2020.3.bb DEPENDS on or otherwise requires it)
+BBMASK += "meta-openembedded/meta-oe/recipes-extended/ostree/ostree_2020.3.bb"
+
+EOF
+
+
+yocto-check-layer \
+	--dependency \
+	$AGLROOT/external/meta-openembedded/meta-oe \
+	-- \
+	$AGLROOT/meta-agl/meta-agl-core
+
+
+[ $? = 0 ] && rm -rf ${TMPROOT}/testbuild-ycl
+
+exit 0
+
+	--dependency \
+	    $AGLROOT/external/meta-openembedded/meta-oe \
+	    $AGLROOT/external/meta-openembedded/meta-python \
+	    $AGLROOT/external/meta-openembedded/meta-networking \