Introduce docker-compose

This commit introduce two compoments:
lavalab-en.py which generate:
- device files
- conmux config files
- users files
- tokens files
- udev rules for host
- final docker-compose.yml file
Except of docker-compose.yml, all other files are ignored for the moment
and will be handled by a following commit.

docker-compose:
The docker compose permits to building and running easily lava-docker
images.
It permit also to remove some contraints like knowing the
LAVA_SERVER_IP/LAVA_MASTER.

5 files changed, 315 insertions, 16 deletions

diff --git a/README.md b/README.md
index de4c1e8..9d9a92d 100644
--- a/README.md
+++ b/README.md
@@ -1,37 +1,96 @@
 # Linaro's Automated Validation Architecture (LAVA) Docker Container
 Preinstalls and preconfigures the latest LAVA server release.
 
-## Building
-To build an image locally, execute the following from the directory you cloned the repo:
+## Prerequisite
+The package docker-compose is necessary
+
+## Name Conventions
+Each board must be named by their device-type as "device-type-XX" where XX is a number
+Each tty will have a name /dev/boardname (via the udev rules)
+Each conmux config file will be named boardname.cf
+Each slave must be named lab-slave-XX
 
+## Know limitation
+The current lava-docker provide support for only one slave
+
+## Architecture
+The host must have a dedicated LAN. (192.168.66.0/24)
+The host must have IP set to 192.168.66.1 on this LAN.
+A sample dhcpd config file is available in the dhcpd directory
+
+## Generating files
+### boards.yaml
+This file describe how are setuped your boards, and how they are connected and powered.
 ```
-sudo docker build -t lava .
+lab-slave-name:
+	devicename:
+		type: the devicetype of this device
+		pdu:
+			daemon: The hostname running the PDU daemon (always localhost)
+			host: The host name of the PDU as named in lavapdu.conf
+			port: portnumber (The port number of the PDU where the device is connected)
+		uart:
+			type:
+			serial: The serial number in case of FTDI uart
 ```
+Notes:
+uart FTDI only need serial
 
-## Running
-To run the image from a host terminal / command line execute the following:
+Examples: see boards.yaml
+
+### tokens.yaml
+The tokens format have two section, one for user generation, the other for callback tokens
+```
+lava_server_users:
+	- name: LAVA username
+	  token: The token of this use
+	  password: Password the this user (generated if not provided)
+callback_tokens:
+  - filename: The filename for storing the informations below, the name should be unique along other callback tokens
+    username: The LAVA user owning the token below. (This user should be created via lava_server_users:)
+    token: The token for this callback
+    description: The description of this token. This string could be used with LAVA-CI.
+```
+Example: see tokens.yaml
+
+### Generate
+```
+lavalab-gen.py
+```
 
+this scripts will generate all necessary files in the following location:
 ```
-sudo docker run -it -v /dev:/dev -p 69:69/udp -p 80:80 -p 3079:3079 -p 5555:5555 -p 5556:5556 -h <HOSTNAME> --privileged kernelci/lava-docker:latest
+conmux/		All files needed by conmux
+tokens/		This is where the callback tokens will be generated
+users/		This is where the users will be generated
+devices/	All LAVA devices files (note that an extran qemu device is also created for the master)
+udev-rules for host
+docker-compose.yml	Generated from docker-compose.template
 ```
-Where HOSTNAME is the hostname used during the container build process (check the docker build log), as that is the name used for the worker configuration. You can use `lava-docker` as the pre-built container hostname.
 
-## Additional Setup
-In order for TFTP requests to find their way back to the running container, you will need to describe the host IP address to the LAVA master node. You can to create a yaml file on the LAVA master node as described below.
+All thoses files (except for udev-rules) will be handled by docker.
+The udev-rules is for generating the right /dev/xxx TTY names.
 
+You can still hack after generated files.
+
+## Building
+To build an image locally, execute the following from the directory you cloned the repo:
+
+```
+docker-compose build
+```
+
+## Running
 ```
-echo "dispatcher_ip: <master host ip" > /etc/lava-server/dispatcher.d/<lava-master-hostname>.yaml
+docker-compose up
 ```
 
+## Process wrapper
+You can use the lavalab-gen.sh wrapper which will do all the above actions
+
 ## Security
 Note that this container provides defaults which are unsecure. If you plan on deploying this in a production enviroment please consider the following items:
 
   * Changing the default admin password
   * Using HTTPS
   
-Secure CSRF tokens are disabled as the container uses HTTP by default. To use SSL with this container you will need to remove the following lines from your ```/etc/lava-server/settings.conf```
-
-```
-   "CSRF_COOKIE_SECURE": false,
-   "SESSION_COOKIE_SECURE": false,
-```
diff --git a/boards.yaml b/boards.yaml
new file mode 100644
index 0000000..1f3788f
--- /dev/null
+++ b/boards.yaml
@@ -0,0 +1,55 @@
+lab-slave-0:
+        bcm2837-rpi-3-b-01:
+                type: bcm2837-rpi-3-b
+                pdu:
+                        daemon: localhost
+                        host: acme-0
+                        port: 1
+                uart:
+                        type: FTDI
+                        serial: FT9QQZTA
+        am335x-boneblack-01:
+                type: beaglebone-black
+                pdu:
+                        daemon: localhost
+                        host: acme-0
+                        port: 2
+                uart:
+                        type: FTDI
+                        serial: FT9QR1A9
+        meson-gxl-s905x-libretech-cc-01:
+                type: meson-gxl-s905x-libretech-cc
+                pdu:
+                        daemon: localhost
+                        host: acme-0
+                        port: 3
+                uart:
+                        type: FTDI
+                        serial: FT9QQYCN
+        dragonboard-410c-01:
+                type: dragonboard-410c
+                pdu:
+                        daemon: localhost
+                        host: acme-0
+                        port: 4
+                uart:
+                        type: FTDI
+                        serial: FT9R7VDB
+        r8a7796-m3ulcb-01:
+                type: r8a7796-m3ulcb
+                pdu:
+                        daemon: localhost
+                        host: acme-0
+                        port: 5
+                uart:
+                        type: FTDI
+                        serial: AK04WW0Q
+        imx6q-sabrelite-01:
+                type: imx6q-sabrelite
+                pdu:
+                        daemon: localhost
+                        host: acme-0
+                        port: 6
+                uart:
+                        type: FTDI
+                        serial: FTAJMM4V
diff --git a/docker-compose.template b/docker-compose.template
new file mode 100644
index 0000000..dfe82c9
--- /dev/null
+++ b/docker-compose.template
@@ -0,0 +1,34 @@
+version: '2.0'
+services:
+  lava-master:
+    hostname: lava-master
+    restart: always
+    stdin_open: true
+    tty: true
+    build:
+      context: lava-master
+    devices:
+      - "/dev/kvm:/dev/kvm"
+    ports:
+      - "10080:80"
+      - "1022:22"
+      - "5555:5555"
+      - "5556:5556"
+    volumes:
+# boot and /lib/modules are for libguestfs
+      - "/boot:/boot"
+      - "/lib/modules:/lib/modules"
+  lava-slave:
+    hostname: lab-slave-0
+    restart: always
+    build:
+      context: lava-slave
+    devices:
+    environment:
+      LAVA_MASTER: "lava-master"
+    ports:
+      - "69:69/udp"
+      - "80:80"
+      - "55980-56000:55980-56000"
+    links:
+      - "lava-master"
diff --git a/lavalab-gen.py b/lavalab-gen.py
new file mode 100755
index 0000000..a584a1f
--- /dev/null
+++ b/lavalab-gen.py
@@ -0,0 +1,141 @@
+#!/usr/bin/env python
+#
+from __future__ import print_function
+import os, sys, time
+import subprocess
+import argparse
+import yaml
+import string
+import socket
+
+# Defaults
+boards_yaml = "boards.yaml"
+tokens_yaml = "tokens.yaml"
+baud_default = 115200
+    
+template = string.Template("""#
+# auto-generated by lavalab-gen.py for ${board}
+#
+listener ${board}
+application console '${board} console' 'exec sg dialout "cu-loop /dev/${board} ${baud}"'
+command 'hardreset' 'Reboot ${board}' 'pduclient --daemon ${daemon} --host ${host} --port ${port} --command reboot ${delay} '
+command 'b' 'Reboot ${board}' 'pduclient --daemon ${daemon} --host ${host} --port ${port} --command reboot '
+command 'off' 'Power off ${board}' 'pduclient --daemon ${daemon} --host ${host} --port ${port} --command off '
+command 'on' 'Power on ${board}' 'pduclient --daemon ${daemon} --host ${host} --port ${port} --command on '
+""")
+
+#no comment it is volontary
+template_device = string.Template("""{% extends '${devicetype}.jinja2' %}
+{% set connection_command = 'conmux-console ${board}' %}
+{% set hard_reset_command = 'pduclient --daemon localhost --hostname acme-0 --port ${port} --command=reboot' %}
+{% set power_off_command = 'pduclient --daemon localhost --hostname acme-0 --port ${port} --command=off' %}
+{% set power_on_command = 'pduclient --daemon localhost --hostname acme-0 --port ${port} --command=on' %}
+""")
+
+template_udev = string.Template("""#
+SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", ATTRS{serial}=="${serial}", MODE="0664", OWNER="uucp", SYMLINK+="${board}"
+""")
+
+def main(args):
+    fp = open(boards_yaml, "r")
+    labs = yaml.load(fp)
+    fp.close()
+    udev_line =""
+    tdc = open("docker-compose.template", "r")
+    dockcomp = yaml.load(tdc)
+    tdc.close()
+    dc_devices = dockcomp["services"]["lava-slave"]["devices"]
+    if dc_devices is None:
+        dockcomp["services"]["lava-slave"]["devices"] = []
+        dc_devices = dockcomp["services"]["lava-slave"]["devices"]
+
+    for lab_name in labs:
+        lab = labs[lab_name]
+        for board_name in lab:
+            b = lab[board_name]
+            if b.get("disabled", None):
+                continue
+
+            if not b.has_key("uart"):
+                print("WARNING: %s missing uart property" % board_name)
+                continue
+
+            baud = b["uart"].get("baud", baud_default)
+            if b.has_key("pdu"):
+                daemon = b["pdu"]["daemon"]
+                host = b["pdu"]["host"]
+                port = b["pdu"]["port"]
+                devicetype = b["type"]
+                delay_opt = ""
+                line = template.substitute(board=board_name, baud=baud, daemon=daemon, host=host, port=port, delay=delay_opt)
+                device_line = template_device.substitute(board=board_name, port=port, devicetype=devicetype)
+                serial = b["uart"]["serial"]
+                udev_line += template_udev.substitute(board=board_name, serial=serial)
+                dc_devices.append("/dev/%s:/dev/%s" % (board_name, board_name))
+            if b.has_key("fastboot_serial_number"):
+                fserial = b["fastboot_serial_number"]
+                device_line += "{%% set fastboot_serial_number = '%s' %%}" % fserial
+
+            if not os.path.isdir("lava-slave/conmux/"):
+                os.mkdir("lava-slave/conmux/")
+            if not os.path.isdir("lava-master/devices/"):
+                os.mkdir("lava-master/devices/")
+            device_path = "lava-master/devices/%s" % lab_name
+            if not os.path.isdir(device_path):
+                os.mkdir(device_path)
+            fp = open("lava-slave/conmux/%s.cf" % board_name, "w")
+            fp.write(line)
+            fp.close()
+            board_device_file = "%s/%s.jinja2" % (device_path, board_name)
+            fp = open(board_device_file, "w")
+            fp.write(device_line)
+            fp.close()
+        fp = open("lavalab-udev-%s.rules" % lab_name, "w")
+        fp.write(udev_line)
+        fp.close()
+        if not os.path.isdir("lava-master/slaves/"):
+            os.mkdir("lava-master/slaves/")
+        fp = open("lava-master/slaves/%s.yaml" % lab_name, "w")
+        fp.write("dispatcher_ip: 192.168.66.1")
+        fp.close()
+
+    #now proceed with tokens
+    fp = open(tokens_yaml, "r")
+    tokens = yaml.load(fp)
+    fp.close()
+    if not os.path.isdir("lava-master/users/"):
+        os.mkdir("lava-master/users/")
+    if not os.path.isdir("lava-master/tokens/"):
+        os.mkdir("lava-master/tokens/")
+    for section_name in tokens:
+        section = tokens[section_name]
+        if section_name == "lava_server_users":
+            for user in section:
+                username = user["name"]
+                ftok = open("lava-master/users/%s" % username, "w")
+                token = user["token"]
+                ftok.write("TOKEN=" + token + "\n")
+                if user.has_key("password"):
+                    password = user["password"]
+                    ftok.write("PASSWORD=" + password)
+                ftok.close()
+        if section_name == "callback_tokens":
+            for token in section:
+                filename = token["filename"]
+                ftok = open("lava-master/tokens/%s" % filename, "w")
+                username = token["username"]
+                ftok.write("USER=" + username + "\n")
+                vtoken = token["token"]
+                ftok.write("TOKEN=" + vtoken + "\n")
+                description = token["description"]
+                ftok.write("DESCRIPTION=" + description)
+                ftok.close()
+    with file('docker-compose.yml', 'w') as f:
+        yaml.dump(dockcomp, f)
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--header", help="use this file as header for output file")
+    args = parser.parse_args()
+    main(args)
+
diff --git a/tokens.yaml b/tokens.yaml
new file mode 100644
index 0000000..c6a430f
--- /dev/null
+++ b/tokens.yaml
@@ -0,0 +1,10 @@
+lava_server_users:
+  - name: example
+    token: longrandomtoken
+    password: examplepassword
+
+callback_tokens:
+  - filename: example
+    username: example
+    token: longrandomtokencallback
+    description: example-callback