waltham: Add documentation for waltham

Signed-off-by: Naoko Tanibata <tnaoko@de.adit-jv.com>

15 files changed, 405 insertions, 0 deletions

diff --git a/docs/waltham/0-waltham-overview.md b/docs/waltham/0-waltham-overview.md
new file mode 100644
index 0000000..c756300
--- /dev/null
+++ b/docs/waltham/0-waltham-overview.md
@@ -0,0 +1,54 @@
+---
+title: AGL - Waltham Overview
+author: Naoko Tanibata (Advanced Driver Information Technology)
+date: 2019-12-18
+
+categories: architecture
+tags: architecture, graphics, wayland, weston, waltham
+layout: techdoc
+
+---
+
+**Table of Content**
+
+1. TOC
+{:toc}
+
+## Context
+
+In today's world, the information for drivers is becoming excessive. For example, safety information to let driver notice the obstacles on the road, telematics information about car accident or traffic jam, media information from connected phones etc. In the future world, it is expected that the more displays will be available in the car and show more information. TFT Cluster will have more information other than engine speed and map, Head-Up-Display on windshield will bring driver to new world. 
+However, too much information would make drivers confused. We need more comprehensive Human Machine Interface, which displays the information what the driver needs on appropriate place and time with comprehensive user interface design. To conclude, Graphics sharing between multiple ECUs is necessary. Waltham protocol is developed for this purpose.
+
+## Waltham protocol
+
+[Waltham protocol](https://github.com/waltham/waltham) is a network IPC library of [Wayland](https://wayland.freedesktop.org) developed by Collabora along with ADIT, a joint venture company owned by Robert Bosch Car Multimedia GmbH and DENSO corporation. It supports surface sharing via network, but sharing function itself is not implemented in Waltham.  
+Please refer [Waltham documentation](https://waltham.github.io/waltham/) for more details. 
+
+### Major differences from Wayland to Waltham
+- Waltham uses TCP sockets for reliable communication  
+- Waltham cannot send file descriptors
+- Waltham API is minimal and symmetric between server and client sides
+- Waltham does not provide an event loop implementation
+- The registry implementation is left out of the library, only the interface is provided
+- No multi-threading support for sharing objects between threads
+
+![image](./images/Diffrence_between_Wayland_and_Waltham.jpg)
+
+### Requirements in automotive industry
+In order to use Waltham in automotive industry, the automotive specific requirements must be covered.  
+
+The below shows very high level requirements. You can find the further requirements at [i**Waltham Requirements**](https://confluence.automotivelinux.org/display/UIGRA/Waltham+backend+requirements).
+
+1. Shall be able to support fast/reliable remoting among multiple ECUs
+2. Shall be able to support input handling
+3. Shall be able to share dedicated application
+4. Shall be able to share complete display output depending on additional communication
+
+* Surface sharing is not part of Waltham protocol, Each system needs to implement the most efficient way for surface sharing. 
+On AGL, we implemented [Waltham client and Receiver](1-waltham-client-and-receiver.md) to enable surface sharing along with GStreamer encoder/decoder. It uses UDP for remoting, which is faster than TCP. Input events communicates with Waltham protocol.
+
+### Links
+* [Announcement of Waltham](https://lists.freedesktop.org/archives/wayland-devel/2016-October/031413.html)
+* [Bi-weekly meeting minutes](https://wiki.automotivelinux.org/eg-ui-graphics)
+* [UI and Graphics wiki](https://confluence.automotivelinux.org/display/UIGRA/UI+and+Graphics+Home)
+* [Waltham source codes](https://gerrit.automotivelinux.org/gerrit/gitweb?p=src/weston-ivi-plugins.git;a=tree;h=refs/heads/master;hb=refs/heads/master)
diff --git a/docs/waltham/1-waltham-client-and-receiver.md b/docs/waltham/1-waltham-client-and-receiver.md
new file mode 100644
index 0000000..061caf9
--- /dev/null
+++ b/docs/waltham/1-waltham-client-and-receiver.md
@@ -0,0 +1,117 @@
+---
+title: AGL - waltham client and receiver
+author: Naoko Tanibata (Advanced Driver Information Technology)
+date: 2019-12-18
+
+categories: architecture
+tags: architecture, graphics, wayland, Weston, waltham
+layout: techdoc
+
+---
+
+**Table of Content**
+
+1. TOC
+{:toc}
+
+## waltham-client-server-applications
+
+![image](./images/Waltham_Architecture.jpg)
+
+### waltham-client
+waltham-client uses Waltham IPC library to connect to remote and transmit client buffers using GStreamer framework. It is designed for AGL upon certain Linux base system with wayland backend.
+
+waltham-client is divided into two components:
+
+* waltham-transmitter plugin: 
+    waltham-transmitter plugin provides API to create remote connections and push surfaces over the network and handles both remote output and remote input.
+
+* waltham-renderer: 
+    waltham-renderer is the implementation for sharing surface. It creates a buffer to be transmitted to other domain. Latest implementation uses GStreamer framework. 
+
+### waltham-receiver
+
+waltham-receiver is a sample implementation of receiver app which shall be running at remote side. It is developed based on [waltham server](https://github.com/waltham/waltham/tree/master/tests), using Waltham protocol to obtain and process remote output which is sent by waltham-transmitter.
+This component is designed to be used for evaluating the functionalities of waltham-transmitter plugin.
+
+## How it works
+
+1. Loading and initialization
+
+The system launches Weston, then Weston loads waltham-transmitter plugin. waltham-client connects to receiver app at remote side during initialization.  
+
+![image](./images/01_Load_transmitter.jpg)
+
+2. Establishing connection
+
+At transmitter_create_remote(), waltham-transmitter creates "struct weston_transmitter_remote" which expresses the receiver object at remote side. Therefore this structure is created for each receiver.   
+waltham-transmitter sends wth_display_get_registry() and wth_display_sync message() to the receiver app as same manner as Wayland protocol. So that receiver app sends back the resource list to waltham-transmitter.
+
+![image](./images/02_Establish_connection.jpg)
+
+3. Forwarding surface 
+
+During Weston redraws surface as waltham-transmitter output, waltham-transmitter sends waltham protocol messages to receiver app to notify surface update. wthp_surface_attach(), wthp_surface_damage() and wthp_surface_commit() which correspond to wl_surface_attach(), wl_surface_damage() and wl_surface_commit() message in wayland protocol. 
+wthp_surface_attach() - Send wthp_buffer as a buffer handling. This is not the actual buffer which contains the data to be rendered but the handle of actual buffer. It abstracts the differences of buffer type.
+wthp_surface_damage() - Tell the updated region to receiver app. 
+wthp_surface_commit() - Tell surface gets updated to receiver app.
+
+![image](./images/03_Forward_surface.jpg)
+
+4. Rendering using GStreamer
+
+![image](./images/04_Rendering_using_gstreamer.jpg)
+
+5. Input handling
+
+For handling input events, waltham-transmitter has 2 ways to secure seat. 
+	1. Usse wl_seat as weston has. 
+	2. Create new wl_seat.  
+Second case is applicable in case transmitter side doesn’t have input device but receiver at remote side has. After wl_seat is created, waltham-transmitter sends input events to weston client application when it gets input event from receiver via waltham protocol. 
+The message wthp_send_XXX shows you that input event is forwarded from receiver to transmitter, XXX is filled by input event name.
+
+![image](./images/05_Input_handling.jpg)
+
+6. Retry connection
+
+In case the connection gets disconnected during surface sharing, waltham-transmitter shall re-establish the connection. The object “struct waltham_display” represents the connection between the transmitter and receiver, which structure has the flag used to detect disconnection named “running”. In function “connection_handle_data()”, running is set to "false" in case disconnection detected. This flag is checked at every “transmitter_surface_gather_state". When running is false state, waltham-transmitter starts retry handling sequence. First of all, It release the waltham protocol objects then goes to Establish connection sequence mentioned in "2. Establishing connection".
+
+![image](./images/06_Retry_connection.jpg)
+
+## Waltham in practice
+Here is the example how waltham can be used in hypervisor use case of real project .
+* Weston is used as the wayland compositor.
+* waltham-client is implemented for Weston which acts as a Waltham  virtual display.
+* Application surface is assigned to Waltham virtual display and it's sent to the other ECU/OS. 
+  Buffers of surface are transferred via GStreamer(UDP), since transferring raw pixel data via Waltham(TCP) is not enough fast.
+* Controlling input events (pointer, keyboard, touch) for the surface is handled by Waltham.
+
+![image](./images/Waltham_In_Practice.jpg)
+## How Waltham can be integrated
+Here are possible integration examples of waltham.
+
+### As EGL backend (theoretical possibility)
+
+Similar to Wayland backend for EGL, Waltham client could be backend in compositer. /                                                                                                        
+For good performance, a generic surface sharing mechanism is needed in hypervisor environment.  
+Applications need to adapt to Waltham. Waltham is not designed with this use in mind. This usage is just a theoretical possibility.
+
+![image](./images/Waltham_Integration_Possibility-01.jpg)
+
+### As GStreamer sink (theoretical possibility)
+
+Similar to Wayland sink, Waltham sink GStreamer plugin can be implemented which sends the buffers to receiver on another domain/OS.  
+Waltham sink can utilize frame sync and presentation feedback protocols for video synchronization.  
+For good performance, a generic surface sharing mechanism is needed in hypervisor environment.  
+Waltham is not designed with this use in mind. This usage is just a theoretical possibility.
+
+![image](./images/Waltham_Integration_Possibility-02.jpg)
+
+### As virtual display in compositor
+
+Virtual display plugin can be implemented in compositor. This plugin sends client buffers to waltham-receiver in another domain.  
+No changes to applications.   
+For good performance, a generic surface sharing mechanism is needed in hypervisor environment.  
+This is the intended use in mind during design.
+
+![image](./images/Waltham_Integration_Possibility-03.jpg)
diff --git a/docs/waltham/2-waltham-how-to-build.md b/docs/waltham/2-waltham-how-to-build.md
new file mode 100644
index 0000000..257d435
--- /dev/null
+++ b/docs/waltham/2-waltham-how-to-build.md
@@ -0,0 +1,234 @@
+---
+title: AGL - Waltham How to build
+author: Veeresh Kadasani ((Advanced Driver Information Technology)
+author: Naoko Tanibata (Advanced Driver Information Technology)
+date: 2019-12-18
+
+categories: architecture
+tags: architecture, graphics, wayland, weston. waltham
+layout: techdoc
+
+---
+
+**Table of Content**
+
+1. TOC
+{:toc}
+
+## Information
+Please also refer [Readme](https://gerrit.automotivelinux.org/gerrit/gitweb?p=src/weston-ivi-plugins.git;a=tree;h=refs/heads/master;hb=refs/heads/master) in source repository.
+
+## How to build waltham-transmitter
+1. Prerequisite before building
+   AGL Image is already built. Please refer [here](https://docs.automotivelinux.org/docs/en/master/getting_started/reference/getting-started/image-workflow-build.html)
+
+2. Go to AGL build folder and configure your environment.
+```
+       $ cd $AGL_TOP 
+       $ source meta-agl/scripts/aglsetup.sh
+```
+3. Build waltham-transmitter by using bitbake.
+
+Since waltham-transmitter is not built by default, this step is mandatory.
+
+```
+       $ bitbake waltham-transmitter
+```
+
+4. You can find the built results under $AGL_TOP/build/tmp/work/<board type>/waltham-transmitter/git-r0/image/
+   - usr/bin/waltham-receiver
+   - usr/lib/transmitter.so
+   - usr/lib/waltham-renderer.so
+
+## How to configure weston.ini and GStreamer pipeline
+
+### weston.ini
+
+In order to load waltham-transmitter plugin to weston, add "transmitter.so" to the "modules" key under "[core]" in weston.ini at transmitter side, 
+Then make sure the "shell" is configured as "ivi-shell.so".
+The destination of remoting also needs to be configured in weston.ini.
+Add output name, receiver IP address, port number, output's width and height key under "[transmitter-output]". You can speficy multiple [transmitter-output] with different output-name.
+
+```
+/* Example_weston.ini - single transmitter-output */
+
+	[core]
+	shell=ivi-shell.so
+	modules=systemd-notify.so,ivi-controller.so,transmitter.so
+
+	[ivi-shell]
+	ivi-module=ivi-controller.so
+	ivi-input-module=ivi-input-controller.so
+
+	[transmitter-output]
+	output-name=transmitter_1
+	server-address=192.168.2.52
+	port=34400
+	width=1920
+	height=1080
+
+```
+
+### GStreamer pipeline
+
+You can use GStreamer pipeline as you want. Please describe pipeline configuration in "/etc/xdg/weston/pipeline.cfg".
+Here are 3 examples. 
+
+```
+
+/* General pipeline which does not use any HW encoder */
+	appsrc name=src ! videoconvert ! video/x-raw,format=I420 ! jpegenc ! rtpjpegpay ! udpsink name=sink host=YOUR_RECIEVER_IP port=YOUR_RECIEVER_PORT sync=false async=false
+
+/* pipeline to use Intel's HW encoder */
+	appsrc name=src ! videoconvert ! video/x-raw,format=I420 ! mfxh264enc bitrate=3000000 rate-control=1 ! rtph264pay config-interval=1 ! udpsink name=sink host=YOUR_RECIEVER_IP port=YOUR_RECIEVER_PORT sync=false async=false
+
+/* pipeline to use Rcar's HW encoder */
+	appsrc name=src ! videoconvert ! video/x-raw,format=I420 ! omxh264enc bitrate=3000000 control-rate=2 ! rtph264pay ! udpsink name=sink host=YOUR_RECIEVER_IP port=YOUR_RECIEVER_PORT sync=false async=false
+
+```
+
+## Connection Establishment
+
+1. Connect two boards over ethernet.
+2. Assign IP to both boards. 
+```Example:
+	transmitter IP: 192.168.2.51
+	waltham-receiver IP: 192.168.2.52
+```
+3. Check if the simple ping works
+```
+        $ ping 192.168.2.52 (you can also ping vice versa)
+```
+
+## Example steps to start remoting 
+
+1. Start target boards.
+ 
+transmitter side must have the above 2 files, the modified weston.ini and GStreamer pipeline,cfg.  
+You can confirm that transmitter is loaded properly from weston log as below.
+
+```
+/* Example(/run/platform/display/weston.log) */
+	[12:28:09.127] Loading module '/usr/lib/weston/transmitter.so'
+	[12:28:09.182] Registered plugin API 'transmitter_v1' of size 88
+	[12:28:09.183] Registered plugin API 'transmitter_ivi_v1' of size 16
+	[12:28:09.186] Loading module '/usr/lib/libweston-6/waltham-renderer.so'
+	[12:28:09.255] Transmitter initialized.
+	[12:28:09.255] transmitter_output_attach_head is called
+	[12:28:09.255] Weston head attached successfully to output
+	[12:28:09.255] Output 'transmitter-192.168.2.52:34400-1' enabled with head(s) transmitter-192.168.2.52:34400-1
+	[12:28:09.255] Transmitter weston_seat 0xaaab2209c800
+	[12:28:09.255] Transmitter created pointer=0xaaab220e6b50 for seat 0xaaab2209c800
+	[12:28:09.255] Transmitter created keyboard=0xaaab22038e60 for seat 0xaaab2209c800
+	[12:28:09.255] Transmitter created touch=0xaaab220c7930 for seat 0xaaab2209c800
+```
+
+2. [receiver side] Start receiver application. 
+
+The below example shows the case if you use waltham-receiver as receiver application.
+
+1. Add surface to the display where you want to receive transmitted contents.   
+You can check which output name is required by using the command "LayerManagerControl get scene". 
+
+```
+	$ layer-add-surfaces -s <surface count> -l <layer id> -d <output display name> &
+/* Example */
+	$ layer-add-surfaces -s 1 -l 1 -d HDMI-A-1 &
+```
+2. Start waltham-receiver. You can use debug option with "-v" if you want.
+```	
+ 	$ waltham-receiver -p 34400 -v &
+```
+
+3. Now, receiver side is waiting for the contents coming from transmitter side.
+
+**You must configure and start receiver side first.**
+
+4. [transmitter side] Start an IVI application
+5. Put the surface of IVI application onto transmitter-output by using LayerManagerControl command.
+This surface will be transmitted.
+
+```
+	$ layer-add-surfaces -d <transmitter -output name> -s <surface count on receiver> -l <layer id on receiver>
+/* Example */
+	$ layer-add-surfaces -d transmitter-192.168.2.52:34400-1 -s 1 -l 1 &
+```
+
+5. [transmitter side] make sure that weston.log shows remoting has been started.
+
+```
+/* Example(/run/platform/display/weston.log) */
+	[12:29:33.224] surface ID 1
+	[12:29:40.622] gst-setting are :-->
+	[12:29:40.622] ip = 192.168.2.52
+	[12:29:40.622] port = 34400
+	[12:29:40.622] bitrate = 3000000
+	[12:29:40.622] width = 1080
+	[12:29:40.622] height = 1920
+	[12:29:42.177] Parsing GST pipeline:appsrc name=src ! videoconvert ! video/x-raw,format=I420 ! jpegenc ! rtpjjpegpay ! udpsink name=sink host=192.168.2.52 port=3440 sync=false async=false
+```
+
+## Typical issues & Tips
+
+### help functions
+You can find the help information of LayerManagerControl command by using
+
+``` 
+$ LayerManagerControl help
+```
+
+### waltham-transmitter and waltham-receiver doesn't communicate
+1. Please check ethernet connection. If you assign 192.168.2.51 and 192.168.2.52 for waltham-transmitter and waltham-receiver, you shall ping vice versa.
+
+``` 
+/* At waltham-transmitter side */
+	$ ping 192.168.2.52 
+
+/* At waltham-receiver side */
+	$ ping 192.168.2.51 
+```
+
+2. Make sure that IP address specified in the weston.ini under [transmitter-output] matches the waltham-receiver IP address.  
+3. Make sure that IP address in pipeline.cfg on the transmitter side match the waltham-receiver's IP address.
+
+### surface,layer or output information is unknown.
+You can check them by using the below command.
+
+```	
+	$ LayerManagerControl get scene
+
+/* Example */
+	screen 0 (0x0)
+	---------------------------------------
+	- connector name:       HDMI-A-1
+	- resolution:           x=1024, y=768
+	- layer render order:   100(0x64), 
+	
+	    layer 100 (0x64)
+	    ---------------------------------------
+	    - destination region:   x=296, y=0, w=432, h=768
+	    - source region:        x=0, y=0, w=432, h=768
+	    - opacity:              1
+	    - visibility:           1
+	    - surface render order: 1(0x1), 
+	    - on screen:            0(0x0) 
+	
+	        surface 1 (0x1)
+	        ---------------------------------------
+			- created by pid:       3338
+	        - original size:      x=432, y=768
+	        - destination region: x=0, y=0, w=432, h=768
+    	    - source region:      x=0, y=0, w=432, h=768
+        	- opacity:            1
+			- visibility:         1
+    	    - frame counter:      47
+			- on layer:           100(0x64) 
+
+	screen 1 (0x1)
+	---------------------------------------
+	- connector name:       transmitter-192.168.2.52:34400-1
+	- resolution:           x=1920, y=1080
+	- layer render order: 
+	
+	/* You can know the output name for remoting is "transmitter-192.168.2.52:34400-1" */
+```
diff --git a/docs/waltham/images/01_Load_transmitter.jpg b/docs/waltham/images/01_Load_transmitter.jpg
new file mode 100755
index 0000000..d9e0527
--- /dev/null
+++ b/docs/waltham/images/01_Load_transmitter.jpg
diff --git a/docs/waltham/images/02_Establish_connection.jpg b/docs/waltham/images/02_Establish_connection.jpg
new file mode 100755
index 0000000..2ad5bc8
--- /dev/null
+++ b/docs/waltham/images/02_Establish_connection.jpg
diff --git a/docs/waltham/images/03_Forward_surface.jpg b/docs/waltham/images/03_Forward_surface.jpg
new file mode 100755
index 0000000..116d36a
--- /dev/null
+++ b/docs/waltham/images/03_Forward_surface.jpg
diff --git a/docs/waltham/images/04_Rendering_using_gstreamer.jpg b/docs/waltham/images/04_Rendering_using_gstreamer.jpg
new file mode 100644
index 0000000..97d7858
--- /dev/null
+++ b/docs/waltham/images/04_Rendering_using_gstreamer.jpg
diff --git a/docs/waltham/images/05_Input_handling.jpg b/docs/waltham/images/05_Input_handling.jpg
new file mode 100644
index 0000000..197bbc0
--- /dev/null
+++ b/docs/waltham/images/05_Input_handling.jpg
diff --git a/docs/waltham/images/06_Retry_connection.jpg b/docs/waltham/images/06_Retry_connection.jpg
new file mode 100644
index 0000000..45cb437
--- /dev/null
+++ b/docs/waltham/images/06_Retry_connection.jpg
diff --git a/docs/waltham/images/Diffrence_between_Wayland_and_Waltham.jpg b/docs/waltham/images/Diffrence_between_Wayland_and_Waltham.jpg
new file mode 100644
index 0000000..73cc5e1
--- /dev/null
+++ b/docs/waltham/images/Diffrence_between_Wayland_and_Waltham.jpg
diff --git a/docs/waltham/images/Waltham_Architecture.jpg b/docs/waltham/images/Waltham_Architecture.jpg
new file mode 100644
index 0000000..f1429f4
--- /dev/null
+++ b/docs/waltham/images/Waltham_Architecture.jpg
diff --git a/docs/waltham/images/Waltham_In_Practice.jpg b/docs/waltham/images/Waltham_In_Practice.jpg
new file mode 100644
index 0000000..f413092
--- /dev/null
+++ b/docs/waltham/images/Waltham_In_Practice.jpg
diff --git a/docs/waltham/images/Waltham_Integration_Possibility-01.jpg b/docs/waltham/images/Waltham_Integration_Possibility-01.jpg
new file mode 100644
index 0000000..4c2d08d
--- /dev/null
+++ b/docs/waltham/images/Waltham_Integration_Possibility-01.jpg
diff --git a/docs/waltham/images/Waltham_Integration_Possibility-02.jpg b/docs/waltham/images/Waltham_Integration_Possibility-02.jpg
new file mode 100644
index 0000000..a45cbe8
--- /dev/null
+++ b/docs/waltham/images/Waltham_Integration_Possibility-02.jpg
diff --git a/docs/waltham/images/Waltham_Integration_Possibility-03.jpg b/docs/waltham/images/Waltham_Integration_Possibility-03.jpg
new file mode 100644
index 0000000..9e98282
--- /dev/null
+++ b/docs/waltham/images/Waltham_Integration_Possibility-03.jpg