aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorMarius Vlad <marius.vlad@collabora.com>2020-10-19 23:58:48 +0300
committerMarius Vlad <marius.vlad@collabora.com>2020-10-20 00:20:31 +0300
commitc9d23a045e6cb46c023c15f8189ef3ee8b1ddf20 (patch)
treedfaab199379ca17c25e7f4f630ed7995435e0085
parent0e79ab59165bf925f1288476dad66109aa01b3fa (diff)
This adds the documentations bits, most, if not all, extracted from the original wayland-ivi-plugins repository. It describes the waltham-transmiter-plugin, and the receiver side, how to configure and run them. It also removes any ivi-shell and with it, any menton about LayoutManagerControls. Bug-AGL: SPEC-3601 Signed-off-by: Marius Vlad <marius.vlad@collabora.com> Change-Id: I2f900f7eee34c431cc8d601131c5925c31280425
-rw-r--r--docs/0-waltham-overview.md76
-rw-r--r--docs/1-waltham-client-and-receiver.md156
-rw-r--r--docs/2-waltham-how-to-build.md108
-rwxr-xr-xdocs/images/01_Load_transmitter.jpgbin0 -> 824236 bytes
-rwxr-xr-xdocs/images/02_Establish_connection.jpgbin0 -> 1054236 bytes
-rw-r--r--docs/images/03_Forward_surface.jpgbin0 -> 47512 bytes
-rw-r--r--docs/images/04_Rendering_using_gstreamer.jpgbin0 -> 556653 bytes
-rw-r--r--docs/images/05_Input_handling.jpgbin0 -> 728067 bytes
-rw-r--r--docs/images/06_Retry_connection.jpgbin0 -> 817541 bytes
-rw-r--r--docs/images/Diffrence_between_Wayland_and_Waltham.jpgbin0 -> 357121 bytes
-rw-r--r--docs/images/Waltham_Architecture.jpgbin0 -> 414419 bytes
-rw-r--r--docs/images/Waltham_In_Practice.jpgbin0 -> 597581 bytes
-rw-r--r--docs/images/Waltham_Integration_Possibility-01.jpgbin0 -> 292194 bytes
-rw-r--r--docs/images/Waltham_Integration_Possibility-02.jpgbin0 -> 313179 bytes
-rw-r--r--docs/images/Waltham_Integration_Possibility-03.jpgbin0 -> 388127 bytes
-rw-r--r--docs/waltham-book.yml16
16 files changed, 356 insertions, 0 deletions
diff --git a/docs/0-waltham-overview.md b/docs/0-waltham-overview.md
new file mode 100644
index 0000000..78f2596
--- /dev/null
+++ b/docs/0-waltham-overview.md
@@ -0,0 +1,76 @@
+**Table of Content**
+
+1. TOC
+{:toc}
+
+## Context
+
+In today's world, the information for drivers is becoming excessive. For
+example, safety information to let the driver notice the obstacles on the road,
+telematics information about car accident or traffic jam, media information
+from connected phones etc. In the future, it is expected that the
+more displays will be available in the car and show more information.
+
+TFT (Thin-Film transistor) Cluster will have more information, other than
+engine speed and a map, with a Head-Up-Display projected onto the windshield
+will bring driver to new world. With the constant bombardment of information
+drivers will have a hard time making sure their eyes on the road, avoiding
+needless distractions.
+
+As we need more comprehensive Human Machine Interfaces, which displays the
+information that the driver needs on appropriate device and on time with
+a comprehensive user interface, a screen-sharing, remoting mechanism between
+multiple ECUs (Electronic Control Unit) will be necessary. For that, a protocol
+working over the network has been devised that could solve this problem.
+
+## Waltham protocol
+
+[Waltham protocol](https://github.com/waltham/waltham) is a IPC library similar
+to [Wayland](https://wayland.freedesktop.org), developed with networking in
+mind. It operates over TCP sockets, while the wayland protocol only
+works locally over a UNIX socket. It retains wayland-esque paradigm, making use
+of XMLs to describe the protocol, and it follows an object-oriented design with
+an asynchronous architecture.
+
+It was developed by Collabora Ltd., along with ADIT, a joint venture company
+owned by Robert Bosch Car Multimedia GmbH and DENSO corporation. It supports
+surface sharing via network, but sharing function itself is not implemented in
+Waltham. Please refer [Waltham
+documentation](https://waltham.github.io/waltham/) for more details.
+
+### Major differences from Wayland to Waltham
+- Waltham uses TCP sockets for reliable communication
+- Waltham cannot send file descriptors
+- Waltham API is minimal and symmetric between server and client sides
+- Waltham does not provide an event loop implementation
+- The registry implementation is left out of the library, only the interface is
+ provided
+- No multi-threading support for sharing objects between threads
+
+![image](./images/Diffrence_between_Wayland_and_Waltham.jpg)
+
+### Requirements in automotive industry
+In order to use Waltham in automotive industry, the automotive specific
+requirements must be covered.
+
+The below shows very high level requirements. You can find the further
+requirements at [**Waltham Requirements**](https://confluence.automotivelinux.org/display/UIGRA/Waltham+backend+requirements).
+
+1. Shall be able to support fast/reliable remoting among multiple ECUs
+2. Shall be able to support input handling
+3. Shall be able to share dedicated application
+4. Shall be able to share complete display output depending on additional
+ communication
+
+* Surface sharing is not part of Waltham protocol, each system needs to
+implement the most efficient way for surface sharing. On AGL, we implemented
+[Waltham client and Receiver](1-waltham-client-and-receiver.md) to enable
+surface sharing along with GStreamer encoder/decoder. It uses UDP for remoting,
+which is faster than TCP. Input events communicates with Waltham
+protocol.
+
+### Links
+* [Announcement of Waltham](https://lists.freedesktop.org/archives/wayland-devel/2016-October/031413.html)
+* [Bi-weekly meeting minutes](https://wiki.automotivelinux.org/eg-ui-graphics)
+* [UI and Graphics wiki](https://confluence.automotivelinux.org/display/UIGRA/UI+and+Graphics+Home)
+* [Waltham source codes](https://gerrit.automotivelinux.org/gerrit/gitweb?p=src/weston-ivi-plugins.git;a=tree;h=refs/heads/master;hb=refs/heads/master)
diff --git a/docs/1-waltham-client-and-receiver.md b/docs/1-waltham-client-and-receiver.md
new file mode 100644
index 0000000..2396972
--- /dev/null
+++ b/docs/1-waltham-client-and-receiver.md
@@ -0,0 +1,156 @@
+**Table of Content**
+
+1. TOC
+{:toc}
+
+## waltham-client-server-applications
+
+![image](./images/Waltham_Architecture.jpg)
+
+### waltham-transmitter-plugin
+
+*waltham-transmitter-plugin* provides an API to create remote connections and
+push surfaces over the network and handles remote input. It is loaded
+automatically at start-up, by the compositor. The (remote) outputs are streamed
+to the remote side by using the remoting plug-in.
+
+### waltham-receiver
+
+waltham-receiver is a sample implementation of the receiver app which should be
+running at the remote side. It is developed based on [waltham
+server](https://github.com/waltham/waltham/tree/master/tests), using Waltham
+protocol to obtain and process remote input -- handled by
+waltham-transmitter-plugin, with the remote output handled by the remoting
+plugin. This component is designed to be used for evaluating the functionality
+of waltham-transmitter plugin.
+
+## How it works
+
+### 1. Loading and initialization
+
+As the compositor is starting up, it loads the waltham-transmitter plugin.
+
+![image](./images/01_Load_transmitter.jpg)
+
+### 2. Establishing connection
+
+At `transmitter_create_remote()`, waltham-transmitter creates
+`weston_transmitter_remote` object which expresses the receiver object at
+remote side, and it is created for each receiver.
+
+*waltham-transmitter* uses `wth_display_get_registry()` and `wth_display_sync()`
+for the receiver applications in the same manner as the Wayland protocol
+would retrieve the interfaces. Then, the receiver applications sends back
+the resource list to the waltham-transmitter.
+
+![image](./images/02_Establish_connection.jpg)
+
+### 3. Forwarding surface
+
+A fake mechanism of sending the buffer attachments was previously in place. The
+following protocol messages to the receiver app were added to notify about
+surface updates.
+
+`wthp_surface_attach()`, `wthp_surface_damage()` and `wthp_surface_commit()`
+correspond to `wl_surface_attach()`, `wl_surface_damage()` and
+`wl_surface_commit()` message in wayland protocol.
+
+- `wthp_surface_attach()` - Send wthp_buffer as a buffer handling. This is not
+the actual buffer which contains the data to be rendered but the handle of an
+actual buffer. It abstracts the differences of buffer type.
+- `wthp_surface_damage()` - Tell the updated region to receiver app.
+- `wthp_surface_commit()` - Tell surface gets updated to receiver app.
+
+These are no longer in place, as the remoting output is being handled entirely
+by the remoting plugin.
+
+![image](./images/03_Forward_surface.jpg)
+
+### 4. Rendering using GStreamer
+
+![image](./images/04_Rendering_using_gstreamer.jpg)
+
+### 5. Input handling
+
+For handling input events, waltham-transmitter has 2 ways to secure seat.
+
+1. Use `wl_seat` as weston has.
+2. Create a new `wl_seat`.
+
+Second case is applicable in case the transmitter side does not have an input
+device but the receiver at remote side has one. After `wl_seat` is created,
+waltham-transmitter sends input events to the client application when
+it gets an input event from the receiver via the Waltham protocol.
+
+The message wthp_send_XXX shows you that input event is forwarded from the
+receiver to the transmitter, XXX is filled by the input event name.
+
+![image](./images/05_Input_handling.jpg)
+
+### 6. Retry connection
+
+In case the connection gets disconnected during surface sharing,
+waltham-transmitter shall re-establish the connection. The `waltham_display`
+objects represents the connection between the transmitter and the receiver,
+and contains a flag that can be used to detect a disconnect.
+
+That happens in `connection_handle_data()` in case a disconnect is detected.
+This flag is checked at every call of `transmitter_surface_gather_state`. When
+running is in false state, waltham-transmitter starts to retry the handling
+sequence.
+
+It release the waltham protocol objects then it goes to establish a connection
+sequence mentioned in 2. Establishing connection.
+
+![image](./images/06_Retry_connection.jpg)
+
+## Waltham in practice
+
+Here is the example how waltham can be used in hypervisor use case of real
+project.
+
+* Weston is used as the wayland compositor.
+* waltham-client is implemented for Weston which acts as a Waltham virtual
+ display.
+* Application surface is assigned to Waltham virtual display and it's sent to
+ the other ECU/OS. Buffers of surface are transferred via GStreamer(UDP),
+ since transferring raw pixel data via Waltham(TCP) is not fast enough.
+* Controlling input events (pointer, keyboard, touch) for the surface is
+ handled by Waltham.
+
+![image](./images/Waltham_In_Practice.jpg)
+
+## How Waltham can be integrated
+
+Some possible integration examples of waltham follow.
+
+### As an EGL backend (theoretical possibility)
+
+Similarly to Wayland back-end for EGL, Waltham client could be a back-end in
+the compositor. For better performance, a generic surface sharing mechanism is
+needed in a hypervisor environment. Applications need to adapt to Waltham.
+As waltham is not designed with this use in mind this usage is just a
+theoretical possibility.
+
+![image](./images/Waltham_Integration_Possibility-01.jpg)
+
+### As a GStreamer sink (theoretical possibility)
+
+Similarly to Wayland sink, a Waltham sink GStreamer plugin can be implemented
+which sends the buffers to a receiver on another domain/OS. Waltham sink can
+utilize frame synchronization and a presentation feedback protocols for video
+synchronization. For better performance, a generic surface sharing mechanism
+is needed in a hypervisor environment. As Waltham is not designed with this
+use in mind this usage is just a theoretical possibility.
+
+![image](./images/Waltham_Integration_Possibility-02.jpg)
+
+### As a virtual display in compositor
+
+Virtual display plugin can be implemented in the compositor. This plugin sends
+client buffers to waltham-receiver in another domain. No changes to
+applications. For good performance, a generic surface sharing mechanism is
+needed in hypervisor environment. This is the intended use in mind during
+design.
+
+![image](./images/Waltham_Integration_Possibility-03.jpg)
diff --git a/docs/2-waltham-how-to-build.md b/docs/2-waltham-how-to-build.md
new file mode 100644
index 0000000..350e728
--- /dev/null
+++ b/docs/2-waltham-how-to-build.md
@@ -0,0 +1,108 @@
+**Table of Content**
+
+1. TOC
+{:toc}
+
+## Information
+
+## How to build waltham-transmitter-plugin
+
+1. Go to AGL build folder and configure your environment. Add
+'agl-weston-waltham-remoting' as distro feature.
+
+```
+ $ cd $AGL_TOP
+ $ source meta-agl/scripts/aglsetup.sh -f -b ${MACHINE_TYPE} [additional_distro_features] agl-weston-waltham-remoting
+```
+
+2. Build agl-demo-platform by using bitbake.
+
+3. Clone/download the waltham-transmitter-plugin on your own and build
+locally using the build system provided. See README.md file for more
+information.
+
+## How to configure weston.ini and GStreamer pipeline
+
+### weston.ini
+
+Add output name, host IP address, port number, and mode key under
+'[transmitter-output]' section.
+You can speficy multiple [transmitter-output] with different output-name.
+
+```
+/* Example_weston.ini - single transmitter-output */
+ [transmitter-output]
+ name=transmitter-1
+ mode=640x720@30
+ host=192.168.20.99
+ port=5005
+ agl-shell-app-id=<APP_ID>
+```
+
+### GStreamer pipeline
+
+You can use GStreamer pipeline as you want. Here are some examples.
+
+```
+
+/* General pipeline which does not use any HW encoder */
+ appsrc name=src ! videoconvert ! video/x-raw,format=I420 ! jpegenc ! \
+ rtpjpegpay ! udpsink name=sink host=YOUR_RECIEVER_IP \
+ port=YOUR_RECIEVER_PORT sync=false async=false
+
+/* pipeline to use Intel's HW encoder */
+ appsrc name=src ! videoconvert ! video/x-raw,format=I420 ! \
+ mfxh264enc bitrate=3000000 rate-control=1 ! rtph264pay config-interval=1 ! \
+ udpsink name=sink host=YOUR_RECIEVER_IP port=YOUR_RECIEVER_PORT \
+ sync=false async=false
+
+/* pipeline to use Rcar's HW encoder */
+ appsrc name=src ! videoconvert ! video/x-raw,format=I420 ! \
+ omxh264enc bitrate=3000000 control-rate=2 ! rtph264pay ! \
+ udpsink name=sink host=YOUR_RECIEVER_IP port=YOUR_RECIEVER_PORT \
+ sync=false async=false
+
+```
+
+## Connection Establishment
+
+1. Connect two boards over ethernet.
+2. Assign IP to both boards.
+```Example:
+ transmitter IP: 192.168.2.51
+ waltham-receiver IP: 192.168.2.52
+```
+3. Check if the simple ping works
+```
+ $ ping 192.168.2.52 (you can also ping vice versa)
+```
+
+## Typical issues & Tips
+
+### help functions
+You can find the help information of LayerManagerControl command by using
+
+```
+$ LayerManagerControl help
+```
+
+### waltham-transmitter and waltham-receiver doesn't not communicate
+
+1. Please check ethernet connection. If you assign 192.168.2.51 and
+ 192.168.2.52 for waltham-transmitter and waltham-receiver, you shall ping
+ vice versa.
+
+```
+/* At waltham-transmitter side */
+ $ ping 192.168.2.52
+
+/* At waltham-receiver side */
+ $ ping 192.168.2.51
+```
+
+2. Make sure that IP address specified in the weston.ini under
+ [transmitter-output] matches the waltham-receiver IP address.
+
+3. Make sure that IP address in pipeline.cfg on the transmitter side match the
+ waltham-receiver's IP address.
+
diff --git a/docs/images/01_Load_transmitter.jpg b/docs/images/01_Load_transmitter.jpg
new file mode 100755
index 0000000..d9e0527
--- /dev/null
+++ b/docs/images/01_Load_transmitter.jpg
Binary files differ
diff --git a/docs/images/02_Establish_connection.jpg b/docs/images/02_Establish_connection.jpg
new file mode 100755
index 0000000..2ad5bc8
--- /dev/null
+++ b/docs/images/02_Establish_connection.jpg
Binary files differ
diff --git a/docs/images/03_Forward_surface.jpg b/docs/images/03_Forward_surface.jpg
new file mode 100644
index 0000000..38c2ef2
--- /dev/null
+++ b/docs/images/03_Forward_surface.jpg
Binary files differ
diff --git a/docs/images/04_Rendering_using_gstreamer.jpg b/docs/images/04_Rendering_using_gstreamer.jpg
new file mode 100644
index 0000000..97d7858
--- /dev/null
+++ b/docs/images/04_Rendering_using_gstreamer.jpg
Binary files differ
diff --git a/docs/images/05_Input_handling.jpg b/docs/images/05_Input_handling.jpg
new file mode 100644
index 0000000..197bbc0
--- /dev/null
+++ b/docs/images/05_Input_handling.jpg
Binary files differ
diff --git a/docs/images/06_Retry_connection.jpg b/docs/images/06_Retry_connection.jpg
new file mode 100644
index 0000000..45cb437
--- /dev/null
+++ b/docs/images/06_Retry_connection.jpg
Binary files differ
diff --git a/docs/images/Diffrence_between_Wayland_and_Waltham.jpg b/docs/images/Diffrence_between_Wayland_and_Waltham.jpg
new file mode 100644
index 0000000..0383654
--- /dev/null
+++ b/docs/images/Diffrence_between_Wayland_and_Waltham.jpg
Binary files differ
diff --git a/docs/images/Waltham_Architecture.jpg b/docs/images/Waltham_Architecture.jpg
new file mode 100644
index 0000000..f1429f4
--- /dev/null
+++ b/docs/images/Waltham_Architecture.jpg
Binary files differ
diff --git a/docs/images/Waltham_In_Practice.jpg b/docs/images/Waltham_In_Practice.jpg
new file mode 100644
index 0000000..f413092
--- /dev/null
+++ b/docs/images/Waltham_In_Practice.jpg
Binary files differ
diff --git a/docs/images/Waltham_Integration_Possibility-01.jpg b/docs/images/Waltham_Integration_Possibility-01.jpg
new file mode 100644
index 0000000..4c2d08d
--- /dev/null
+++ b/docs/images/Waltham_Integration_Possibility-01.jpg
Binary files differ
diff --git a/docs/images/Waltham_Integration_Possibility-02.jpg b/docs/images/Waltham_Integration_Possibility-02.jpg
new file mode 100644
index 0000000..a45cbe8
--- /dev/null
+++ b/docs/images/Waltham_Integration_Possibility-02.jpg
Binary files differ
diff --git a/docs/images/Waltham_Integration_Possibility-03.jpg b/docs/images/Waltham_Integration_Possibility-03.jpg
new file mode 100644
index 0000000..9e98282
--- /dev/null
+++ b/docs/images/Waltham_Integration_Possibility-03.jpg
Binary files differ
diff --git a/docs/waltham-book.yml b/docs/waltham-book.yml
new file mode 100644
index 0000000..313ebf7
--- /dev/null
+++ b/docs/waltham-book.yml
@@ -0,0 +1,16 @@
+type: books
+books:
+-
+ id: waltham-docs-source
+ title: "AGL waltham"
+ description: AGL waltham Description
+ keywords:
+ author: "Advanced Driver Information Technology"
+ version: master
+ chapters:
+ - url: 0-waltham-overview.md
+ name: Waltham Overview
+ - url: 1-waltham-client-and-receiver.md
+ name: Waltham Client and Receiver
+ - url: 2-waltham-how-to-build.md
+ name: Waltham How to Build