diff options
16 files changed, 505 insertions, 0 deletions
diff --git a/docs/0-waltham-overview.md b/docs/0-waltham-overview.md new file mode 100644 index 0000000..78f2596 --- /dev/null +++ b/docs/0-waltham-overview.md @@ -0,0 +1,76 @@ +**Table of Content** + +1. TOC +{:toc} + +## Context + +In today's world, the information for drivers is becoming excessive. For +example, safety information to let the driver notice the obstacles on the road, +telematics information about car accident or traffic jam, media information +from connected phones etc. In the future, it is expected that the +more displays will be available in the car and show more information. + +TFT (Thin-Film transistor) Cluster will have more information, other than +engine speed and a map, with a Head-Up-Display projected onto the windshield +will bring driver to new world. With the constant bombardment of information +drivers will have a hard time making sure their eyes on the road, avoiding +needless distractions. + +As we need more comprehensive Human Machine Interfaces, which displays the +information that the driver needs on appropriate device and on time with +a comprehensive user interface, a screen-sharing, remoting mechanism between +multiple ECUs (Electronic Control Unit) will be necessary. For that, a protocol +working over the network has been devised that could solve this problem. + +## Waltham protocol + +[Waltham protocol](https://github.com/waltham/waltham) is a IPC library similar +to [Wayland](https://wayland.freedesktop.org), developed with networking in +mind. It operates over TCP sockets, while the wayland protocol only +works locally over a UNIX socket. It retains wayland-esque paradigm, making use +of XMLs to describe the protocol, and it follows an object-oriented design with +an asynchronous architecture. + +It was developed by Collabora Ltd., along with ADIT, a joint venture company +owned by Robert Bosch Car Multimedia GmbH and DENSO corporation. It supports +surface sharing via network, but sharing function itself is not implemented in +Waltham. Please refer [Waltham +documentation](https://waltham.github.io/waltham/) for more details. + +### Major differences from Wayland to Waltham +- Waltham uses TCP sockets for reliable communication +- Waltham cannot send file descriptors +- Waltham API is minimal and symmetric between server and client sides +- Waltham does not provide an event loop implementation +- The registry implementation is left out of the library, only the interface is + provided +- No multi-threading support for sharing objects between threads + + + +### Requirements in automotive industry +In order to use Waltham in automotive industry, the automotive specific +requirements must be covered. + +The below shows very high level requirements. You can find the further +requirements at [**Waltham Requirements**](https://confluence.automotivelinux.org/display/UIGRA/Waltham+backend+requirements). + +1. Shall be able to support fast/reliable remoting among multiple ECUs +2. Shall be able to support input handling +3. Shall be able to share dedicated application +4. Shall be able to share complete display output depending on additional + communication + +* Surface sharing is not part of Waltham protocol, each system needs to +implement the most efficient way for surface sharing. On AGL, we implemented +[Waltham client and Receiver](1-waltham-client-and-receiver.md) to enable +surface sharing along with GStreamer encoder/decoder. It uses UDP for remoting, +which is faster than TCP. Input events communicates with Waltham +protocol. + +### Links +* [Announcement of Waltham](https://lists.freedesktop.org/archives/wayland-devel/2016-October/031413.html) +* [Bi-weekly meeting minutes](https://wiki.automotivelinux.org/eg-ui-graphics) +* [UI and Graphics wiki](https://confluence.automotivelinux.org/display/UIGRA/UI+and+Graphics+Home) +* [Waltham source codes](https://gerrit.automotivelinux.org/gerrit/gitweb?p=src/weston-ivi-plugins.git;a=tree;h=refs/heads/master;hb=refs/heads/master) diff --git a/docs/1-waltham-client-and-receiver.md b/docs/1-waltham-client-and-receiver.md new file mode 100644 index 0000000..3b00c11 --- /dev/null +++ b/docs/1-waltham-client-and-receiver.md @@ -0,0 +1,157 @@ +**Table of Content** + +1. TOC +{:toc} + +## waltham-client-server-applications + + + +### waltham-client +waltham-client uses Waltham IPC library to connect to remote host and transmit +client buffers using GStreamer framework. + +waltham-client is divided into two components: + +* waltham-transmitter plugin: waltham-transmitter plugin provides API to create + remote connections and push surfaces over the network and handles both remote + output and remote input. Loaded automatically at start-up, by the compositor. + +* waltham-renderer: waltham-renderer is the implementation for sharing surface. + It creates a buffer to be transmitted to other domain. Latest implementation + uses GStreamer framework. + +### waltham-receiver + +waltham-receiver is a sample implementation of the receiver app which shall be +running at the remote side. It is developed based on [waltham +server](https://github.com/waltham/waltham/tree/master/tests), using Waltham +protocol to obtain and process remote output, which is sent by +waltham-transmitter. This component is designed to be used for evaluating the +functionality of waltham-transmitter plugin. + +## How it works + +### 1. Loading and initialization + +As the compositor is starting up, it loads the waltham-transmitter plugin. +waltham-client connects to the receiver application at the remote side during +initialization. + + + +### 2. Establishing connection + +At `transmitter_create_remote()`, waltham-transmitter creates +`weston_transmitter_remote` object which expresses the receiver object at +remote side, and it is created for each receiver. + +*waltham-transmitter* uses `wth_display_get_registry()` and `wth_display_sync()` +for the receiver applications in the same manner as the Wayland protocol +would retrieve the interfaces. Then, the receiver applications sends back +the resource list to the waltham-transmitter. + + + +### 3. Forwarding surface + +While the compositor redraws the surface, waltham-transmitter sends waltham +protocol messages to the receiver app to notify about those surface updates. +`wthp_surface_attach()`, `wthp_surface_damage()` and `wthp_surface_commit()` +correspond to `wl_surface_attach()`, `wl_surface_damage()` and +`wl_surface_commit()` message in wayland protocol. + +- `wthp_surface_attach()` - Send wthp_buffer as a buffer handling. This is not +the actual buffer which contains the data to be rendered but the handle of an +actual buffer. It abstracts the differences of buffer type. +- `wthp_surface_damage()` - Tell the updated region to receiver app. +- `wthp_surface_commit()` - Tell surface gets updated to receiver app. + + + +### 4. Rendering using GStreamer + + + +### 5. Input handling + +For handling input events, waltham-transmitter has 2 ways to secure seat. + +1. Use `wl_seat` as weston has. +2. Create a new `wl_seat`. + +Second case is applicable in case the transmitter side does not have an input +device but the receiver at remote side has one. After `wl_seat` is created, +waltham-transmitter sends input events to the client application when +it gets an input event from the receiver via the Waltham protocol. + +The message wthp_send_XXX shows you that input event is forwarded from the +receiver to the transmitter, XXX is filled by the input event name. + + + +### 6. Retry connection + +In case the connection gets disconnected during surface sharing, +waltham-transmitter shall re-establish the connection. The `waltham_display` +objects represents the connection between the transmitter and the receiver, +and contains a flag that can be used to detect a disconnect. + +That happens in `connection_handle_data()` in case a disconnect +is detected. This flag is checked at every call of +`transmitter_surface_gather_state`. When running is in false state, +waltham-transmitter starts to retry the handling sequence + +It release the waltham protocol objects then it goes to establish a connection +sequence mentioned in 2. Establishing connection. + + + +## Waltham in practice +Here is the example how waltham can be used in hypervisor use case of real +project. + +* Weston is used as the wayland compositor. +* waltham-client is implemented for Weston which acts as a Waltham virtual + display. +* Application surface is assigned to Waltham virtual display and it's sent to + the other ECU/OS. Buffers of surface are transferred via GStreamer(UDP), + since transferring raw pixel data via Waltham(TCP) is not fast enough. +* Controlling input events (pointer, keyboard, touch) for the surface is + handled by Waltham. + + +## How Waltham can be integrated + +Some possible integration examples of waltham follow. + +### As an EGL backend (theoretical possibility) + +Similarly to Wayland back-end for EGL, Waltham client could be a back-end in +the compositor. For better performance, a generic surface sharing mechanism is +needed in a hypervisor environment. Applications need to adapt to Waltham. +As waltham is not designed with this use in mind this usage is just a +theoretical possibility. + + + +### As a GStreamer sink (theoretical possibility) + +Similarly to Wayland sink, a Waltham sink GStreamer plugin can be implemented +which sends the buffers to a receiver on another domain/OS. Waltham sink can +utilize frame synchronization and a presentation feedback protocols for video +synchronization. For better performance, a generic surface sharing mechanism +is needed in a hypervisor environment. As Waltham is not designed with this +use in mind this usage is just a theoretical possibility. + + + +### As a virtual display in compositor + +Virtual display plugin can be implemented in the compositor. This plugin sends +client buffers to waltham-receiver in another domain. No changes to +applications. For good performance, a generic surface sharing mechanism is +needed in hypervisor environment. This is the intended use in mind during +design. + + diff --git a/docs/2-waltham-how-to-build.md b/docs/2-waltham-how-to-build.md new file mode 100644 index 0000000..189bd2c --- /dev/null +++ b/docs/2-waltham-how-to-build.md @@ -0,0 +1,256 @@ +**Table of Content** + +1. TOC +{:toc} + +## Important notice + +Do note that even though waltham-transmitter can be built, AGL will no longer +use weston as its default compositor, and although it relies on it (more +exactly it uses its underlying library -- libweston), the following +build instructions are most likely just a guide, and not a verified +walk-through that reflects the current status. They all assume that you'll use +weston and with it ivi-shell, which is not longer the case in AGL. + +Integration with the newer compositor is at this point in a WIP state. + +## Information +Please also refer [Readme](https://gerrit.automotivelinux.org/gerrit/gitweb?p=src/weston-ivi-plugins.git;a=tree;h=refs/heads/master;hb=refs/heads/master) in source repository. + +## How to build waltham-transmitter +1. Prerequisite before building + AGL Image is already built. Please refer [here](https://docs.automotivelinux.org/docs/en/master/getting_started/reference/getting-started/image-workflow-build.html) + +2. Go to AGL build folder and configure your environment. +``` + $ cd $AGL_TOP + $ source meta-agl/scripts/aglsetup.sh +``` +3. Build waltham-transmitter by using bitbake. + +Since waltham-transmitter is not built by default, this step is mandatory. + +``` + $ bitbake waltham-transmitter +``` + +4. You can find the built results under $AGL_TOP/build/tmp/work/<board type>/waltham-transmitter/git-r0/image/ + - usr/bin/waltham-receiver + - usr/lib/transmitter.so + - usr/lib/waltham-renderer.so + +## How to configure weston.ini and GStreamer pipeline + +### weston.ini + +In order to load waltham-transmitter plugin to weston, add "transmitter.so" to +the "modules" key under "[core]" in weston.ini at transmitter side, +then make sure the "shell" is configured as "ivi-shell.so". + +The destination of remoting also needs to be configured in weston.ini. +Add output name, receiver IP address, port number, output's width and height +key under "[transmitter-output]". You can speficy multiple [transmitter-output] +with different output-name. + +``` +/* Example_weston.ini - single transmitter-output */ + + [core] + shell=ivi-shell.so + modules=systemd-notify.so,ivi-controller.so,transmitter.so + + [ivi-shell] + ivi-module=ivi-controller.so + ivi-input-module=ivi-input-controller.so + + [transmitter-output] + output-name=transmitter_1 + server-address=192.168.2.52 + port=34400 + width=1920 + height=1080 + +``` + +### GStreamer pipeline + +You can use GStreamer pipeline as you want. Please describe pipeline +configuration in "/etc/xdg/weston/pipeline.cfg". +Here are some examples. + +``` + +/* General pipeline which does not use any HW encoder */ + appsrc name=src ! videoconvert ! video/x-raw,format=I420 ! jpegenc ! \ + rtpjpegpay ! udpsink name=sink host=YOUR_RECIEVER_IP \ + port=YOUR_RECIEVER_PORT sync=false async=false + +/* pipeline to use Intel's HW encoder */ + appsrc name=src ! videoconvert ! video/x-raw,format=I420 ! \ + mfxh264enc bitrate=3000000 rate-control=1 ! rtph264pay config-interval=1 ! \ + udpsink name=sink host=YOUR_RECIEVER_IP port=YOUR_RECIEVER_PORT \ + sync=false async=false + +/* pipeline to use Rcar's HW encoder */ + appsrc name=src ! videoconvert ! video/x-raw,format=I420 ! \ + omxh264enc bitrate=3000000 control-rate=2 ! rtph264pay ! \ + udpsink name=sink host=YOUR_RECIEVER_IP port=YOUR_RECIEVER_PORT \ + sync=false async=false + +``` + +## Connection Establishment + +1. Connect two boards over ethernet. +2. Assign IP to both boards. +```Example: + transmitter IP: 192.168.2.51 + waltham-receiver IP: 192.168.2.52 +``` +3. Check if the simple ping works +``` + $ ping 192.168.2.52 (you can also ping vice versa) +``` + +## Example steps to start remoting + +1. Start target boards. + +transmitter side must have the above 2 files, the modified weston.ini and +GStreamer pipeline,cfg. You can confirm that transmitter is loaded properly +from weston log as below. + +``` +/* Example(/run/platform/display/weston.log) */ + [12:28:09.127] Loading module '/usr/lib/weston/transmitter.so' + [12:28:09.182] Registered plugin API 'transmitter_v1' of size 88 + [12:28:09.183] Registered plugin API 'transmitter_ivi_v1' of size 16 + [12:28:09.186] Loading module '/usr/lib/libweston-6/waltham-renderer.so' + [12:28:09.255] Transmitter initialized. + [12:28:09.255] transmitter_output_attach_head is called + [12:28:09.255] Weston head attached successfully to output + [12:28:09.255] Output 'transmitter-192.168.2.52:34400-1' enabled with head(s) transmitter-192.168.2.52:34400-1 + [12:28:09.255] Transmitter weston_seat 0xaaab2209c800 + [12:28:09.255] Transmitter created pointer=0xaaab220e6b50 for seat 0xaaab2209c800 + [12:28:09.255] Transmitter created keyboard=0xaaab22038e60 for seat 0xaaab2209c800 + [12:28:09.255] Transmitter created touch=0xaaab220c7930 for seat 0xaaab2209c800 +``` + +2. [receiver side] Start receiver application. + +The below example shows the case if you use waltham-receiver as receiver application. + +1. Add surface to the display where you want to receive transmitted contents. +You can check which output name is required by using the command +`LayerManagerControl get scene`. + +``` + $ layer-add-surfaces -s <surface count> -l <layer id> -d <output display name> & +/* Example */ + $ layer-add-surfaces -s 1 -l 1 -d HDMI-A-1 & +``` +2. Start waltham-receiver. You can use debug option with "-v" if you want. +``` + $ waltham-receiver -p 34400 -v & +``` + +3. Now, receiver side is waiting for the contents coming from transmitter side. + +**You must configure and start receiver side first.** + +4. [transmitter side] Start an IVI application +5. Put the surface of IVI application onto transmitter-output by using + LayerManagerControl command. + +This surface will be transmitted. + +``` + $ layer-add-surfaces -d <transmitter -output name> -s <surface count on receiver> -l <layer id on receiver> +/* Example */ + $ layer-add-surfaces -d transmitter-192.168.2.52:34400-1 -s 1 -l 1 & +``` + +5. [transmitter side] make sure that weston.log shows remoting has been started. + +``` +/* Example(/run/platform/display/weston.log) */ + [12:29:33.224] surface ID 1 + [12:29:40.622] gst-setting are :--> + [12:29:40.622] ip = 192.168.2.52 + [12:29:40.622] port = 34400 + [12:29:40.622] bitrate = 3000000 + [12:29:40.622] width = 1080 + [12:29:40.622] height = 1920 + [12:29:42.177] Parsing GST pipeline:appsrc name=src ! videoconvert ! video/x-raw,format=I420 ! jpegenc ! rtpjjpegpay ! udpsink name=sink host=192.168.2.52 port=3440 sync=false async=false +``` + +## Typical issues & Tips + +### help functions +You can find the help information of LayerManagerControl command by using + +``` +$ LayerManagerControl help +``` + +### waltham-transmitter and waltham-receiver doesn't not communicate + +1. Please check ethernet connection. If you assign 192.168.2.51 and + 192.168.2.52 for waltham-transmitter and waltham-receiver, you shall ping + vice versa. + +``` +/* At waltham-transmitter side */ + $ ping 192.168.2.52 + +/* At waltham-receiver side */ + $ ping 192.168.2.51 +``` + +2. Make sure that IP address specified in the weston.ini under + [transmitter-output] matches the waltham-receiver IP address. + +3. Make sure that IP address in pipeline.cfg on the transmitter side match the + waltham-receiver's IP address. + +### surface,layer or output information is unknown. +You can check them by using the below command. + +``` + $ LayerManagerControl get scene + +/* Example */ + screen 0 (0x0) + --------------------------------------- + - connector name: HDMI-A-1 + - resolution: x=1024, y=768 + - layer render order: 100(0x64), + + layer 100 (0x64) + --------------------------------------- + - destination region: x=296, y=0, w=432, h=768 + - source region: x=0, y=0, w=432, h=768 + - opacity: 1 + - visibility: 1 + - surface render order: 1(0x1), + - on screen: 0(0x0) + + surface 1 (0x1) + --------------------------------------- + - created by pid: 3338 + - original size: x=432, y=768 + - destination region: x=0, y=0, w=432, h=768 + - source region: x=0, y=0, w=432, h=768 + - opacity: 1 + - visibility: 1 + - frame counter: 47 + - on layer: 100(0x64) + + screen 1 (0x1) + --------------------------------------- + - connector name: transmitter-192.168.2.52:34400-1 + - resolution: x=1920, y=1080 + - layer render order: + + /* You can know the output name for remoting is "transmitter-192.168.2.52:34400-1" */ +``` diff --git a/docs/images/01_Load_transmitter.jpg b/docs/images/01_Load_transmitter.jpg Binary files differnew file mode 100755 index 0000000..d9e0527 --- /dev/null +++ b/docs/images/01_Load_transmitter.jpg diff --git a/docs/images/02_Establish_connection.jpg b/docs/images/02_Establish_connection.jpg Binary files differnew file mode 100755 index 0000000..2ad5bc8 --- /dev/null +++ b/docs/images/02_Establish_connection.jpg diff --git a/docs/images/03_Forward_surface.jpg b/docs/images/03_Forward_surface.jpg Binary files differnew file mode 100644 index 0000000..38c2ef2 --- /dev/null +++ b/docs/images/03_Forward_surface.jpg diff --git a/docs/images/04_Rendering_using_gstreamer.jpg b/docs/images/04_Rendering_using_gstreamer.jpg Binary files differnew file mode 100644 index 0000000..97d7858 --- /dev/null +++ b/docs/images/04_Rendering_using_gstreamer.jpg diff --git a/docs/images/05_Input_handling.jpg b/docs/images/05_Input_handling.jpg Binary files differnew file mode 100644 index 0000000..197bbc0 --- /dev/null +++ b/docs/images/05_Input_handling.jpg diff --git a/docs/images/06_Retry_connection.jpg b/docs/images/06_Retry_connection.jpg Binary files differnew file mode 100644 index 0000000..45cb437 --- /dev/null +++ b/docs/images/06_Retry_connection.jpg diff --git a/docs/images/Diffrence_between_Wayland_and_Waltham.jpg b/docs/images/Diffrence_between_Wayland_and_Waltham.jpg Binary files differnew file mode 100644 index 0000000..0383654 --- /dev/null +++ b/docs/images/Diffrence_between_Wayland_and_Waltham.jpg diff --git a/docs/images/Waltham_Architecture.jpg b/docs/images/Waltham_Architecture.jpg Binary files differnew file mode 100644 index 0000000..f1429f4 --- /dev/null +++ b/docs/images/Waltham_Architecture.jpg diff --git a/docs/images/Waltham_In_Practice.jpg b/docs/images/Waltham_In_Practice.jpg Binary files differnew file mode 100644 index 0000000..f413092 --- /dev/null +++ b/docs/images/Waltham_In_Practice.jpg diff --git a/docs/images/Waltham_Integration_Possibility-01.jpg b/docs/images/Waltham_Integration_Possibility-01.jpg Binary files differnew file mode 100644 index 0000000..4c2d08d --- /dev/null +++ b/docs/images/Waltham_Integration_Possibility-01.jpg diff --git a/docs/images/Waltham_Integration_Possibility-02.jpg b/docs/images/Waltham_Integration_Possibility-02.jpg Binary files differnew file mode 100644 index 0000000..a45cbe8 --- /dev/null +++ b/docs/images/Waltham_Integration_Possibility-02.jpg diff --git a/docs/images/Waltham_Integration_Possibility-03.jpg b/docs/images/Waltham_Integration_Possibility-03.jpg Binary files differnew file mode 100644 index 0000000..9e98282 --- /dev/null +++ b/docs/images/Waltham_Integration_Possibility-03.jpg diff --git a/docs/waltham-book.yml b/docs/waltham-book.yml new file mode 100644 index 0000000..313ebf7 --- /dev/null +++ b/docs/waltham-book.yml @@ -0,0 +1,16 @@ +type: books +books: +- + id: waltham-docs-source + title: "AGL waltham" + description: AGL waltham Description + keywords: + author: "Advanced Driver Information Technology" + version: master + chapters: + - url: 0-waltham-overview.md + name: Waltham Overview + - url: 1-waltham-client-and-receiver.md + name: Waltham Client and Receiver + - url: 2-waltham-how-to-build.md + name: Waltham How to Build |
