1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
|
**Table of Content**
1. TOC
{:toc}
## waltham-client-server-applications
### waltham-client
waltham-client uses Waltham IPC library to connect to remote host and transmit
client buffers using GStreamer framework.
waltham-client is divided into two components:
* waltham-transmitter plugin: waltham-transmitter plugin provides API to create
remote connections and push surfaces over the network and handles both remote
output and remote input. Loaded automatically at start-up, by the compositor.
* waltham-renderer: waltham-renderer is the implementation for sharing surface.
It creates a buffer to be transmitted to other domain. Latest implementation
uses GStreamer framework.
### waltham-receiver
waltham-receiver is a sample implementation of the receiver app which shall be
running at the remote side. It is developed based on [waltham
server](https://github.com/waltham/waltham/tree/master/tests), using Waltham
protocol to obtain and process remote output, which is sent by
waltham-transmitter. This component is designed to be used for evaluating the
functionality of waltham-transmitter plugin.
## How it works
### 1. Loading and initialization
As the compositor is starting up, it loads the waltham-transmitter plugin.
waltham-client connects to the receiver application at the remote side during
initialization.
### 2. Establishing connection
At `transmitter_create_remote()`, waltham-transmitter creates
`weston_transmitter_remote` object which expresses the receiver object at
remote side, and it is created for each receiver.
*waltham-transmitter* uses `wth_display_get_registry()` and `wth_display_sync()`
for the receiver applications in the same manner as the Wayland protocol
would retrieve the interfaces. Then, the receiver applications sends back
the resource list to the waltham-transmitter.
### 3. Forwarding surface
While the compositor redraws the surface, waltham-transmitter sends waltham
protocol messages to the receiver app to notify about those surface updates.
`wthp_surface_attach()`, `wthp_surface_damage()` and `wthp_surface_commit()`
correspond to `wl_surface_attach()`, `wl_surface_damage()` and
`wl_surface_commit()` message in wayland protocol.
- `wthp_surface_attach()` - Send wthp_buffer as a buffer handling. This is not
the actual buffer which contains the data to be rendered but the handle of an
actual buffer. It abstracts the differences of buffer type.
- `wthp_surface_damage()` - Tell the updated region to receiver app.
- `wthp_surface_commit()` - Tell surface gets updated to receiver app.
### 4. Rendering using GStreamer
### 5. Input handling
For handling input events, waltham-transmitter has 2 ways to secure seat.
1. Use `wl_seat` as weston has.
2. Create a new `wl_seat`.
Second case is applicable in case the transmitter side does not have an input
device but the receiver at remote side has one. After `wl_seat` is created,
waltham-transmitter sends input events to the client application when
it gets an input event from the receiver via the Waltham protocol.
The message wthp_send_XXX shows you that input event is forwarded from the
receiver to the transmitter, XXX is filled by the input event name.
### 6. Retry connection
In case the connection gets disconnected during surface sharing,
waltham-transmitter shall re-establish the connection. The `waltham_display`
objects represents the connection between the transmitter and the receiver,
and contains a flag that can be used to detect a disconnect.
That happens in `connection_handle_data()` in case a disconnect
is detected. This flag is checked at every call of
`transmitter_surface_gather_state`. When running is in false state,
waltham-transmitter starts to retry the handling sequence
It release the waltham protocol objects then it goes to establish a connection
sequence mentioned in 2. Establishing connection.
## Waltham in practice
Here is the example how waltham can be used in hypervisor use case of real
project.
* Weston is used as the wayland compositor.
* waltham-client is implemented for Weston which acts as a Waltham virtual
display.
* Application surface is assigned to Waltham virtual display and it's sent to
the other ECU/OS. Buffers of surface are transferred via GStreamer(UDP),
since transferring raw pixel data via Waltham(TCP) is not fast enough.
* Controlling input events (pointer, keyboard, touch) for the surface is
handled by Waltham.
## How Waltham can be integrated
Some possible integration examples of waltham follow.
### As an EGL backend (theoretical possibility)
Similarly to Wayland back-end for EGL, Waltham client could be a back-end in
the compositor. For better performance, a generic surface sharing mechanism is
needed in a hypervisor environment. Applications need to adapt to Waltham.
As waltham is not designed with this use in mind this usage is just a
theoretical possibility.
### As a GStreamer sink (theoretical possibility)
Similarly to Wayland sink, a Waltham sink GStreamer plugin can be implemented
which sends the buffers to a receiver on another domain/OS. Waltham sink can
utilize frame synchronization and a presentation feedback protocols for video
synchronization. For better performance, a generic surface sharing mechanism
is needed in a hypervisor environment. As Waltham is not designed with this
use in mind this usage is just a theoretical possibility.
### As a virtual display in compositor
Virtual display plugin can be implemented in the compositor. This plugin sends
client buffers to waltham-receiver in another domain. No changes to
applications. For good performance, a generic surface sharing mechanism is
needed in hypervisor environment. This is the intended use in mind during
design.
|
