Passing of input devices

[samuelhackwill]

Hello!

(context) I came across joonas' blog post while doing some research about multi-pointer interfaces for a theatre performance i'm currently working on (I want every member of the audience to have a mouse and to be able to interact with a big screen). Thank you for sharing your research!

However, we did not manage to get the VNC server running. we tried three methods :

• first, we tried running the docker image on a raspberry pi 3B+, on a fresh install of raspbian (Raspberry Pi OS 6.6), but we got an error because the docker image was intended for use on a AMD/64 machine and the raspberry is ARM.
• then we tried running the docker image on a PC running GNU/Linux, I don't remember the distro but probably an oldish (18.04?) version of Ubuntu. We did not manage to get the docker image running, the logs show that Openbox exits, which causes the image to stop.
• we then tried running older versions of the docker image on the same machine, and this time we did get the container to run, but localhost:80 showed an empty input, and the logs of docker contained an error related to a missing .Xauthority file. We tried manually creating an .Xauthority file to no avail either.

I'm not sure which problem would be easiest to solve, could you please give us a hint to what we could try next?
Please forgive me if my wording is a bit vague, i used my colleague's computer and have no access to the logs at this time.

thank you for your time! :)

[joonas-fi]

Hey!

It is true there is no arm64 version, as I've not needed it.

Seems to work out of the box, the most recent version. I used this command:

docker run -d --name screen-server -p 80:80 --shm-size 512m -e "SCREEN_1=5900,800,1280,Galaxy Tab 2" fn61/screen-server:20240121_1530_c8d20f9d

but localhost:80 showed an empty input

Is it possible that you might get confused by the particular screen being all transparent? That shows on the webpage as white, so in the beginning the frontpage is just the heading "Galaxy Tab 2" because remember this is meant to host multiple screens and so frontpage has this structure:

• Screen 1
  • name as heading
  • content
• Screen 2
  • name as heading
  • content

See, this is from most recent version:

Peek.2024-04-13.18-38.mp4

So one has to click on the thumbnail (all white, I agree it's sucky UX!) to open the web-based VNC viewer. Another alternative is to just use VNC without the webpage.

[joonas-fi]

Seems like adding multi-arch images (ARM 64 & 32 bit) was not a pain at all. Please test if you can! #9

[samuelhackwill]

Dear Jonas,
Thank you for your kind answers. I was absolutely confused and i absolutely got the last image (the multi-platform one) working (on XUbuntu 20.4.3 running on an USB key). That's awesome! I will experiment with the mouse events forwarding next. I must report that i did not manage to run the image on my raspberry pi though, i'm still getting an exec format error. thank you!

[joonas-fi]

Great that got you forward!

The multi-arch build had a bug, turns out it was a bit more work but not bad still. The above mentioned PR has an update, if you have time please try again

[samuelhackwill]

I tested the new image on my Raspberry Pi 3B+ running on Raspbian 6.6, it's working! thank you :))

also : adding --security-opt seccomp=unconfined to the docker run command solved the openbox error that i was getting on my other computer. I guess it's not good practice though, i copypasted it from here.

[samuelhackwill]

However, i haven't managed to pass the mouse events to x11vnc yet. I might be misunderstanding how it's supposed to work :

what i expected was my mouse to stop being responsive (being "grabbed") as soon as i launched the docker image with the proper flag (--device /dev/input/by-id/<mouse>). I was also kind of expecting a second mouse pointer to appear in the vnc preview window, but neither happened. Instead my mouse continued to work as usual. I also tried passing my usb keyboard, but that did not seem to work either.

in both cases, using cat from inside the container, as suggested in the blog post, did show symbols being printed as i interacted with the relevant usb device (mouse or keyboard), which suggests the code is working! .. So i'm probably not testing the right way.

thank you for your help, i appreciate it.

[joonas-fi]

also : adding --security-opt seccomp=unconfined to the docker run command solved the openbox error that i was getting on my other computer

Now that you mention it, this worked without the seccomp flag on my work computer, but on my other computer I also had to use that flag.. maybe the need for it is fixed in updated Docker versions or such (I have 24.0.5 on where I don't need the flag, 20.10.7 where I need it)

[joonas-fi]

Anyways, converting to discussion as the discussion no longer concerns openbox or a bug in screen-server

[joonas-fi]

For context, the blog post you mentioned is this: https://joonas.fi/2020/12/attach-a-keyboard-to-a-docker-container/

However, i haven't managed to pass the mouse events to x11vnc yet.

what i expected was my mouse to stop being responsive (being "grabbed") as soon as i launched the docker image with the proper flag (--device /dev/input/by-id/)

Thanks for adding context via the expectation. So --device is only first part of the puzzle, that makes the device visible to inside of the container, but the app (screen-server) does not do anything with it yet.

The second part of the puzzle is this code:

screen-server/cmd/screen-server/main.go

Line 219 in 4521f64

processes.Start("evdev", inputDev.ScanInputGrabbed)

That gets only triggered if this option is set:

screen-server/cmd/screen-server/main.go

Line 193 in 4521f64

if screen.Opts.AttachInputDevice != "" {

That gets set here:

screen-server/cmd/screen-server/main.go

Line 323 in 4521f64

AttachInputDevice: screenDefParts[5],

So it's based on the SCREEN_1 ENV variable, the format of which is:

The format for the SCREEN_n parameter is <VNC port>,<display width>,<height>,<screen name>,[<input device>]

Source: readme

So you need to also add the device that you gave to Docker as --device to the ENV var as well.

Yes, this is too little documented..

[samuelhackwill]

Thank you so much for your time, i got everything working!

In the video below i am interacting with firefox though my "grabbed" mouse as expected. My pointer is not visible but that's not an issue in my use case.

trimmedout.mp4

In this other video i'm using the signal from two concurrent docker images running a custom webapp in the browser. The two "grabbed" mouse positions are relayed to the webapp and used to dynamically move the position of two "pointers" on another client.

multipointer.SUCCESS.mov

I am throttling the number of messages sent to the server but the next step is to try implementing a webRTC data channel to see if smooth 24 FPS updates of the pointer positions are possible. The goal is to get ~100 mice connected together in the end project!

i'd like to thank you once more for your clarity and your kindness.
I'll keep you updated on my progress!

[joonas-fi]

Excellent to hear that you got it working! 😄 Yes I'm very much interested in the end result that you manage to make! 💪

[samuelhackwill]

Dear @joonas-fi, I'm back to report on the completion of my initial project, which was kickstarted thanks to your research. Let me break up the steps i took :

• first prototype, which we discussed here, was made in april 2024

• and it was playtested for the first time in december 2024 ; i actually got 40 people in a cinema to get a feel of what the multiplayer madness could look like. I also got the insight that setting up the whole stack with 10 raspberries, 40+ mice and the virtualization was barely manageable, it was time for a refactor!

• i then prompted my way into making a python script (i sadly don't know python yet) to pipe the evdevs towards a node.js server using websockets, in order to simulate mouse pointers on a web client. That ended up being a more direct way of achieving was i was looking for.

• many playtests, design residencies, writing and coding later, the show premiered in April of this year. The subject of the play had become a playful digression on the subject of CAPTCHAs. I'm now busy touring the show in France.

you can watch a teaser here to get a glimpse of what games the crowd engage in.

tech-wise, the bottom line of the project is that i did end up running into performance issues, probably because i'm animating many objects in certainly not the most efficient way in a web browser. The tradeoff of not using "native" pointers was that i had to pretty much re-code all the nice & free browser interactions & animations in Javascript (like CSS on-hover effects for instance).
Using the web browser as a game engine is probably good for quick iteration & multiplayer playtesting, but it does have its limitations (esp. when using non-specialized frameworks, or "vanilla" js & CSS rather than webGL? i guess).

Anyway that was a super fun project, thank you very much for helping with the very first steps, which put me right on track!
until next time (:

[samuelhackwill]

fwiw, the code lives in this repo : https://github.com/samuelhackwill/tryhard

[samuelhackwill]

and the mouse grabber script is here : https://github.com/samuelhackwill/tryhard/blob/main/rasps/mouse-grabr.py