WIP chore: audio recognition (opus)

[marinakr]

draft for an audio recognition (does not work yet, prototype)
TO DO: fix rooms.js, room.ex

[mickel8]

Hi @marinakr, that would be super cool! Why closing?

[marinakr]

Hi @mickel8! I accidentally created the PR to main branch instead of my fork
It was not ready yet at, draft with few debug messages
The current issue that audio stream separated from video stream is not working (predictions is nonsense) and I am trying to get separate audio stream from room.js
Current implementation is not working, data sent from rooms.js (pushed to channel) produces {:error, :no_keyframe} on decode
It probably should be attached to RTCPeerConnection
But simple attach audiotrack does not send audio data where you expect it in

  {:ex_webrtc, _pc, {:rtp, track_id, nil, packet}},
        %{audio_track: %{id: track_id}} = state

[mickel8]

I think you should focus on making sure that audio packets are actually received on the backend side. It looks like you don't add the audio track to the peer connection?

[marinakr]

I have managed to get the audio (actually I already did it in not committed, we could use audio from video, but results was not good so I decided to get audio separately)
In my branch I have the audio, the current issue is I have nonsense predictions (asked this question in elixir forum)
Everything seems to be working but text prediction have low quality

UPD:
fixed predictions, will do PR for audio ~next weekend

[marinakr]

Updated PR: #81