How to achieve live transcription

[csukuangfj]

The title of the paper https://arxiv.org/pdf/2410.15608 is

Moonshine: Speech Recognition for Live Transcription and Voice Commands

However, the model is a non-streaming model, could you describe how to achieve live transcription?

The demo in this repo is for decoding files,
it would be nice if you can provide a demo for live transcriptions.

[info-wordcab]

Definitely would be interesting to see a live transcription demo!

[evmaki]

Thanks for the request! We are working on a live transcription demo that we will add to the repo soon.

[andimarafioti]

I know it's not exactly what you asked for, for I added the model to Hugging Face's speech to speech library, so it does live translation

[sleepingcat4]

@evmaki I tried to ran a benchmark and results were very disappointing. I had used below specifications:

1. JAX backend
2. 10s splits
3. percussive component is used for transcription
4. librosa to remove silence (trim func)
5. tiny variation of moonshine was used

in the benchmark only moonshine was loaded and used for transcriptions on the audio segments. rest were preprocessing steps. result was 48 minutes to transcribe 1 hour 28 minutes long video on CPU.

I ran a benchmark using Faster-whisper tiny model on CPU with below specifications

1. CPU backend
2. int8
3. same preprocessing steps
4. 10s segments was fed at a given moment
5. same func as moonshine was used except model was changed

It took under 8 minutes to transcribe that same 1 hour 28 minutes audio file. Even I liked faster Whisper's tiny model quality of transcriptions better.

provided these results how you guys are promising live-transcriptions?

[evmaki]

@sleepingcat4 thanks for benchmarking and sharing your results. The Keras implementation currently has some speed issues, which is what's causing this. We've added ONNX models that run much faster. I encourage people to try those out.

Re: live transcriptions. We'll soon be merging a demo (using the ONNX models) that shows live captioning in action. The branch is already public if you want to check it out. Demo script is located here.

PS: What CPU are you running on? And are you willing to share your script so we can reproduce your results?

[csukuangfj]

@evmaki

Thank you for sharing the demo!

By the way, for the following two lines:
https://github.com/usefulsensors/moonshine/blob/5689bdf58818d26029cb59a9f25b186f8971e4a9/moonshine/demo/live_captions.py#L153
https://github.com/usefulsensors/moonshine/blob/5689bdf58818d26029cb59a9f25b186f8971e4a9/moonshine/demo/live_captions.py#L179

they result in redundant computation, which is not efficient.

Is there a plan to release a streaming model?

[sleepingcat4]

@evmaki below is the function I used to transcribe 10s segement at a moment.

def transcribe_folder(audio_folder, output_file, model='moonshine/tiny'):
    initial_time = time.time()

    transcription = ""
    audio_files = sorted(
        [f for f in os.listdir(audio_folder) if f.endswith('.wav')],
        key=lambda x: int(re.search(r'part(\d+)\.wav', x).group(1))
    )

    for i, file_name in enumerate(audio_files):
        file_path = os.path.join(audio_folder, file_name)
        transcript = moonshine.transcribe(file_path, model)
        transcript = " ".join(transcript)

        seg_st_time = i * 10
        seg_en_time = seg_st_time + 10

        start_h = seg_st_time // 3600
        start_m = (seg_st_time % 3600) // 60
        start_s = seg_st_time % 60

        end_h = seg_en_time // 3600
        end_m = (seg_en_time % 3600) // 60
        end_s = seg_en_time % 60

        transcription += f"{start_h:02}:{start_m:02}:{start_s:02}-{end_h:02}:{end_m:02}:{end_s:02}s: {transcript}\n"

    with open(output_file, 'w') as f:
        f.write(transcription)

    end_time = time.time()
    execution_time = (end_time - initial_time) / 60
    print(f"Execution Time: {execution_time:.2f} minutes")
    print(f"Transcription saved to: {output_file}")

transcribe_folder("Yj7ZDcHGtK", output_file="bench_script.txt")

[csukuangfj]

@evmaki I tried to ran a benchmark and results were very disappointing. I had used below specifications:

1. JAX backend
2. 10s splits
3. percussive component is used for transcription
4. librosa to remove silence (trim func)
5. tiny variation of moonshine was used

in the benchmark only moonshine was loaded and used for transcriptions on the audio segments. rest were preprocessing steps. result was 48 minutes to transcribe 1 hour 28 minutes long video on CPU.

I ran a benchmark using Faster-whisper tiny model on CPU with below specifications

1. CPU backend
2. int8
3. same preprocessing steps
4. 10s segments was fed at a given moment
5. same func as moonshine was used except model was changed

It took under 8 minutes to transcribe that same 1 hour 28 minutes audio file. Even I liked faster Whisper's tiny model quality of transcriptions better.

provided these results how you guys are promising live-transcriptions?

@sleepingcat4

I suggest that you test it again with sherpa-onnx, which has supported Moonshine models.

The following colab notebook guides you step-by-step how to do that.

https://github.com/k2-fsa/colab/blob/master/sherpa-onnx/RTF_comparison_betwen_whisper_and_moonshine.ipynb

[csukuangfj]

By the way, if you want to try Moonshine models on Android, you can download pre-built APKs from
https://k2-fsa.github.io/sherpa/onnx/vad/apk-asr.html

E.g., the following links:

• https://huggingface.co/csukuangfj/sherpa-onnx-apk/resolve/main/vad-asr/1.10.29/sherpa-onnx-1.10.29-arm64-v8a-vad_asr-en-moonshine_tiny_int8.apk
• https://huggingface.co/csukuangfj/sherpa-onnx-apk/resolve/main/vad-asr/1.10.29/sherpa-onnx-1.10.29-armeabi-v7a-vad_asr-en-moonshine_tiny_int8.apk
• https://huggingface.co/csukuangfj/sherpa-onnx-apk/resolve/main/vad-asr/1.10.29/sherpa-onnx-1.10.29-x86_64-vad_asr-en-moonshine_tiny_int8.apk
• https://huggingface.co/csukuangfj/sherpa-onnx-apk/resolve/main/vad-asr/1.10.29/sherpa-onnx-1.10.29-x86-vad_asr-en-moonshine_tiny_int8.apk

[keveman]

Thanks so much @csukuangfj, for providing the notebook, and the comparing with Whisper. Just to summarize, Moonshine tiny can transcribe 335.2 seconds of audio in 19.6 seconds whereas whisper tiny.en needs 64.1 seconds. That's a 3.3x speedup, and the transcription quality looks identical. @sleepingcat4 I hope this helps alleviate some of your disappointment. The provided Keras implementation is indeed non-optimal and we released it to be a reference with future enhancement in mind. Most deployments we had in mind (such as on SBCs) would never have Torch or TF or JAX or other such large frameworks installed. ONNX, TFLite or some other homegrown runtimes are best suitable for seeing the benefits of Moonshine.

[sleepingcat4]

@csukuangfj thanks for the notebook. I was writing an audio transcription pipeline so I needed to have benchmarks on large audio before to get started. results on your notebook looks interesting (my benchmark avoided ONNX models) but I don't think moonshine is still a good fit for making my dataset or pipeline.

@keveman Moonshine's transcription quality is par to OpenAI Whipser model but it not exactly at the same level with faster-whisper tiny int8 model. Faster-whisper can capture more nuanced information. If moonshine can match both the speed of tiny whisper on faster-whipser library, I would definitely integrate it into my pipeline.

But, I will look into the ONNX models and few other models as well. Because from LAION AI, we are planning to maintain an open-source repo to share our pipeline and benchmark models and share them with everyone. Meanwhile really thanks for the responses and help.