How to get inference for batch size > 1

[chrisreese-if]

This is critical for server usecases, to improve concurrency on GPU. Faster Whisper added SYSTRAN/faster-whisper#856. For live use case this means multiple concurrent transcriptions can happen on the same GPU/model. This will significantly lower costs of deployment.

Is there a way to do this now? Or is this something on the roadmap?

Thanks!

[njeffrie]

As you pointed out, we currently don't support batching. It's probably quite straightforward to add batching to our CTranslate2 Moonshine implementation. Adding batching support to the models in this repo is currently on my roadmap, and something I'll be working on in the next few weeks.

[kalradivyanshu]

@njeffrie So in the ctranslate implementation I can pass multiple audios in and it will transcribe them together?

[njeffrie]

I addressed a dimension ordering issue in the PR and I checked that batching now works correctly with CTranslate2. You have to pass in multiple batched audios (padded to match length) as well as one [int(1)] SOT prompt for each batch.

[LazyCoder2k3]

Which version CTranslate2 did you use sir? I'm using version 4.6.0 but can't load the package model in CTranslate2 format downloaded from your HuggingFace card.