Add audio property to VideoDecoder

[Samoed]

Previously, accessing audio from a video file required creating a separate AudioDecoder alongside VideoDecoder #1158. This split made it difficult to integrate with external libraries such as huggingface datasets utilities that expect audio to be accessible directly from the video decoder - the absence of an audio attribute on VideoDecoder either required workarounds or left audio support missing entirely in those integrations.

This PR adds a lazy audio property to VideoDecoder that returns an AudioDecoder for audio stream in the same source, or None if the source contains no audio stream

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1442

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• Run pull request jobs on OSDC runners in shadow mode

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[NicolasHug]

Hi @Samoed , can you share more about this:

This split made it difficult to integrate with external libraries such as huggingface/datasets#8007 utilities that expect audio to be accessible directly from the video decoder - the absence of an audio attribute on VideoDecoder either required workarounds or left audio support missing entirely in those integrations

Specifically I'd like to understand why having two seprate VideoDecoder and AudioDecoder objects lead to friction, and how does the proposed design solve it

[Samoed]

The Datasets library outputs video inputs as VideoDecoder, and to be able to easily extract audio information it typically splits inputs into video and audio columns before uploading. Extracting audio input from such videos requires some hacks. Also, I think it would be good to be able to work with one object during data processing to handle both audio and video

[Samoed]

Hi @NicolasHug! I thought about this feature more and found more use cases. Currently Transformers have a processor kwarg load_audio_from_video https://github.com/huggingface/transformers/blob/63d64d0be41940cb6383d109194a7dc323261252/src/transformers/processing_utils.py#L544 and use_audio_in_video https://github.com/huggingface/transformers/blob/63d64d0be41940cb6383d109194a7dc323261252/src/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py#L74 which extractiang audio only from video paths, but not VideoDecoder objects (overall, transformers don't support VideoDecoder inputs, but I think this could change in the future). If you add this feature it would help to standardize this and provide more opportunities to use VideoDecoder

[NicolasHug]

Sorry I still don't get it - can you share a code example of what this would enable, and why it can't already be achieved by just keeping a ref to the source video (and creating a separate AudioDecoder from it)?

[Samoed]

HuggingFace datasets decodes a video column directly into a VideoDecoder:

from datasets import load_dataset

test_ds = load_dataset("Samoed/testds")
video = test_ds["train"][0]["video"]  # VideoDecoder without path

# Today: no clean way to get audio from `video`. Users either
#   - reach into `video._source` with Video(decode=False)
#   - split the dataset into separate `video` + `audio` columns upstream.

audio = video.audio

For huggingface transformers

def __call__(self, videos: list[VideoDecoder | ...], ...):
    for v in videos:
        frames = v.get_frames_in_range(...)
        if self.use_audio_in_video and decoder.audio is not None:
            audio = v.audio.get_all_samples()

[NicolasHug]

The above is one reason why we can't support the proposal as-is. We don't want to convert a file-like to bytes, because it forces materialization or full download of what is supposed to be a streaming object - it defeats the entire purpose of file-like support.

I thought of perhaps exposing the .source attribute publicly instead of exposing an AudioDecoder, but that would still be subject to the same issue with file-like objects. Let me ask the datasets maintainer to see if there's anything that can be done on that side

[NicolasHug]

@lhoestq curious if you have any suggestion on datasets side to address @Samoed 's workflow from #1442 (comment) ?

[tomaarsen]

This also affects Sentence Transformers currently, e.g.:

from datasets import load_dataset
from sentence_transformers import SentenceTransformer

test_ds = load_dataset("Samoed/testds")
video = test_ds["train"][0]["video"]

model = SentenceTransformer("...")  # a video-capable model
embedding = model.encode(video)

Internally, ST converts this VideoDecoder into 1) frame-data plus 2) video_metadata like fps, duration, etc., which is then passed to transformers. I can't easily also take the audio from this video variable, I believe.
On the other hand, if I use load_audio_from_video (https://github.com/huggingface/transformers/blob/63d64d0be41940cb6383d109194a7dc323261252/src/transformers/processing_utils.py#L544), and I just pass a URL or path, then I can have transformers grab the audio as well (where transformers uses AudioDecoder for it, this could also perhaps be simplified if it could be grabbed directly from the VideoDecoder).

• Tom Aarsen