Sortformer BNNS Graph Error

[ashayk]

Using any of the Sortformer models (v2)... seeing:

Failed to configure ML Program for the feature types declared in the model description. It is likely an invalid or broken model. (Underlying exception: Couldn't find tensor output with name chunk_pre_encoder_embs_out)
BNNS Graph Compile: Function main has tensor chunk_pre_encoder_embs_out as both an input and output. Inputs and outputs must be distict, please an explicit identity op. at /private/var/mobile/Containers/Data/Application/2C0038B9-647A-4326-BD52-19C65729C874/Library/Application Support/FluidAudio/Models/sortformer/Sortformer_v2.1.mlmodelc/model1/model.mil:4:4
E5RT encountered an STL exception. msg = unordered_map::at: key not found.
E5RT: unordered_map::at: key not found (11)

XCode 26.5, FluiAudio (main)

I thought this had been previously addressed?

Thanks.

[Alex-Wengg]

New iOS software updates often cause this

[ashayk]

Thanks. Actually running iOS 18 on device, but XCode 26.

Curiosly, despite the error, it still runs in the simulator. But there I'm noticing each call to SortformerModelInference.runMainModel():

let output = try mainModel.prediction(from: inputFeatures)

is taking .22 seconds, or about 27x slower than real time based on 112 chunk data length.

I'm getting nearly perfect parity with Argmax v2 in terms of diarization accuracy, just impossibly slow.

[Alex-Wengg]

@ashayk if you have a benchmark or wav as comparisons i could use that would be great so i can see about optimizations

[ashayk]

Suffer_1080_Trim_16.wav.zip

Thanks, I've attached one. The easiest thing in terms of benchmarking is to run the Argmax Playground and compare how its CoreML Sortformer performs. It's blazingly fast even with Parakeet transcription running on top. 5 min audio transcribed and diarized in 1.36sec (from .m4a) on an M4 Pro.

[ashayk]

I have an update. Model loading on an A14 (iPhone Mini), requires .cpuOnly compute units to be compatible with the model. This seems to be an issue that's common on pre-M1 devices. The time delay issue was related to a mismatch between diarizer and model configs. Perhaps the API can check for mismatches? With both set to highContextV2_1, performance is quite good. The only other small issue is that highContextV2_1 uses an enormous amount of RAM making it crash on older devices. I'm not sure what, if anything can be done about that?

[Alex-Wengg]

#728 i think i might have fix for this ready . could you give it a try here.

[ashayk]

Thanks. Can confirm models load without the BNNS error.
ModelNames.modelsSubdirectory is a let constant and can't be changed to enable the palettized models, however I manually changed the source to confirm memory usage and can confirm ~330mb RAM usage down from 2.4gb. However, loading on A14 still requires .cpuOnly units with .highContextV2_1, otherwise loading hangs. It seems to resolve eventually, but it takes minutes.

[ashayk]

It's also worth adding in terms of benchmarking that Argmax v2 Playground is freely available on MacOS. It uses their Sortformer CoreML model which diarizes and transcribes (Sortformer/Parakeet) 5 min of compressed audio in 1.32 sec on an M4 Pro Mac. Their models are downloaded locally and available (via the folder icon). They are proprietary, however they may be a good target for analysis to understand why their Sortformer implementation remans > 10x faster than the v3/highContextV2_1.