Validate downloaded model artifacts before caching (#740)#741
Conversation
An unhealthy network path — captive portal, corporate proxy, mirror 5xx — can return an HTML error page or a truncated body under a model's filename. That content was written into the cache, so the presence-based modelsExist(...) check reported the model as installed and the failure only surfaced later as a confusing CoreML compile error far from the root cause. Validate each downloaded file to a temp path before moving it into the cache, in both download paths (downloadRepo via downloadFileWithRetry, and downloadSubdirectory): - Reject Content-Type: text/html responses (proxy/captive-portal pages). - Reject empty bodies. - Sniff leading bytes for <!DOCTYPE html / <html / <?xml markup (error pages served with 200 under the model's filename). - Exact size match against the size HuggingFace reports for the object (verified to be the resolved LFS object size, not the pointer) to catch truncation. Failures throw a distinct HuggingFaceDownloadError.invalidArtifact(path:reason:) so callers get "download corrupt — retry" instead of a downstream load crash. The error is classified retryable, so a transient bad fetch is retried with the existing backoff instead of poisoning the cache on the first blip. Adds DownloadArtifactValidationTests covering HTML/XML/doctype detection, binary/JSON pass-through, content-type rejection, empty/truncated/oversized bodies, and the unknown-size skip.
Supertonic3 Smoke Test ✅
Runtime: 0m19s Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf. |
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 0m55s • 06/28/2026, 11:34 AM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
PocketTTS Smoke Test ✅
Runtime: 0m6s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon. |
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 130.0s processing • Test runtime: 2m 10s • 06/28/2026, 11:43 AM EST |
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 37.3s diarization time • Test runtime: 2m 21s • 06/28/2026, 11:35 AM EST |
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
Move the rationale out of code comments and into history, where it belongs. Context for the three helpers, for the record: - looksLikeHTML(_:): returns true when the body begins with HTML/XML markup (<!doctype html / <html / <?xml). Catches proxy / captive-portal error pages served as a normal 200 body under the requested filename. Only the leading 512 bytes are inspected, decoded lossily so a stray non-UTF8 byte further in can't hide an HTML preamble. - validateDownloadedArtifact(at:response:path:expectedSize:): runs before the temp file is moved into the cache. Rejects, in order: text/html Content-Type, empty body, HTML-markup body, and size mismatch. Throws HuggingFaceDownloadError.invalidArtifact, which isRetryableDownloadError classifies as retryable, so a transient bad fetch is retried with the existing backoff instead of poisoning the cache. expectedSize <= 0 (unknown) skips the size check. - invalidArtifact(path:reason:): a downloaded artifact failed validation before being cached (HTML error page, empty body, or size mismatch); the bad bytes are discarded so the cache is never poisoned. No behavior change — comments only.
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 4m 41s • 2026-06-28T15:37:47.495Z |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 8m48s • 06/28/2026, 11:46 AM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
1 participant
Closes #740.
Problem
AsrModels.download(...)/DiarizerModels.downloadIfNeeded()write HuggingFace artifacts straight into the cache. When the network path is unhealthy (captive portal, corporate proxy, mirror 5xx/redirects), the HTTP body can be an HTML error page or a truncated file rather than the real artifact. It gets written under the expected filename, so the presence-basedmodelsExist(at:version:)check reports the model as installed. The failure only surfaces later at load/compile time as an opaque CoreML error, far from the root cause.Change
All model downloads route through
DownloadUtils. Each file is fetched to a temp path and validated before it is moved into the cache, so a bad fetch never poisons the cache dir. Validation is wired into both download paths —downloadRepo(viadownloadFileWithRetry) anddownloadSubdirectory:Content-Type: text/htmlresponses (proxy / captive-portal pages).<!DOCTYPE html/<html/<?xmlmarkup (error pages served with200under the model's filename).Failures throw a distinct
HuggingFaceDownloadError.invalidArtifact(path:reason:)so callers can present "download corrupt — retry" instead of a downstream load crash. The error is classified retryable, so a transient bad fetch is retried with the existing bounded backoff rather than failing the whole repo download on the first blip.Size check is safe for LFS
Verified against the HF tree API that the
sizefield is the resolved LFS object size, not the 130-byte pointer (e.g.Encoder.mlmodelc/weights/weight.binreports445187200, matchinglfs.size). So the exact-match truncation check won't false-positive on large weights. Legitimately-empty files (APIsize == 0) are still created directly and bypass validation.Tests
Adds
DownloadArtifactValidationTests(13 cases): HTML/XML/doctype detection, binary/JSON pass-through, content-type rejection, empty / truncated / oversized bodies, unknown-size skip, and the error description.