Non-commercial research use only
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
These files are Core ML conversions of DiariZen (BUT-FIT/diarizen-wavlm-large-s80-md-v2) by BUT Speech@FIT. The DiariZen weights are licensed under CC BY-NC 4.0, and BUT Speech@FIT states that by downloading them you agree to use them for research and academic purposes only. The converted files carry the same terms: no commercial use, and you must give appropriate credit to DiariZen (BUT Speech@FIT) and to the WavLM, pyannote.audio and WeSpeaker work it builds on, link to the license, and indicate changes. License: https://creativecommons.org/licenses/by-nc/4.0/
Log in or Sign Up to review the conditions and access this model content.
DiariZen (WavLM Large s80, v2) for Core ML
The neural parts of BUT Speech@FIT's DiariZen speaker diarization pipeline, checkpoint BUT-FIT/diarizen-wavlm-large-s80-md-v2, converted to Core ML for Apple Silicon. The checkpoint is a WavLM Large front end, structurally pruned to 80% sparsity (316.6M to 63.3M parameters), under a Conformer with a powerset head that allows up to four overlapping speakers; DiariZen pairs it with a WeSpeaker ResNet34 embedding and VBx clustering, with no cap on the number of speakers.
These are the files the Swift runtime diarizen-apple loads. That project has the rest of the pipeline in Swift (sliding window, powerset decoding, Kaldi filterbank, VBx, reconstruction to RTTM) and the steps to convert the models yourself; this repository saves you the conversion.
It is the more accurate and slower of the two converted checkpoints; the base model is in nikioth/diarizen-wavlm-base-s80-md-coreml. Its Core ML segmentation is less faithful to PyTorch at the tensor level than the base model's, while its diarization error is lower (see Accuracy).
Non-commercial use only. See the license section.
Files
Segmentation.mlpackage/ 152 MB, float16: WavLM Large s80 + Conformer, powerset head
Embedding.mlpackage/ 13 MB, float16: WeSpeaker ResNet34-LM with masked statistics pooling
plda_transform.json 1.0 MB: DiariZen's VBx PLDA, eigendecomposition folded in
conversion/ the export scripts (MIT, from diarizen-apple)
LICENSE CC BY-NC 4.0 legal code
Accept the terms on this page, log in (hf auth login), download, and point
the runtime at the folder from a clone of diarizen-apple:
hf download nikioth/diarizen-wavlm-large-s80-md-v2-coreml --local-dir models/large
swift run -c release diarizen-cli meeting.wav --models models/large --output meeting.rttm
Model inputs and outputs
| Model | Input | Output |
|---|---|---|
Segmentation.mlpackage |
waveform (1, 1, 256000): one 16 s chunk, mono 16 kHz, samples in [-1, 1] |
segmentation (1, 799, 16): per-frame powerset log-probabilities |
Embedding.mlpackage |
fbank (B, 1598, 80) and weights (B, 799), B = 1 to 64 |
embedding (B, 256) |
Arrays are declared Float16; Core ML converts Float32 input.
Segmentation frames are 25 ms every 20 ms (frame f starts at about
f * 0.02 - 0.0075 s within the chunk). The 16 powerset classes encode up to
4 local speakers, all of whom may be active in one frame: the empty set, the 4
single speakers, the 6 pairs, the 4 triples and the set of all four, in
pyannote's powerset order. Take the argmax per frame and map it to its
speaker set. DiariZen slides the chunk with a 1.6 s step, applies an 11-frame median filter to the decoded
activities, and aggregates overlapping chunks.
The embedding input is WeSpeaker's frontend: torchaudio.compliance.kaldi.fbank
on samples scaled by 32768 (80 mel bins, 25 ms Hamming frames every 10 ms,
DC removal, pre-emphasis 0.97, no dither) with the mean over the 1598 frames
subtracted, computed on the same 16 s chunk. weights marks the frames of one
local speaker (pyannote uses the speaker's non-overlapped frames when there are
enough of them). The ResNet34 is the same network as
nikioth/wespeaker-resnet34-lm-coreml,
which documents the frontend in detail.
plda_transform.json holds mean1, lda, mean2, plda_mu, plda_tr
and plda_psi as {"shape", "data"} float64 arrays: the output of VBx's
vbx_setup for DiariZen's plda/ files. A 256-d embedding x becomes VBx
features with
y = sqrt(128) * l2(lda.T @ (sqrt(256) * l2(x - mean1)) - mean2) and
features = (y - plda_mu) @ plda_tr.T; plda_psi is VBx's Phi. DiariZen's
clustering settings for this checkpoint: AHC initialization at threshold 0.6,
VBx with Fa 0.07, Fb 0.8, LDA dimension 128, 20 iterations.
Accuracy
Measured with the diarizen-apple Swift runtime using these files, on an
Apple M2 Pro, against human references (pyannote/AMI-diarization-setup,
pyannote.metrics, collar 0 unless noted):
| Test | This conversion | Reference |
|---|---|---|
| AMI-SDM test set, 16 meetings, corpus DER | 13.76% | 13.9% published by DiariZen for this checkpoint |
| AMI EN2002a (35.7 min, 4 speakers), DER | 17.50% | base conversion: 21.16% |
| VoxConverse files with 12 to 21 speakers (31 files), collar 0.25 | 6.35% | base conversion: 7.24% |
Known divergence: this segmentation model does not reproduce PyTorch's tensors as closely as the base model does. About 29% of frames get a different argmax among the 16 powerset classes. A layer-by-layer comparison traced it to ordinary numerical differences between the Core ML runtime and PyTorch, not to a conversion error: the base model shows the same per-layer divergence rate, the correlation with PyTorch stays near 0.997 through the last WavLM layer, and a float32 export diverges the same way. The large model has twice the layers and larger late activations, which pushes more frames over the argmax flip point of a 16-class head. The downstream pipeline tolerates it: DER on AMI matches the published figure and improves on the base model. If you need tensor-level fidelity, use the base checkpoint. The embedding stage is the same ResNet34 as the base conversion (cosine 0.99991 against PyTorch).
Speed
On an Apple M2 Pro (32 GB), CPU and GPU, model loading excluded:
| Time | |
|---|---|
| Segmentation, one 16 s chunk (coremltools) | 74 ms |
| Segmentation, one 16 s chunk, base model for comparison | 28 ms |
A 16 s window every 1.6 s means about 37 segmentation calls per minute of audio, so segmentation costs roughly 2.8 s per minute here against 1 s for the base model. The embedding stage is identical to the base model's (the full base pipeline runs a 35.7 min meeting in 151 s). The full large pipeline was not timed separately.
Limitations
- Do not use the Neural Engine for the segmentation model. On the base model,
compute units
.alldeadlock the first prediction and.cpuAndNeuralEngineplaces no ops on the Neural Engine and runs about 12 times slower than the GPU; the large model has the same transformer structure and was only run on CPU and GPU. Use.cpuAndGPU. - Tensor-level divergence from PyTorch, described under Accuracy.
- Fixed shapes: one 16 s chunk per segmentation call. Pad the last chunk with zeros as pyannote does.
- Speed comes from one machine. Accuracy figures are the public AMI and VoxConverse sets above.
- The training data is far-field meeting and conversational audio (AMI, AISHELL-4, AliMeeting, NOTSOFAR-1, MSDWild, DIHARD3, RAMC, VoxConverse); other conditions were not measured here.
Converting it yourself
From a clone of diarizen-apple,
with its Python 3.11 environment (requirements.txt, torch 2.1.1,
coremltools 9.0, DiariZen installed from source):
python conversion/export_coreml.py --model BUT-FIT/diarizen-wavlm-large-s80-md-v2 --out out
python conversion/export_embedding_coreml.py --model BUT-FIT/diarizen-wavlm-large-s80-md-v2 --out out
python conversion/export_plda.py --model BUT-FIT/diarizen-wavlm-large-s80-md-v2 --out out/plda_transform.json
conversion/ here is a copy of those scripts (diarizen-apple commit
ca61fa0). The published packages were exported in September 2026 with an
earlier revision of the same scripts. plda_transform.json is reproduced byte
for byte by applying vbx_setup to the checkpoint's plda/plda.npz and
plda/xvec_transform.npz at revision 027b3e22 (the same files as the base
checkpoint's).
License and attribution
The repository as a whole is released under
CC BY-NC 4.0 (LICENSE):
non-commercial research and academic use only, with attribution.
Segmentation.mlpackage: converted from DiariZen'sBUT-FIT/diarizen-wavlm-large-s80-md-v2by BUT Speech@FIT, CC BY-NC 4.0 (some of its training sets are research-only). It contains a pruned WavLM Large (Microsoft) and pyannote.audio's powerset formulation.Embedding.mlpackage: WeSpeaker ResNet34-LM trained on VoxCeleb, as packaged by pyannote (pyannote/wespeaker-voxceleb-resnet34-LM), CC BY 4.0. It is the embedding DiariZen uses; the same file under its own terms is in nikioth/wespeaker-resnet34-lm-coreml.plda_transform.json: derived from the checkpoint'splda/files, a PLDA and x-vector transform trained by BUT Speech@FIT on VoxCeleb2, which BUT licenses under CC BY 4.0.conversion/: MIT (conversion/LICENSE).
Changes made by nikioth: the segmentation and embedding networks were
converted from PyTorch to Core ML ML Programs (float16, fixed 16 s chunk, two
export-only patches that are identity at inference, see
conversion/patches/README.md); the PLDA eigendecomposition was precomputed
into one JSON file. No weights were retrained. Provided as is, without
warranties of any kind.
Citation
@inproceedings{han2025leveraging,
title = {Leveraging self-supervised learning for speaker diarization},
author = {Han, Jiangyu and Landini, Federico and Rohdin, Johan and Silnova, Anna and
Diez, Mireia and Burget, Luk{\'a}{\v{s}}},
booktitle = {Proc. ICASSP},
year = {2025}
}
@article{han2025fine,
title = {Fine-tune Before Structured Pruning: Towards Compact and Accurate
Self-Supervised Models for Speaker Diarization},
author = {Han, Jiangyu and Landini, Federico and Rohdin, Johan and Silnova, Anna and
Diez, Mireia and Cernocky, Jan and Burget, Lukas},
journal = {arXiv preprint arXiv:2505.24111},
year = {2025}
}
@article{han2025efficient,
title = {Efficient and Generalizable Speaker Diarization via Structured Pruning of
Self-Supervised Models},
author = {Han, Jiangyu and P{\'a}lka, Petr and Delcroix, Marc and Landini, Federico and
Rohdin, Johan and Cernock{\`y}, Jan and Burget, Luk{\'a}{\v{s}}},
journal = {arXiv preprint arXiv:2506.18623},
year = {2025}
}
@article{chen2022wavlm,
title = {WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing},
author = {Chen, Sanyuan and Wang, Chengyi and Chen, Zhengyang and Wu, Yu and Liu, Shujie and
Chen, Zhuo and Li, Jinyu and Kanda, Naoyuki and Yoshioka, Takuya and Xiao, Xiong and
Wu, Jian and Zhou, Long and Ren, Shuo and Qian, Yanmin and Qian, Yao and Wu, Jian and
Zeng, Michael and Yu, Xiangzhan and Wei, Furu},
journal = {IEEE Journal of Selected Topics in Signal Processing},
volume = {16},
number = {6},
pages = {1505--1518},
year = {2022}
}
@inproceedings{plaquet2023powerset,
title = {Powerset multi-class cross entropy loss for neural speaker diarization},
author = {Plaquet, Alexis and Bredin, Herv{\'e}},
booktitle = {Proc. INTERSPEECH 2023},
year = {2023}
}
@inproceedings{bredin2023pyannote,
title = {pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe},
author = {Bredin, Herv{\'e}},
booktitle = {Proc. INTERSPEECH 2023},
pages = {1983--1987},
year = {2023}
}
@inproceedings{wang2023wespeaker,
title = {Wespeaker: A research and production oriented speaker embedding learning toolkit},
author = {Wang, Hongji and Liang, Chengdong and Wang, Shuai and Chen, Zhengyang and
Zhang, Binbin and Xiang, Xu and Deng, Yanlei and Qian, Yanmin},
booktitle = {ICASSP 2023 - IEEE International Conference on Acoustics, Speech and Signal Processing},
year = {2023}
}
@article{landini2022vbx,
title = {Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization:
theory, implementation and analysis on standard tasks},
author = {Landini, Federico and Profant, J{\'a}n and Diez, Mireia and Burget, Luk{\'a}{\v{s}}},
journal = {Computer Speech \& Language},
volume = {71},
pages = {101254},
year = {2022}
}
- Downloads last month
- -
Model tree for nikioth/diarizen-wavlm-large-s80-md-v2-coreml
Base model
BUT-FIT/diarizen-wavlm-large-s80-md-v2