File size: 2,409 Bytes
7963518 71ebaeb 7963518 1d54f36 7963518 55b16df 7963518 e50af93 7963518 71ebaeb 7963518 8097bde 7963518 be01b00 7963518 be01b00 7963518 71ebaeb 1d54f36 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | ---
license: apache-2.0
library_name: pytorch
pipeline_tag: audio-to-audio
tags:
- audio-visual
- target-speaker-extraction
- speech-separation
- streaming
datasets:
- JusperLee/RealSSA-Real
- JusperLee/RealSSA-Sim
---
<p align="center"><img src="https://huggingface.co/JusperLee/Falcon/resolve/main/logo.png" alt="Falcon logo" width="120"></p>
# Falcon
Causal audio-visual target speaker extraction model from the paper: a single-pass multi-scale separator trained with an encoder-space speaker contrastive loss (0.5 M trainable parameters).
Code, training and evaluation: <https://github.com/JusperLee/Falcon>. Paper: [*Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes*](https://arxiv.org/abs/2610.32749) (NeurIPS 2026), Wendi Sang\*, Kai Li\*, Yifan Li, Jianqiang Huang, Xiaolin Hu (\*equal contribution).
| File | Config in the code repo | Audio network | Total params |
|---|---|---|---|
| `falcon.pt` | `configs/Falcon.yml` | `FalconSpeakerContrastive` | 11.72 M |
Total parameters include the frozen 11.2 M-parameter FRCNN lip-reading encoder. The file holds only `state_dict` (`audio_model.*`, `video_model.*`), the architecture name and the training epoch; load it with `torch.load(..., weights_only=True)`. `config.yaml` is a copy of `configs/Falcon.yml` from the code repo; the Hub counts downloads by this file, and `FalconRuntime` requests it on every load.
## Usage
```bash
git clone https://github.com/JusperLee/Falcon.git
cd Falcon/studio && pip install -r requirements.txt
```
```python
import numpy as np
from falcon_runtime import FalconRuntime
model = FalconRuntime.from_pretrained("JusperLee/Falcon") # downloads falcon.pt on first use
audio = np.zeros(32000, dtype=np.float32) # 2 s mono 16 kHz mixture
mouth = np.zeros((50, 96, 96), dtype=np.uint8) # 25 fps grayscale mouth crops of the target speaker
target = model.infer(audio, mouth) # (32000,) target speech
```
Whole clips: `python infer_file.py --audio mix.wav --mouth mouth.npy --out target.wav --model JusperLee/Falcon`. Live separation in the browser and in Zoom / Tencent Meeting: see `studio/README.md` in the code repo.
Input: mono 16 kHz audio and synchronized 25 fps grayscale 96×96 mouth crops (center-cropped to 88×88 and normalized inside the runtime).
## License
Apache License 2.0, the same as the code repository. |