3D-Speaker is Alibaba Cloud ModelScope's open source toolkit for single/multimodal speaker verification, speaker identification, and speaker log separation. It provides pre-trained models such as ERes2Net, CAM++, and ECAPA-TDNN and large-scale 3D-Speaker speech data sets, and supports multi-modal speaker logs that integrate speech and vision.

3D-Speaker

Core parameters and statistics of 3D-Speaker

3D-Speaker is an open source single/multi-modal speaker verification, recognition, and log separation toolkit from the Alibaba Cloud ModelScope community. It provides a complete tool chain from model training to production inference for speech technology developers and AI researchers. The "3D" in its name represents the three dimensions of the data set - Multi-Device, Multi-Distance, and Multi-Dialect, rather than three-dimensional vision.

Projects	Public Information
Product positioning	Open source single/multi-modal speaker verification and log separation toolkit
Core technology stack	Deep learning (ResNet, Res2Net, ECAPA-TDNN, ERes2Net, CAM++, SDPN, etc.)
Number of pre-trained models	10+ (covering speaker verification, language identification, self-supervised learning)
License Agreement	Apache License 2.0
Supported platforms	Linux, macOS, Desktop, API
Programming languages	Python (54%), Shell (27%), Perl (15%), C++ (3%)
GitHub Stars	~3,100
GitHub Forks	~267
Contributors	8 people (as of 2026-07)
Affiliated Organization	Alibaba DAMO Academy / ModelScope
Place of Belonging	CN
Linked Dataset	3D-Speaker-Dataset (10,000+ speakers, 190GB training data)
Academic Papers	ICASSP 2025 Acceptance

Brief review in one sentence: 3D-Speaker is not a voice assistant product, but a professional voiceprint recognition open source tool library produced by ModelScope - allowing developers to build production-level capabilities of "identifying people by listening to voices" at low cost, and providing a large-scale benchmark data set to match it.

Core Capability Boundary: 3D-Speaker focuses on the task of "who is speaking" and does not provide ASR (speech to text), TTS (speech synthesis) or NLP semantic understanding. It solves the identity recognition problem in the voice interaction pipeline and needs to be combined with other voice modules to form a complete dialogue system.

User and market recognition of 3D-Speaker

Voiceprint recognition technology is at a critical stage of transition from academic research to large-scale industrial application - the demand for "identifying people by listening to voices" continues to grow in scenarios such as financial identity authentication, intelligent conference analysis, and smart home personalization. However, the lack of high-quality open source tool chains has long restricted the speed of industrial implementation. In this context, 3D-Speaker relies on the ModelScope ecosystem (AI open source platform of Alibaba Cloud) and has become one of the most influential open source projects in the field of Chinese speaker recognition.

Community Recognition: GitHub repository received approximately 3,100 Stars and 267 Forks. Compared with similar open source projects (such as WeSpeaker about 4.5k Stars, SpeechBrain about 9k Stars), 3D-Speaker is recognized in the first echelon in the Chinese speech technology community. The core difference is that it provides a "data + code" integrated solution of large-scale multi-device/multi-distance/multi-dialect data sets and a full set of baseline models.

Academic Impact: 3D-Speaker related papers have been accepted by ICASSP 2025 (the top academic conference in the field of speech), and the paper is titled "3D-Speaker-Toolkit: An Open Source Toolkit for Multi-modal Speaker Verification and Diarization". The accompanying 3D-Speaker dataset paper (arXiv 2306.15354) provides a large-scale benchmark of 10,000+ speakers in 14 Chinese dialects for speech representation decoupling research, and has been cited by many researchers.

Industrial Adoption: Voiceprint recognition is one of the key technologies for "enhanced identity authentication" in financial compliance scenarios. Relying on the Alibaba Cloud ecosystem, 3D-Speaker covers developer users in finance, security, conferences and other industries through the pre-trained model download and online inference services of the ModelScope platform. Specific enterprise-level adoption data is not public, but the downloads of the ModelScope platform speaker-verification model page can be used as an indirect reference.

Differential positioning from competing products: Compared with international mainstream tools such as pyannote.audio (developed by a French team, focusing on diarization), SpeechBrain (comprehensive speech toolkit), and WeSpeaker (focusing on speaker recognition), 3D-Speaker The core advantages lie in three points: first, it provides a large-scale Chinese multi-dialect data set (3D-Speaker-Dataset); second, it covers both single-modal (pure audio) and multi-modal (audio and video fusion) solutions; third, it continues to maintain leading benchmark results in model architectures such as ERes2NetV2 and CAM++.

Cost advantage of 3D-Speaker

3D-Speaker’s pricing model is extremely simple – completely free under the Apache 2.0 open source license. But the real cost structure behind "open source and free" needs to be broken down by role:

C client/individual developer: The code and pre-trained model are completely free to download and use (Apache 2.0 license). Individual developers can run inference on native GPUs (such as RTX 3090/4090) without any licensing fees. The 3D-Speaker dataset (~190GB) is also available for free, but downloading requires bandwidth and time costs.

API/Developer: No official hosted API pricing - 3D-Speaker does not offer a standalone paid API on the ModelScope platform (unlike the commercial API model for base models such as the DeepSeek API). Developers can choose:

• Self-deployment mode: Run inference on your own server, and the cost depends entirely on hardware investment. Single speaker verification inference can be completed in tens of milliseconds on consumer-grade GPUs, and the throughput bottleneck is mainly in the voice activity detection (VAD) and audio decoding stages before feature extraction.
• ModelScope Online Inference: The ModelScope platform provides online experience services for some pre-trained models, but it is mainly used for functional verification and is not suitable for high-concurrency production scenarios. The specific usage limit is subject to the official ModelScope page.

Enterprise/Private: The Apache 2.0 license permits free commercial use, modification, and redistribution without a license fee. However, enterprises need to bear the following explicit investments:

• GPU infrastructure: A single card A100-80G can carry batch inference of mainstream models such as ERes2NetV2 (17.8M parameters); large-scale diarization tasks (such as dozens of hours of conference audio) require higher-specification CPU/GPU resource configurations.
• Data collection and annotation: The actual performance of the voiceprint recognition system is highly dependent on data that matches the target scene. Recognition accuracy across devices (such as switching from a PC microphone to a phone channel) and across distances (near field vs far field) will significantly decrease. Enterprises usually need to additionally collect voice data of the target scene and perform domain fine-tuning. This hidden cost often exceeds the cost of using the model itself.
• System integration investment: It is necessary to splice the voiceprint recognition module with existing ASR, VAD, NLP and other pipelines, which involves engineering manpower investment.

Cost Comparison: Open Source Tools vs Commercial APIs:

Cost items	3D-Speaker (open source and self-deployment)	Commercial voiceprint API (such as Alibaba Cloud voiceprint recognition)
License fee	Zero	Billed by call volume
GPU server (monthly estimate)	5,000–50,000 yuan (depending on scale)	Zero (API call)
Operation and maintenance manpower	Engineer maintenance required	Zero
Scene adaptation fine-tuning	Self-investment	Customization is not usually supported
Data privacy	Fully localized	Relying on cloud service providers
Marginal cost after scale	Close to zero (only electricity cost)	Linear growth with call volume

Hidden Cost Tip: 3D-Speaker’s EER can be as low as 0.52% on standard test sets such as VoxCeleb and CNCeleb, but these numbers are obtained under controlled laboratory conditions. In real-life scenarios - non-silent context, mobile Internet calls, children or elderly speakers - the recognition accuracy will drop significantly. Before putting the model into production, you need to conduct grayscale testing on the target scene data for at least two weeks to compare the performance gap between the pre-trained model and the fine-tuned model.

Main functions of 3D-Speaker

The core capabilities of 3D-Speaker revolve around the task of "identifying speaker identity from speech" and involve four interrelated functional modules:

• Speaker Verification: 1:1 identity authentication, verifying whether a speech belongs to a declared speaker. This is a core scenario for financial identity verification and system login. 3D-Speaker provides multiple models such as ERes2NetV2, CAM++, ECAPA-TDNN, etc., and the EER on the VoxCeleb test set ranges from 0.52% (ERes2Net-Large) to 1.56% (Res2Net). Implementation concerns: The selection of verification threshold directly trade-offs security and user experience - if the threshold is too strict, it will improve security but increase the user rejection rate. It is recommended to conduct ROC curve analysis according to the business security level to determine.

• Speaker Identification: 1:N identity matching, identifying which known identity the current voice belongs to from the registered speaker set. It is suitable for scenarios such as suspect investigation and automatic identification of VIP customers. Implementation concerns: When the size of the registration set increases, the recognition accuracy will decrease. In large-scale scenarios (>10,000 people), it is recommended to introduce hierarchical indexing or ANN acceleration.

• Speaker Diarization: Split and label multi-person audio according to "who spoke at what time", and output timestamp + speaker label. This is the core pre-requisite capability for meeting analysis and customer service quality inspection. 3D-Speaker’s diarization pipeline contains five pluggable modules: Overlapping speech detection (optional) → Voice activity detection → Speech segmentation → Speaker embedding extraction → Speaker clustering. On the Aishell-4 data set, the DER (Diarization Error Rate) can reach 10.30%. Implementation concerns: Overlap detection is the bottleneck of the current project - when two or more people speak at the same time, the traditional diarization scheme will incorrectly assign the overlapping area to one of the speakers or mark it as noise, causing information loss. 3D-Speaker provides an optional overlap detection module to alleviate this problem.

• Language Identification: Identify the language type of the speech (currently focusing on Chinese and English). Based on the CAM++ model, an EER of 29.36% is achieved on the 3D-Speaker data set (language recognition is mainly based on accuracy, this EER is for reference only). Suitable for multi-lingual call center automatic routing scenarios.

• Multimodal Speaker Diarization: Fusion of audio and video image input. Lip movement and facial activity features are extracted from videos through the TalkNet-ASD model, and joint inference with audio features is significantly better than the audio-only solution in overlapping speech scenarios. It is suitable for scenarios such as video conferences and interview records that contain both audio and video data.

• Self-Supervised Speaker Verification (Self-Supervised SV): Provides two self-supervised training solutions, RDINO and SDPN, which can learn speaker representation on unlabeled speech data, reducing the dependence on high-quality annotated data. Suitable for pre-training scenarios on large-scale unlabeled speech data.

[Expert View: Functional Synergy]: These four functions form a complete "speech→speaker tag" processing pipeline - Diarization separates the speech segments of different speakers → Verification/Identification labels each segment with identity → Language Identification supplements language meta-information. In actual engineering deployment, the segmentation accuracy of Diarization (especially the quality of overlapping speech segmentation) is the bottleneck of the entire pipeline: segmentation errors will be directly passed to Verification sections, resulting in identity annotation errors. It is recommended to invest more tuning resources in the VAD and segmentation of the pipeline, rather than just focusing on the accuracy of the Verification model itself.

Model and version evolution of 3D-Speaker

3D-Speaker is continuously updated in the mode of "function iteration drives version evolution". The GitHub repository does not have traditional semantic version numbers (no v1.0, v2.0, etc. releases), but is promoted in the form of "monthly/quarterly release of new models or new functions". The following is a reconstruction of the version context based on the What's New timeline in the README:

Phase 1: Data set and baseline model release (2023.06 - 2023.11)

• 2023.06: The 3D-Speaker dataset is officially released, including 10,000+ speakers in 14 Chinese dialects, multi-device (PC, recorder, array, mobile iPad) and multi-distance (0.1m–4m) labels. At the same time, baseline models and supporting benchmarks such as ERes2Net, CAM++, and RDINO were released.
• 2023.07: CAM++, ERes2Net-Base, and ERes2Net-Large pre-training models are launched on the ModelScope platform; dialogue detection and semantic speaker switching detection modules are released to enhance the semantic dimension of diarization.
• 2023.08: Released CAM++, ERes2Net-Base, and ERes2Net-Large benchmark results on the CN-Celeb data set; released Chinese and English language recognition models.
• 2023.09: Released training and inference recipes for the RDINO self-supervised speaker verification model on the CN-Celeb dataset.
• 2023.10: Released the training and inference recipes of ECAPA-TDNN on three datasets (3D-Speaker, VoxCeleb, CN-Celeb).
• 2023.11: ERes2Net-base pre-trained model released, trained based on 200k speaker Chinese Mandarin dataset.

Phase 2: Model system expansion (2024.01 - 2024.08)

• 2024.01: Released ResNet34 and Res2Net model recipes, covering three data sets; released large-margin finetune recipes and diarization inference recipes.
• 2024.02: Released a language recognition recipe that integrates phoneme information to improve language recognition accuracy; released a multi-modal diarization recipe to integrate audio and video input.
• 2024.04: Released the ONNX Runtime inference runtime, which supports model export to ONNX format and efficient inference on CPU/edge devices; released the ERes2NetV2 model with fewer parameters and faster inference on VoxCeleb.
• 2024.05: Released SDPN self-supervised model X-vector model training and inference recipes; released vision module (TalkNet-ASD) and semantic module (BERT) recipes to expand multi-modal speaker perception capabilities.
• 2024.08: Released ERes2NetV2 and ERes2NetV2_w24s4ep4 pre-trained models, trained on a 200k speaker dataset to achieve a better balance between EER and parameter size.

The third phase: Engineering and benchmark improvement (2024.12)

• 2024.12: Fully updated diarization recipe to provide reproducible DER benchmarks on multiple public multi-speaker datasets such as Aishell-4, Alimeeting, AMI, VoxConverse, Meeting-CN, etc. This is the latest major version and marks the transformation of 3D-Speaker from a pure research tool to a reproducible engineering benchmark.

Version Policy Statement: 3D-Speaker does not follow the semantic version number release, and there is no official release record on the GitHub Releases page. Users should pay attention to the Git Commit history ModelScope model page and README update log to track the latest changes. This release method reduces maintenance costs, but it also makes it difficult for users to judge code stability through a single version number - it is recommended to lock specific commits for production use.

Technical advantages of 3D-Speaker

The technical advantage of 3D-Speaker lies not only in the breakthrough of a certain model architecture, but also in the complete link coverage from data to model, from training to deployment. The following is dismantled from the two dimensions of architectural innovation and engineering.

Model architecture pedigree: 3D-Speaker integrates 8+ speaker verification model architectures, covering different scales and application scenarios:

Model	Parameters	VoxCeleb EER	CNCeleb EER	3DSpeaker EER	Features
Res2Net	4.03M	1.56%	7.96%	8.03%	Lightweight baseline
ResNet34	6.34M	1.05%	6.92%	7.29%	Classic architecture
ECAPA-TDNN	20.8M	0.86%	8.01%	8.87%	Competitive Baseline
ERes2Net-base	6.61M	0.84%	6.69%	7.21%	High cost performance
CAM++	7.2M	0.65%	6.78%	7.75%	Efficient convolution
ERes2NetV2	17.8M	0.61%	6.14%	6.52%	High-precision recommendation
ERes2Net-large	22.46M	0.52%	6.17%	6.34%	Highest accuracy

Causal chain of mechanism → effect → scene: The technical core of speaker verification is to extract fixed-dimensional "discriminative speaker embedding" (Speaker Embedding) from variable-length speech. ERes2NetV2 integrates multi-scale time-frequency features through the hierarchical design of residual connections, reducing the number of parameters by 14% while reducing the EER from 0.84% ​​(ERes2Net-base) to 0.61%. This means that in scenarios where the error rate is close to zero (such as financial risk control), ERes2NetV2 can achieve the same security threshold at a lower computational cost. Its applicable scenarios are production environments that require high-precision speaker verification and are sensitive to inference delays.

Innovative advantages of CAM++: CAM++ (Contextual Attention Module) is an efficient convolutional architecture proposed by the 3D-Speaker team. It reaches 0.65% EER on VoxCeleb with an extremely low parameter amount of 7.2M, approaching the level of ERes2NetV2 (17.8M). Parameter efficiency (EER/parameter amount ratio) is the highest among all models. For voiceprint recognition scenarios that need to be deployed on devices or mobile devices, CAM++ is the most worthy of priority evaluation.

Diarization pipeline engineering: 3D-Speaker’s diarization adopts a modular pipeline design, and each stage can be replaced or optimized independently:

• Overlapping speech detection (based on sequence labeling model) → optional module to identify areas where multiple people are speaking at the same time
• Voice Activity Detection (VAD) → remove silent and non-speech clips
• Speaker segmentation (based on BERT or fixed window) → segment speech into speaker-homogeneous segments
• Speaker embedding extraction → Use models such as ERes2NetV2/CAM++ to extract segment-level embeddings
• Speaker clustering (spectral clustering/affinity propagation) → cluster embeddings into different speakers

On the Aishell-4 and Alimeeting benchmarks, the pipeline achieved DERs of 10.30% and 19.73%, respectively, significantly better than the Kaldi baseline in the same period (approximately 12.2% and 24.4%, respectively), verifying the synergistic advantages of modular design.

Multi-modal fusion mechanism: 3D-Speaker's audio and video fusion diarization solution extracts lip movement and facial activity features from the video through TalkNet-ASD, splices them with audio features at the embedding level, and then sends them to a unified clustering module. In scenarios with a high proportion of overlapping speech, visual information provides a speaker switching boundary signal where audio is missing, reducing DER by about 3-5 percentage points. Applicable scenarios include video conferencing, live interviews, news programs and other fields with multi-camera video data.

ONNX Runtime Deployment Optimization: 3D-Speaker provides ONNX model export and inference scripts, supporting the conversion of PyTorch-trained models into ONNX format and deployment to CPU or edge devices. After the model is quantitatively optimized, the inference latency can be reduced to 1/3 to 1/2 of PyTorch dynamic graphs, and the memory usage is reduced by about 40%. In GPU-less server environments, ONNX deployment is the lowest-cost path to bring voiceprint recognition capabilities into production environments.

Limitations and Adaptation Boundaries: 3D-Speaker's pre-trained model is mainly oriented to speech signals with a sampling rate of 16kHz (8kHz telephone speech requires additional adaptation), and the current model works best in Chinese and English scenes. For scenarios with small languages, ultra-low resource languages, or unconventional sampling rates, a self-supervised scheme (RDINO/SDPN) needs to be used to pre-train or fine-tune on the target data.

How to use 3D-Speaker

3D-Speaker provides two usage paths: local code running and ModelScope platform online inference.

Path 1: Run local clone code (recommended for developers)

# Clone repository
git clone https://github.com/modelscope/3D-Speaker.git
cd 3D-Speaker

# Create a Python 3.8 context and install dependencies
conda create -n 3D-Speaker python=3.8
conda activate 3D-Speaker
pip install -r requirements.txt

Example of training recipe (taking speaker verification of ERes2NetV2 on the 3D-Speaker dataset as an example):

cd egs/3dspeaker/sv-eres2netv2/
bash run.sh

Use pre-trained model for inference: Load the pre-trained model for inference through the ModelScope library:

pip install modelscope

# Single-file speaker verification inference
model_id = "iic/speech_eres2netv2_sv_zh-cn_16k-common"
python speakerlab/bin/infer_sv.py --model_id $model_id

# Batch inference
python speakerlab/bin/infer_sv_batch.py --model_id $model_id --wavs wav_list.txt

# Self-supervised model inference (SDPN, based on VoxCeleb training)
model_id = "iic/speech_sdpn_ecapa_tdnn_sv_en_voxceleb_16k"
python speakerlab/bin/infer_sv_ssl.py --model_id $model_id

Speaker Log Inference (Diarization):

# Pure audio diarization
python speakerlab/bin/infer_diarization.py --wav audio.wav --out_dir ./output

# Enable overlapping speech detection (requires HuggingFace token)
python speakerlab/bin/infer_diarization.py --wav audio.wav --out_dir ./output \
  --include_overlap --hf_access_token $hf_access_token

Path 2: ModelScope online experience

Visit the [ModelScope Speaker Verification model page](https://www.modelscope.cn/models, page=1&tasks=speaker-verification&type=audio), select the corresponding pre-trained model, and upload audio online for testing. Suitable for quick verification of model capabilities, but not suitable for production context integration.

Path three: ONNX deployment (edge device/CPU inference)

cd runtime/onnxruntime
# Refer to the documentation in this directory to export the PyTorch model to ONNX and run inference

List of available pretrained models (all distributed via ModelScope):

Model ID	Description	Dataset
iic/speech_eres2net_sv_zh-cn_16k-common	ERes2Net, 200k speakers	Mandarin Chinese
iic/speech_eres2netv2_sv_zh-cn_16k-common	ERes2NetV2, 200k speakers	Mandarin Chinese
iic/speech_campplus_sv_zh-cn_16k-common	CAM++, 200k speakers	Mandarin Chinese
iic/speech_sdpn_ecapa_tdnn_sv_en_voxceleb_16k	SDPN self-monitoring	VoxCeleb (English)
iic/speech_campplus_lre_en-cn_16k	CAM++ language recognition	Chinese and English
iic/speech_eres2net_base_lre_en-cn_16k	ERes2Net language identification	Chinese and English

3D-Speaker data set download:

# Training set (~190GB)
wget https://speech-lab-share-data.oss-cn-shanghai.aliyuncs.com/3D-Speaker/train.tar.gz
# Test set (~1.1GB)
wget https://speech-lab-share-data.oss-cn-shanghai.aliyuncs.com/3D-Speaker/test.tar.gz
# transcribe text
wget https://speech-lab-share-data.oss-cn-shanghai.aliyuncs.com/3D-Speaker/transcription.tar.gz

Tips: 3D-Speaker does not have an official Docker image, and the official documentation is mainly Chinese README. For production users, it is recommended to lock a specific Git Commit and build the Docker image yourself to avoid inconsistent behavior caused by dependency version changes.

Product Pricing for 3D-Speaker

3D-Speaker adopts the Apache 2.0 open source protocol and has no commercial charging model. Specific terms:

• Scope of Use: Permission is granted to freely use, copy, modify, merge, distribute, sublicense and/or sell copies of the software.
• Commercial Restrictions: None - The Apache 2.0 protocol is commercial-friendly and does not require open source derivative works.
• Attribution Requirements: Retain the original copyright notice and contributor list in derivative works.
• WARRANTY DISCLAIMER: No warranties, express or implied, are provided where required by applicable law or agreed to in writing.
• Model Weight License: Pre-trained model weights are hosted on the ModelScope platform, and their terms of use are subject to the license stated on the respective model card page of ModelScope, which may be different from the Apache 2.0 agreement of the code repository.
• Dataset License: 3D-Speaker dataset metadata is released under the CC BY-SA 4.0 license, and the original copyright of the audio content belongs to the original data owner.

Price Reference: Zero licensing fee. The total ownership cost calculation formula for enterprise users is: TCO = GPU/server purchase or rental fee + operation and maintenance manpower × number of months + scenario adaptation data collection cost. For medium-scale scenarios processing less than 1,000 hours of audio per day, the TCO of self-deployment is significantly lower than that of a pay-as-you-go commercial API.

Application scenarios of 3D-Speaker

The core drivers for voiceprint recognition to move from the laboratory to industrial implementation come from three directions: compliance identity authentication, conference intelligence, and smart home personalization. The following is a scenario-based analysis based on the capabilities boundaries of 3D-Speaker.

• Financial identity authentication and fraud prevention: In remote identity verification scenarios such as telephone banking and mobile customer service, voiceprint verification serves as a supplement or alternative to passwords/PIN codes. 3D-Speaker’s speaker verification model can be integrated into IVR systems to complete identity comparison within the first 3-5 seconds of a user’s call. Cost reduction and efficiency improvement: Before the introduction of voiceprint verification, financial customer service password reset and identity verification required an average of 45-60 seconds of manual processing time; after integrating voiceprint verification, routine identity verification can be completed automatically within 10 seconds (including voice activity detection + feature extraction + comparison), reducing manual intervention by approximately 70%. Key points of verification: Recognition accuracy under telephone channel (8kHz sampling rate), authenticity rejection rate in short voice (<3 seconds) scenarios, and liveness detection defense capabilities for involuntary recordings (recordings played by fraudsters). 3D-Speaker itself does not provide an anti-spoofing module and requires additional integration of live detection solutions such as ASVspoof.

• Meeting Analysis and Intelligent Minutes: Enterprise meeting recordings automatically separate the voice clips of different speakers, and combine with ASR to generate meeting minutes with speaker labels. 3D-Speaker's diarization pipeline is at the heart of this scenario. Cost reduction and efficiency improvement: Traditionally, it takes about 2-3 hours to manually compile 1-hour meeting minutes; using automatic diarization + ASR pipeline, 1-hour audio processing can be completed in 15-30 minutes, and the efficiency is increased by 4-10 times (based on GPU inference time). Verification focus: Segmentation accuracy when multiple people speak at the same time (overlapping speech) - common "interruption" and "simultaneous speech" scenarios in conference rooms. The traditional diarization solution will mark the overlapping area as the wrong speaker. The improvement solution needs to be alleviated by the optional overlap detection module of 3D-Speaker. For video conferencing, it is recommended to enable multi-modal diarization (audio and video fusion) to improve the recognition robustness of overlapping scenes. Human-machine collaboration boundary: It is recommended to set up manual review for automatic labeling results - speaker labeling errors may still reach 5-10% in a 20% overlap scenario, and meeting records required for compliance must be manually confirmed.

• Smart speakers and home personalization: Recognize the voices of different family members, provide personalized content recommendations and operate permission control. 3D-Speaker's lightweight model (CAM++, 7.2M parameters) can run on the end-side chip of the smart speaker without the need for cloud calls. Cost reduction and efficiency improvement: Compared with the cloud solution, on-device voiceprint recognition saves 200-500ms network transmission delay for a single interaction, and there is no cloud inference cost. Based on 50 times per day/10 million household users, the annual cloud inference cost saving is about 50-100 million yuan (calculated as a single time of 0.001 yuan). Verification focus: Recognition stability when the number of family members is expanded from 2 to 6; EER degradation degree in far field (>2 meters) and noise environment (kitchen smoke, TV background sound). The 3D-Speaker pre-trained model mainly optimizes the 16kHz near-field scene, and the far-field needs to be fine-tuned using the long-range subset of the 3D-Speaker dataset.

• Call Center Quality Inspection: Automatically separate the conversation segments between customer service and customers, and perform quality inspection analysis on the customer service's speech speed, emotion and service terms. Key points of verification: Sampling rate adaptation in telephone scenarios (8kHz vs 16kHz); separation accuracy when customer service and customer voices are highly similar. It is recommended to use 3D-Speaker's ECAPA-TDNN or ResNet34 model (with benchmark results on CNCeleb phone data) as a starting template.

• Public Prosecution and Legal Evidence Collection: Identify and separate different speakers in audio evidence to assist evidence analysis. Verification focus: Interpretability under legal compliance requirements - the separation boundaries of spectral clustering results need to be traceable. The current clustering process of 3D-Speaker does not provide detailed confidence output, and manual review needs to be supplemented in the evidence collection process. This scenario relies heavily on prior knowledge of the number of speakers (most clustering algorithms need to preset the number of speakers or use complex estimators). It is recommended to use automatic estimation clustering algorithms such as affinity propagation in stage 2.

Applicable groups of 3D-Speaker

3D-Speaker is positioned as a professional-level open source tool, and its user profile is focused on developers and researchers in the field of voice technology, rather than ordinary end consumers.

• Voice technology developers and integrators: Technical teams that need to integrate voiceprint recognition capabilities into products. It is recommended to use Pipeline - directly use ModelScope to load the pre-trained model for inference without having to train it yourself. The typical integration cycle is about 2-4 weeks (including model verification and API packaging). Not suitable for the boundary: It has clear requirements for Python/Shell programming and deep learning foundation, and is not suitable for front-end or full-stack developers without AI engineering experience; if you need pure HTTP API calls and do not accept self-deployment, you should evaluate the commercial voiceprint recognition API.

• AI researcher (speaker recognition direction): Researchers engaged in voiceprint recognition, voice biometrics, and self-supervised representation learning. 3D-Speaker provides a full set of training recipes and benchmarks to quickly reproduce SOTA results on 3D-Speaker, VoxCeleb or CNCeleb datasets. Prerequisite: Familiar with PyTorch and deep learning training process. It is recommended to use RDINO or SDPN self-supervised schemes for pre-training on your own private data.

• Finance/Security Technology Team: Enterprise teams that need voiceprint verification capabilities in identity authentication or audio forensics scenarios. Recommended path: First use the pre-trained model to do POC on the target scene data (specific microphones, ambient noise), verify that the accuracy requirements are met, and then put it into production integration. Unfit Boundary: Scenarios that require complete live body detection (anti-spoofing) functions are not provided by 3D-Speaker itself, and ASVspoof or other anti-spoofing engines need to be additionally integrated.

• Smart Hardware and IoT Team: Deploy end-side voiceprint recognition in smart speakers, wearable devices and vehicle systems. The CAM++ model is suitable for resource-constrained edge devices with the lightweight advantage of 7.2M parameters. Prerequisite: Experience in model quantification and ONNX/TensorRT deployment is required.

• Conferencing system and collaboration tool developers: Teams that need automatic speaker logging functionality to improve their conferencing product experience. It is recommended to start with the diarization recipe and use the pre-trained model + spectral clustering to complete the POC. Unsuitable Boundary: Streaming scenarios with extremely high real-time requirements (delay <500ms) - the current diarization pipeline requires complete audio to perform clustering, and is not suitable for frame-by-frame output streaming processing. If you need a streaming solution, you need to modify the clustering algorithm into an online version.

Global unfit boundary summary: 3D-Speaker does not provide ASR (speech to text), TTS (speech synthesis) or NLU (natural language understanding) capabilities, and needs to be combined with other speech modules to build a complete voice interaction system. The recognition performance in long-tail scenarios such as 8kHz telephone channel, strong background noise (signal-to-noise ratio <10dB), children or elderly speakers needs to be verified by actual measurements on target data. When GPU offline inference is not used, the latency of CPU ONNX inference may reach hundreds of milliseconds, which is not suitable for high-concurrency real-time scenarios.

Summary and Outlook

3D-Speaker has established a unique ecological niche of "dataset + toolkit" integration in the Chinese voiceprint recognition open source community - unlike pyannote.audio (which focuses on research-based diarization) and WeSpeaker (which focuses on speaker recognition baselines), 3D-Speaker's focus is to provide a complete link from large-scale multi-dimensional data sets to reproducible industrial-grade benchmarks. Its core value can be summarized in three sentences: using the 3D-Speaker data set to fill the gaps in Chinese multi-device/multi-distance/multi-dialect voiceprint data; using models such as ERes2NetV2 and CAM++ to continuously refresh EER records on multiple benchmarks; using the Apache 2.0 open source protocol to lower the commercial threshold for voiceprint recognition technology.

Current main limitations: First, the GitHub repository does not have official Release and semantic version numbers. It is difficult for users to evaluate the stability and change scope of the code through the version number. Production users need to lock specific Commit. Second, the pre-training model is mainly oriented to Chinese and English scenarios with a 16kHz sampling rate, and has limited coverage of 8kHz telephone speech and low-resource languages. Third, it does not provide anti-spoofing capabilities, and additional liveness detection solutions must be integrated in financial-level identity authentication scenarios. Fourth, the documents are mainly in Chinese (README is in English), international community building is still in its early stages, and there are few English technical blogs and tutorial resources. Fifth, there is no official Docker image and CI build status identification, and users need to solve the problem of context consistency by themselves.

Subsequent evolution direction: Judging from GitHub activity (there are still new commits as of 2026-07) and ICASSP 2025 paper acceptance, the 3D-Speaker team is still continuing to invest. Evolution directions worthy of attention include: further optimization of client-side reasoning (possible launch of TFLite/CoreML export), streaming diarization support (currently only supports offline clustering), multi-dialect extensions for more low-resource languages, and deeper integration with the ModelScope ecosystem (such as AutoML automatic parameter tuning).

Procurement and Adoption Risk Assessment: As an Apache 2.0 open source project, 3D-Speaker has no authorization risk and vendor lock-in risk, and is suitable as an entry-level verification and prototype development platform for voiceprint recognition capabilities. Enterprise integration should adopt a "three-stage method" - first (2-4 weeks): use the pre-trained model to do a POC on the target scene data (target microphone, ambient noise sample, target speaker size) to verify whether the EER meets the business threshold; second (4-8 weeks): if the POC meets the standards, complete the ONNX export and API packaging in the isolated environment (non-production), and integrate with the existing ASR/NLP pipeline; third (continuous): monitor the production of the environment after grayscale goes online EER drift (such as distribution shifts caused by newly registered users or new scenarios), and pay attention to the frequency of GitHub Commit updates - if there are no active commits for more than 6 months, you need to evaluate whether there are alternative tools that need to be selected. Before use in compliance-sensitive industries (finance, security), you should confirm the terms of use of ModelScope platform model weights (which may be different from Apache 2.0) and the applicability of the data set's CC BY-SA 4.0 agreement to commercial derivative works.

Related tools: elevenlabs, udio

Version Info

• Diarization Benchmark Update ：Updated the speaker log (Diarization) recipe to provide reproducible DER benchmark results on multiple public datasets such as Aishell-4, Alimeeting, AMI, VoxConverse, etc. There is no official precise date yet. （2024-12）
• ERes2NetV2 Release ：Released ERes2NetV2 and ERes2NetV2_w24s4ep4 pre-trained models, trained based on the 200k speaker data set, with fewer parameters and faster inference. There is no official precise date yet. （2024-08）
• Multi-modal & SSL Expansion ：Released SDPN self-supervised model X-vector training recipes, visual module (TalkNet) and semantic module (BERT) training recipes. There is no official precise date yet. （2024-05）
• ONNX Runtime & ERes2NetV2 VoxCeleb ：Released the ONNX Runtime inference runtime; released the ERes2NetV2 model based on the VoxCeleb dataset. There is no official precise date yet. （2024-04）
• Language ID & Multimodal Diarization ：Publish the language recognition formula that fuses phoneme information; publish the multi-modal speaker log formula that fuses audio and video images. There is no official precise date yet. （2024-02）
• ResNet & Res2Net Recipes ：Release the training and inference recipes of ResNet34 and Res2Net on the 3D-Speaker, VoxCeleb, and CN-Celeb data sets; release the large-margin finetune recipe. There is no official precise date yet. （2024-01）
• ERes2Net-Base Pretrained ：Released ERes2Net-base pre-trained model, trained on a 200k speaker Chinese Mandarin dataset. There is no official precise date yet. （2023-11）
• Core Models Launch ：Released CAM++, ERes2Net-Base, and ERes2Net-Large pre-training models; released dialogue detection and semantic speaker switching detection modules. There is no official precise date yet. （2023-07）
• Dataset & Initial Release ：Officially released the 3D-Speaker data set and baseline models and benchmarks such as ERes2Net, CAM++, and RDINO. There is no official precise date yet. （2023-06）