NeMo Audio 配置文件完全指南数据集、Lhotse 加载器、模型架构与微调实战【免费下载链接】SpeechA scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)项目地址: https://gitcode.com/GitHub_Trending/nem/Speech本指南以 NeMo Speech 仓库中 docs/source/audio/configs.rst 为核心系统讲解音频Audio集合模型配置文件的组织方式与全部核心参数从经典的 NeMo manifest 数据集配置到 Lhotse CutSet/Shar 高性能数据加载再到支持温度重加权的多数据集混合采样、模型架构声明与三种微调启动方式。读完本文你将能够独立读懂并编写一套可运行的 NeMo Audio 训练配置并能用命令行覆盖任意参数完成从零训练与基于预训练模型的微调。一、配置文件整体结构NeMo Audio 集合的配置文件采用 Hydra OmegaConf 体系组织。每个配置文件通常由name、model、trainer、exp_manager四大部分组成其中model段是核心一般需要包含数据集信息通过model.train_ds、model.validation_ds、model.test_ds小节声明训练、验证、测试数据增强参数在线增强如 RIR 卷积、加性噪声的开关与数据路径模型架构规格通过_target_指向具体的类实现如nemo.collections.audio.modules.transforms.AudioToSpectrogram声明编码器、解码器、估计器/掩码网络、损失函数与评估指标。关于 Experiment Manager 与 PyTorch Lightning trainer 等所有 NeMo 模型通用的参数参见 核心配置文档。所有 NeMo Audio 模型的示例配置文件均可在 examples/audio/conf 目录中找到仓库当前包含 15 个覆盖不同任务的配置masking.yaml、masking_with_online_augmentation.yaml、predictive.yaml、predictive_conformer.yaml、flow_matching_generative.yaml、flow_matching_generative_ssl_pretraining.yaml、score_based_generative.yaml、schroedinger_bridge.yaml、beamforming.yaml、maxine_bnr.yaml等。训练入口脚本为 examples/audio/audio_to_audio_train.py它通过hydra_runner(config_path./conf, config_namemasking)加载配置再根据model.type字段分发到对应的模型类mask_based→EncMaskDecAudioToAudioModelpredictive→PredictiveAudioToAudioModelscore_based→ScoreBasedGenerativeAudioToAudioModelschroedinger_bridge→SchroedingerBridgeAudioToAudioModelflow_matching→FlowMatchingAudioToAudioModelbnr→BNR2随后调用model.maybe_init_from_pretrained_checkpoint(cfg)加载预训练权重并执行trainer.fit(model)。二、NeMo Dataset 配置manifest 方式训练、验证、测试参数分别通过model.train_ds、model.validation_ds、model.test_ds小节指定。根据任务不同可能包含加载音频的采样率或时长等参数部分字段可以留空在运行时通过命令行指定。数据集处理类的完整参数列表参见 API 文档的数据集章节。一个典型的 train / validation / test 数据集配置示例如下对应 examples/audio/conf/masking.yamlmodel: sample_rate: 16000 skip_nan_grad: false train_ds: manifest_filepath: ??? input_key: audio_filepath # key of the input signal path in the manifest target_key: target_filepath # key of the target signal path in the manifest target_channel_selector: 0 # target signal is the first channel from files in target_key audio_duration: 4.0 # in seconds, audio segment duration for training random_offset: true # if the file is longer than audio_duration, use random offset to select a subsegment min_duration: ${model.train_ds.audio_duration} batch_size: 64 # batch size may be increased based on the available memory shuffle: true num_workers: 8 pin_memory: true validation_ds: manifest_filepath: ??? input_key: audio_filepath # key of the input signal path in the manifest target_key: target_filepath # key of the target signal path in the manifest target_channel_selector: 0 # target signal is the first channel from files in target_key batch_size: 64 # batch size may be increased based on the available memory shuffle: false num_workers: 4 pin_memory: true test_ds: manifest_filepath: ??? input_key: audio_filepath # key of the input signal path in the manifest target_key: target_filepath # key of the target signal path in the manifest target_channel_selector: 0 # target signal is the first channel from files in target_key batch_size: 1 # batch size may be increased based on the available memory shuffle: false num_workers: 4 pin_memory: true关键参数说明参数作用manifest_filepath指向 JSON Lines 格式的 manifest 文件???表示必填必须通过命令行或配置提供input_keymanifest 中输入信号路径对应的键如audio_filepath含噪/含混响信号target_keymanifest 中目标信号路径对应的键如target_filepath干净信号target_channel_selector当目标文件为多通道时选择第几通道作为目标0 表示第一通道audio_duration训练音频段时长秒。注意它与 STFT 参数联动例如hop_length256时audio_duration4.0对应的 STFT 时间帧数为1 4.0 // 0.256 16若想得到 256 帧则需audio_duration 1 256 * 0.256 ≈ 65.5实际配置中masking_with_online_augmentation.yaml使用truncate_duration: 4.0hop_length: 256并注释Number of STFT time frames 1 truncate_duration // encoder.hop_length 256这里的//是整除关系配置时应以实际帧数为准random_offset文件长于audio_duration时用随机偏移选取子片段训练min_duration过滤掉短于该时长的样本常用${model.train_ds.audio_duration}引用保持一致性batch_size批大小可按显存上调shuffle是否打乱数据训练 true、验证/测试 falsenum_workersDataLoader 工作进程数pin_memory是否固定内存加速 GPU 传输三、Lhotse Dataset 配置3.1 Lhotse CutSet启用 Lhotse 数据加载器后训练数据以 CutSet 清单形式组织。目标信号需要存放在自定义的target_recording字段中train_ds: use_lhotse: true # enable Lhotse data loader cuts_path: ??? # path to Lhotse cuts manifest with input signals and the corresponding target signals (target signals should be in the custom target_recording field) truncate_duration: 4.00 # truncate audio to 4 seconds truncate_offset_type: random # if the file is longer than truncate_duration, use random offset to select a subsegment batch_size: 64 # batch size may be increased based on the available memory shuffle: true num_workers: 8 pin_memory: true与 manifest 方式相比Lhotse 使用truncate_durationtruncate_offset_type组合替代audio_durationrandom_offsettruncate_duration将音频截断到固定时长truncate_offset_type: random在文件过长时随机选取子片段。3.2 Lhotse CutSet 在线增强在线增强通过 RIR房间冲激响应卷积与加性噪声实现需要在训练集配置中额外开启rir_enabled并提供 RIR 与噪声数据的 Lhotse 清单路径。完整示例见 examples/audio/conf/masking_with_online_augmentation.yamltrain_ds: use_lhotse: true # enable Lhotse data loader cuts_path: ??? # path to Lhotse cuts manifest with speech signals for augmentation (including custom target_recording field with the same signals) truncate_duration: 4.00 # truncate audio to 4 seconds truncate_offset_type: random # if the file is longer than truncate_duration, use random offset to select a subsegment batch_size: 64 # batch size may be increased based on the available memory shuffle: true num_workers: 8 pin_memory: true rir_enabled: true # enable room impulse response augmentation rir_path: ??? # path to Lhotse recordings manifest with room impulse response signals noise_path: ??? # path to Lhotse cuts manifest with noise signals注意在线增强模式下cuts_path提供的是用于增强的干净语音其target_recording字段与自身相同作为干净目标而validation_ds/test_ds的cuts_path则应直接提供含噪语音与干净目标的配对数据。增强相关的在线增强教程可参考 Speech_Enhancement_with_Online_Augmentation.ipynb。3.3 Lhotse Shar 格式Shar 是 Lhotse 的高性能打包存储格式适合海量数据与分布式训练。其配置只需要shar_path指向 shar 目录无需cuts_pathtrain_ds: shar_path: ??? use_lhotse: true truncate_duration: 4.00 # truncate audio to 4 seconds truncate_offset_type: random batch_size: 8 # batch size may be increased based on the available memory shuffle: true num_workers: 8 pin_memory: true一个完整的 Shar 配置示例见 examples/audio/conf/flow_matching_generative_ssl_pretraining.yaml该文件同时演示了 Shar 训练集与 manifest 验证集的混合用法。由于 Shar 数据量通常极大其trainer段需要显式设置max_steps如 10000与limit_train_batches如 1000来控制每个伪 epoch 的训练量并设置use_distributed_sampler: falseLhotse 数据加载所必需。3.4 多数据集温度重加权Dataset Reweighting with Temperature当通过嵌套的input_cfg组合多个数据集时可以用reweight_temperature控制采样分布。该特性允许你在增删数据集时无需手工重算权重即可平衡各数据集的采样比例。温度缩放公式为$$\hat{w}i \frac{w_i^{\tau}}{\sum{j} w_j^{\tau}}$$其中 $w_i$ 是数据集 $i$ 的原始权重$\tau$ 是温度$\hat{w}_i$ 是归一化后的采样概率。温度的作用机制temperature 1.0保持原始权重比例中性不重加权temperature 0.0所有数据集等概率采样与原始权重无关0 temperature 1.0相对大数据集更多采样小数据集temperature 1.0放大数据集权重之间的差异。两种配置格式标量值应用于所有嵌套层级会打印警告日志train_ds: use_lhotse: true reweight_temperature: 0.5 # Applied to all levels, warning logged input_cfg: - type: group input_cfg: - type: lhotse_shar shar_path: /path/to/dataset1 weight: 900 - type: lhotse_shar shar_path: /path/to/dataset2 weight: 100 - type: lhotse_shar shar_path: /path/to/dataset3 weight: 200 - type: nemo_tarred manifest_filepath: /path/to/dataset4/manifest.json tarred_audio_filepath: /path/to/dataset4/audio.tar weight: 300列表格式与最大嵌套深度一一对应每层一个温度train_ds: use_lhotse: true reweight_temperature: [1.0, 0.0] # Level 1: preserve ratios, Level 2: equalize input_cfg: - type: group weight: 0.7 input_cfg: - type: lhotse_shar shar_path: /path/to/dataset1 weight: 600 - type: lhotse_shar shar_path: /path/to/dataset2 weight: 400 - type: group weight: 0.3 input_cfg: - type: lhotse_shar shar_path: /path/to/dataset3 weight: 100注意若reweight_temperature以列表形式提供其长度必须与input_cfg的最大嵌套深度完全一致过多或过少都会抛出ValueError。若希望所有层级使用相同温度请使用标量值。最大嵌套深度计算最大嵌套深度即配置中input_cfg键的最大嵌套层数同一层的兄弟 group 共享同一个温度值。例如# This has maximum nesting depth 2 input_cfg: # Level 1 - type: group input_cfg: # Level 2 - type: lhotse_shar - type: group # Same level as above (sibling) input_cfg: # Level 2 (same as above) - type: lhotse_shar当input_cfg通过 CLI 覆盖为 YAML 文件路径时如model.train_ds.input_cfgtrain_all.yaml深度计算会加载被引用文件并遍历其内容统计嵌套的input_cfg键支持多级文件引用# train_all.yaml (referenced via input_cfgtrain_all.yaml) - type: group weight: 100 input_cfg: ${oc.env:MANIFEST_ROOT}/train_en.yaml # resolved at runtime - type: group weight: 200 input_cfg: ${oc.env:MANIFEST_ROOT}/train_de.yaml注意包含 OmegaConf 插值的路径如${oc.env:MANIFEST_ROOT}在深度统计阶段无法解析——它们会在运行时由OmegaConf.create()才被解析此类路径被当作单个额外嵌套层级处理。实战示例平衡多任务数据组train_ds: use_lhotse: true reweight_temperature: [1.0, 0.0] # Level 1: Preserve task ratios, Level 2: Equalize within tasks input_cfg: - type: group weight: 0.7 tags: task: asr input_cfg: - type: nemo_tarred manifest_filepath: /path/to/asr1/manifest.json tarred_audio_filepath: /path/to/asr1/audio.tar weight: 600 # Large dataset - type: nemo_tarred manifest_filepath: /path/to/asr2/manifest.json tarred_audio_filepath: /path/to/asr2/audio.tar weight: 100 # Small dataset (will be upsampled with temp0.0) - type: group weight: 0.3 tags: task: ast input_cfg: - type: nemo_tarred manifest_filepath: /path/to/ast1/manifest.json tarred_audio_filepath: /path/to/ast1/audio.tar weight: 50 - type: nemo_tarred manifest_filepath: /path/to/ast2/manifest.json tarred_audio_filepath: /path/to/ast2/audio.tar weight: 200该示例的效果Level 1 温度为1.0ASR 组与 AST 组之间保持 70/30 的原始比例Level 2 温度为0.0每组内部所有数据集等概率采样无论其原始权重如何小数据集被上采样。从源码实现看reweight_temperature参数定义于 nemo/collections/common/data/lhotse/dataloader.pyreweight_temperature: Any None支持 float / int / list 混合格式温度重加权逻辑位于 nemo/collections/common/data/lhotse/cutset.py对应的深度计算与行为验证测试见 tests/collections/common/test_lhotse_temperature_reweighting.py 与 tests/collections/common/test_lhotse_dataloading.py。input_cfg还支持type: nemo_tarredNeMo tarred 音频数据集与type: lhotse_shar两种底层数据集类型tags字段可用于给数据集打标签。四、模型架构配置每个配置文件都应声明实验所用的模型架构。下面是一个简单预测式predictive模型的完整示例对应 examples/audio/conf/predictive.yamlmodel: type: predictive sample_rate: 16000 skip_nan_grad: false num_outputs: 1 normalize_input: true # normalize the input signal to 0dBFS train_ds: manifest_filepath: ??? input_key: noisy_filepath target_key: clean_filepath audio_duration: 2.00 # trim audio to 2 seconds random_offset: true normalization_signal: input_signal batch_size: 8 # batch size may be increased based on the available memory shuffle: true num_workers: 8 pin_memory: true validation_ds: manifest_filepath: ??? input_key: noisy_filepath target_key: clean_filepath batch_size: 8 shuffle: false num_workers: 4 pin_memory: true encoder: _target_: nemo.collections.audio.modules.transforms.AudioToSpectrogram fft_length: 510 # Number of subbands in the STFT fft_length // 2 1 256 hop_length: 128 magnitude_power: 0.5 scale: 0.33 decoder: _target_: nemo.collections.audio.modules.transforms.SpectrogramToAudio fft_length: ${model.encoder.fft_length} hop_length: ${model.encoder.hop_length} magnitude_power: ${model.encoder.magnitude_power} scale: ${model.encoder.scale} estimator: _target_: nemo.collections.audio.parts.submodules.ncsnpp.SpectrogramNoiseConditionalScoreNetworkPlusPlus in_channels: 1 # single-channel noisy input out_channels: 1 # single-channel estimate num_res_blocks: 3 # increased number of res blocks pad_time_to: 64 # pad to 64 frames for the time dimension pad_dimension_to: 0 # no padding in the frequency dimension loss: _target_: nemo.collections.audio.losses.MSELoss # computed in the time domain metrics: val: sisdr: # output SI-SDR _target_: torchmetrics.audio.ScaleInvariantSignalDistortionRatio optim: name: adam lr: 1e-4 # optimizer arguments betas: [0.9, 0.999] weight_decay: 0.0架构配置的要点type字段predictive对应PredictiveAudioToAudioModel训练脚本据此选择模型类不写该字段时默认使用mask_basedEncMaskDecAudioToAudioModel并打印警告。encoder/decoder通过_target_指向nemo.collections.audio.modules.transforms中的 STFT 变换类。注意fft_length510时子带数为510 // 2 1 256decoder通过${model.encoder.fft_length}等 OmegaConf 插值自动与encoder保持一致避免参数漂移。estimator预测式模型在此使用 NCSNNoise Conditional Score Network Plus Plus网络in_channels/out_channels对应输入输出通道数num_res_blocks控制残差块数量pad_time_to/pad_dimension_to控制时间/频率维度的 padding。lossnemo.collections.audio.losses.MSELoss在时域计算即预测目标为时域波形。metricstorchmetrics.audio.ScaleInvariantSignalDistortionRatio计算 SI-SDR 作为验证指标。optim支持adam/adamw等优化器及lr、betas、weight_decay参数生成式配置如 flow matching还可追加sched调度器小节CosineAnnealingwarmup_steps、min_lr等。不同模型家族的架构段差异较大可通过对比仓库示例快速掌握掩码式mask-basedmask_estimatorRNNmask_processor参考通道掩码损失为SDRLoss支持scale_invariant见 masking.yaml生成式flow matchingestimatorTransformer UNetflow最优传输流samplerEuler 条件流匹配采样器num_steps: 20ssl_pretrain_masking掩码补丁自监督预训练mask_fraction: 0.7指标常用 SI-SDR / ESTOI / PESQ 组合见 flow_matching_generative_ssl_pretraining.yaml。五、微调配置Finetuning所有训练脚本均支持通过部分/完全加载预训练 checkpoint 权重到当前实例化的模型来实现便捷微调。前提是当前实例化模型的参数结构与预训练 checkpoint 匹配否则权重无法正确加载。预训练权重可通过两种方式提供提供 NeMo 模型文件路径init_from_nemo_model提供预训练 NeMo 模型名称init_from_pretrained_model将从云端自动下载。5.1 从零训练python examples/audio/audio_to_audio_train.py \ --config-pathpath to dir of configs --config-namename of config without .yaml) \ model.train_ds.manifest_filepathpath to manifest file \ model.validation_ds.manifest_filepathpath to manifest file \ trainer.devices1 \ trainer.acceleratorgpu \ trainer.max_epochs505.2 基于 NeMo 模型文件微调在命令行中追加init_from_nemo_model参数Hydra 的前缀表示新增配置键指向本地.nemo文件python examples/audio/audio_to_audio_train.py \ --config-pathpath to dir of configs --config-namename of config without .yaml) \ model.train_ds.manifest_filepathpath to manifest file \ model.validation_ds.manifest_filepathpath to manifest file \ trainer.devices1 \ trainer.acceleratorgpu \ trainer.max_epochs50 \ init_from_nemo_modelpath to .nemo model file5.3 基于预训练模型名称微调python examples/audio/audio_to_audio_train.py \ --config-pathpath to dir of configs --config-namename of config without .yaml) \ model.train_ds.manifest_filepathpath to manifest file \ model.validation_ds.manifest_filepathpath to manifest file \ trainer.devices1 \ trainer.acceleratorgpu \ trainer.max_epochs50 \ init_from_pretrained_modelname of pretrained checkpoint从训练脚本源码看model.maybe_init_from_pretrained_checkpoint(cfg)会在模型实例化后统一处理这两种预训练权重注入方式因此三条训练命令共用同一套参数覆盖机制。微调类配置可直接参考 examples/audio/conf/flow_matching_generative_finetuning.yaml。六、trainer 与 exp_manager 常用配置速查虽然trainer/exp_manager属于所有 NeMo 模型通用的配置详见 核心配置文档但 Audio 示例配置中有一批与音频训练强相关的惯用设置值得在编写配置文件时留意trainer.devices: -1使用全部可用 GPUstrategy: ddp开启分布式数据并行trainer.precision: 32O1/O2 自动混合精度场景下应设为 16trainer.enable_checkpointing: false与trainer.logger: falsecheckpoint 与日志交给exp_manager统一管理exp_manager.checkpoint_callback_params用monitor/mode控制保存依据如掩码式用val_lossmin预测式用val_sisdrmaxalways_save_nemo: true表示直接保存.nemo文件而非 PTL checkpointexp_manager.ema启用指数移动平均如decay: 0.999exp_manager.create_early_stopping_callback基于验证指标早停注意strict: false避免恢复训练时的监控指标报错exp_manager.resume_from_checkpoint/resume_if_exists断点续训。综上NeMo Audio 配置体系以 Hydra 覆盖机制为骨架以train_ds/validation_ds/test_ds为数据入口以encoder/decoder/estimator或mask_estimator为架构声明配合 Lhotse 高性能加载与温度重加权实现大规模多数据集训练。实际动手时建议先复制 examples/audio/conf 中与任务最接近的 YAML再用命令行覆盖 manifest 路径与训练参数即可快速跑通。【免费下载链接】SpeechA scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)项目地址: https://gitcode.com/GitHub_Trending/nem/Speech创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 NeMo Audio 配置文件完全指南：数据集、Lhotse 加载器、模型架构与微调实战
NeMo Audio 配置文件完全指南数据集、Lhotse 加载器、模型架构与微调实战【免费下载链接】SpeechA scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)项目地址: https://gitcode.com/GitHub_Trending/nem/Speech本指南以 NeMo Speech 仓库中 docs/source/audio/configs.rst 为核心系统讲解音频Audio集合模型配置文件的组织方式与全部核心参数从经典的 NeMo manifest 数据集配置到 Lhotse CutSet/Shar 高性能数据加载再到支持温度重加权的多数据集混合采样、模型架构声明与三种微调启动方式。读完本文你将能够独立读懂并编写一套可运行的 NeMo Audio 训练配置并能用命令行覆盖任意参数完成从零训练与基于预训练模型的微调。一、配置文件整体结构NeMo Audio 集合的配置文件采用 Hydra OmegaConf 体系组织。每个配置文件通常由name、model、trainer、exp_manager四大部分组成其中model段是核心一般需要包含数据集信息通过model.train_ds、model.validation_ds、model.test_ds小节声明训练、验证、测试数据增强参数在线增强如 RIR 卷积、加性噪声的开关与数据路径模型架构规格通过_target_指向具体的类实现如nemo.collections.audio.modules.transforms.AudioToSpectrogram声明编码器、解码器、估计器/掩码网络、损失函数与评估指标。关于 Experiment Manager 与 PyTorch Lightning trainer 等所有 NeMo 模型通用的参数参见 核心配置文档。所有 NeMo Audio 模型的示例配置文件均可在 examples/audio/conf 目录中找到仓库当前包含 15 个覆盖不同任务的配置masking.yaml、masking_with_online_augmentation.yaml、predictive.yaml、predictive_conformer.yaml、flow_matching_generative.yaml、flow_matching_generative_ssl_pretraining.yaml、score_based_generative.yaml、schroedinger_bridge.yaml、beamforming.yaml、maxine_bnr.yaml等。训练入口脚本为 examples/audio/audio_to_audio_train.py它通过hydra_runner(config_path./conf, config_namemasking)加载配置再根据model.type字段分发到对应的模型类mask_based→EncMaskDecAudioToAudioModelpredictive→PredictiveAudioToAudioModelscore_based→ScoreBasedGenerativeAudioToAudioModelschroedinger_bridge→SchroedingerBridgeAudioToAudioModelflow_matching→FlowMatchingAudioToAudioModelbnr→BNR2随后调用model.maybe_init_from_pretrained_checkpoint(cfg)加载预训练权重并执行trainer.fit(model)。二、NeMo Dataset 配置manifest 方式训练、验证、测试参数分别通过model.train_ds、model.validation_ds、model.test_ds小节指定。根据任务不同可能包含加载音频的采样率或时长等参数部分字段可以留空在运行时通过命令行指定。数据集处理类的完整参数列表参见 API 文档的数据集章节。一个典型的 train / validation / test 数据集配置示例如下对应 examples/audio/conf/masking.yamlmodel: sample_rate: 16000 skip_nan_grad: false train_ds: manifest_filepath: ??? input_key: audio_filepath # key of the input signal path in the manifest target_key: target_filepath # key of the target signal path in the manifest target_channel_selector: 0 # target signal is the first channel from files in target_key audio_duration: 4.0 # in seconds, audio segment duration for training random_offset: true # if the file is longer than audio_duration, use random offset to select a subsegment min_duration: ${model.train_ds.audio_duration} batch_size: 64 # batch size may be increased based on the available memory shuffle: true num_workers: 8 pin_memory: true validation_ds: manifest_filepath: ??? input_key: audio_filepath # key of the input signal path in the manifest target_key: target_filepath # key of the target signal path in the manifest target_channel_selector: 0 # target signal is the first channel from files in target_key batch_size: 64 # batch size may be increased based on the available memory shuffle: false num_workers: 4 pin_memory: true test_ds: manifest_filepath: ??? input_key: audio_filepath # key of the input signal path in the manifest target_key: target_filepath # key of the target signal path in the manifest target_channel_selector: 0 # target signal is the first channel from files in target_key batch_size: 1 # batch size may be increased based on the available memory shuffle: false num_workers: 4 pin_memory: true关键参数说明参数作用manifest_filepath指向 JSON Lines 格式的 manifest 文件???表示必填必须通过命令行或配置提供input_keymanifest 中输入信号路径对应的键如audio_filepath含噪/含混响信号target_keymanifest 中目标信号路径对应的键如target_filepath干净信号target_channel_selector当目标文件为多通道时选择第几通道作为目标0 表示第一通道audio_duration训练音频段时长秒。注意它与 STFT 参数联动例如hop_length256时audio_duration4.0对应的 STFT 时间帧数为1 4.0 // 0.256 16若想得到 256 帧则需audio_duration 1 256 * 0.256 ≈ 65.5实际配置中masking_with_online_augmentation.yaml使用truncate_duration: 4.0hop_length: 256并注释Number of STFT time frames 1 truncate_duration // encoder.hop_length 256这里的//是整除关系配置时应以实际帧数为准random_offset文件长于audio_duration时用随机偏移选取子片段训练min_duration过滤掉短于该时长的样本常用${model.train_ds.audio_duration}引用保持一致性batch_size批大小可按显存上调shuffle是否打乱数据训练 true、验证/测试 falsenum_workersDataLoader 工作进程数pin_memory是否固定内存加速 GPU 传输三、Lhotse Dataset 配置3.1 Lhotse CutSet启用 Lhotse 数据加载器后训练数据以 CutSet 清单形式组织。目标信号需要存放在自定义的target_recording字段中train_ds: use_lhotse: true # enable Lhotse data loader cuts_path: ??? # path to Lhotse cuts manifest with input signals and the corresponding target signals (target signals should be in the custom target_recording field) truncate_duration: 4.00 # truncate audio to 4 seconds truncate_offset_type: random # if the file is longer than truncate_duration, use random offset to select a subsegment batch_size: 64 # batch size may be increased based on the available memory shuffle: true num_workers: 8 pin_memory: true与 manifest 方式相比Lhotse 使用truncate_durationtruncate_offset_type组合替代audio_durationrandom_offsettruncate_duration将音频截断到固定时长truncate_offset_type: random在文件过长时随机选取子片段。3.2 Lhotse CutSet 在线增强在线增强通过 RIR房间冲激响应卷积与加性噪声实现需要在训练集配置中额外开启rir_enabled并提供 RIR 与噪声数据的 Lhotse 清单路径。完整示例见 examples/audio/conf/masking_with_online_augmentation.yamltrain_ds: use_lhotse: true # enable Lhotse data loader cuts_path: ??? # path to Lhotse cuts manifest with speech signals for augmentation (including custom target_recording field with the same signals) truncate_duration: 4.00 # truncate audio to 4 seconds truncate_offset_type: random # if the file is longer than truncate_duration, use random offset to select a subsegment batch_size: 64 # batch size may be increased based on the available memory shuffle: true num_workers: 8 pin_memory: true rir_enabled: true # enable room impulse response augmentation rir_path: ??? # path to Lhotse recordings manifest with room impulse response signals noise_path: ??? # path to Lhotse cuts manifest with noise signals注意在线增强模式下cuts_path提供的是用于增强的干净语音其target_recording字段与自身相同作为干净目标而validation_ds/test_ds的cuts_path则应直接提供含噪语音与干净目标的配对数据。增强相关的在线增强教程可参考 Speech_Enhancement_with_Online_Augmentation.ipynb。3.3 Lhotse Shar 格式Shar 是 Lhotse 的高性能打包存储格式适合海量数据与分布式训练。其配置只需要shar_path指向 shar 目录无需cuts_pathtrain_ds: shar_path: ??? use_lhotse: true truncate_duration: 4.00 # truncate audio to 4 seconds truncate_offset_type: random batch_size: 8 # batch size may be increased based on the available memory shuffle: true num_workers: 8 pin_memory: true一个完整的 Shar 配置示例见 examples/audio/conf/flow_matching_generative_ssl_pretraining.yaml该文件同时演示了 Shar 训练集与 manifest 验证集的混合用法。由于 Shar 数据量通常极大其trainer段需要显式设置max_steps如 10000与limit_train_batches如 1000来控制每个伪 epoch 的训练量并设置use_distributed_sampler: falseLhotse 数据加载所必需。3.4 多数据集温度重加权Dataset Reweighting with Temperature当通过嵌套的input_cfg组合多个数据集时可以用reweight_temperature控制采样分布。该特性允许你在增删数据集时无需手工重算权重即可平衡各数据集的采样比例。温度缩放公式为$$\hat{w}i \frac{w_i^{\tau}}{\sum{j} w_j^{\tau}}$$其中 $w_i$ 是数据集 $i$ 的原始权重$\tau$ 是温度$\hat{w}_i$ 是归一化后的采样概率。温度的作用机制temperature 1.0保持原始权重比例中性不重加权temperature 0.0所有数据集等概率采样与原始权重无关0 temperature 1.0相对大数据集更多采样小数据集temperature 1.0放大数据集权重之间的差异。两种配置格式标量值应用于所有嵌套层级会打印警告日志train_ds: use_lhotse: true reweight_temperature: 0.5 # Applied to all levels, warning logged input_cfg: - type: group input_cfg: - type: lhotse_shar shar_path: /path/to/dataset1 weight: 900 - type: lhotse_shar shar_path: /path/to/dataset2 weight: 100 - type: lhotse_shar shar_path: /path/to/dataset3 weight: 200 - type: nemo_tarred manifest_filepath: /path/to/dataset4/manifest.json tarred_audio_filepath: /path/to/dataset4/audio.tar weight: 300列表格式与最大嵌套深度一一对应每层一个温度train_ds: use_lhotse: true reweight_temperature: [1.0, 0.0] # Level 1: preserve ratios, Level 2: equalize input_cfg: - type: group weight: 0.7 input_cfg: - type: lhotse_shar shar_path: /path/to/dataset1 weight: 600 - type: lhotse_shar shar_path: /path/to/dataset2 weight: 400 - type: group weight: 0.3 input_cfg: - type: lhotse_shar shar_path: /path/to/dataset3 weight: 100注意若reweight_temperature以列表形式提供其长度必须与input_cfg的最大嵌套深度完全一致过多或过少都会抛出ValueError。若希望所有层级使用相同温度请使用标量值。最大嵌套深度计算最大嵌套深度即配置中input_cfg键的最大嵌套层数同一层的兄弟 group 共享同一个温度值。例如# This has maximum nesting depth 2 input_cfg: # Level 1 - type: group input_cfg: # Level 2 - type: lhotse_shar - type: group # Same level as above (sibling) input_cfg: # Level 2 (same as above) - type: lhotse_shar当input_cfg通过 CLI 覆盖为 YAML 文件路径时如model.train_ds.input_cfgtrain_all.yaml深度计算会加载被引用文件并遍历其内容统计嵌套的input_cfg键支持多级文件引用# train_all.yaml (referenced via input_cfgtrain_all.yaml) - type: group weight: 100 input_cfg: ${oc.env:MANIFEST_ROOT}/train_en.yaml # resolved at runtime - type: group weight: 200 input_cfg: ${oc.env:MANIFEST_ROOT}/train_de.yaml注意包含 OmegaConf 插值的路径如${oc.env:MANIFEST_ROOT}在深度统计阶段无法解析——它们会在运行时由OmegaConf.create()才被解析此类路径被当作单个额外嵌套层级处理。实战示例平衡多任务数据组train_ds: use_lhotse: true reweight_temperature: [1.0, 0.0] # Level 1: Preserve task ratios, Level 2: Equalize within tasks input_cfg: - type: group weight: 0.7 tags: task: asr input_cfg: - type: nemo_tarred manifest_filepath: /path/to/asr1/manifest.json tarred_audio_filepath: /path/to/asr1/audio.tar weight: 600 # Large dataset - type: nemo_tarred manifest_filepath: /path/to/asr2/manifest.json tarred_audio_filepath: /path/to/asr2/audio.tar weight: 100 # Small dataset (will be upsampled with temp0.0) - type: group weight: 0.3 tags: task: ast input_cfg: - type: nemo_tarred manifest_filepath: /path/to/ast1/manifest.json tarred_audio_filepath: /path/to/ast1/audio.tar weight: 50 - type: nemo_tarred manifest_filepath: /path/to/ast2/manifest.json tarred_audio_filepath: /path/to/ast2/audio.tar weight: 200该示例的效果Level 1 温度为1.0ASR 组与 AST 组之间保持 70/30 的原始比例Level 2 温度为0.0每组内部所有数据集等概率采样无论其原始权重如何小数据集被上采样。从源码实现看reweight_temperature参数定义于 nemo/collections/common/data/lhotse/dataloader.pyreweight_temperature: Any None支持 float / int / list 混合格式温度重加权逻辑位于 nemo/collections/common/data/lhotse/cutset.py对应的深度计算与行为验证测试见 tests/collections/common/test_lhotse_temperature_reweighting.py 与 tests/collections/common/test_lhotse_dataloading.py。input_cfg还支持type: nemo_tarredNeMo tarred 音频数据集与type: lhotse_shar两种底层数据集类型tags字段可用于给数据集打标签。四、模型架构配置每个配置文件都应声明实验所用的模型架构。下面是一个简单预测式predictive模型的完整示例对应 examples/audio/conf/predictive.yamlmodel: type: predictive sample_rate: 16000 skip_nan_grad: false num_outputs: 1 normalize_input: true # normalize the input signal to 0dBFS train_ds: manifest_filepath: ??? input_key: noisy_filepath target_key: clean_filepath audio_duration: 2.00 # trim audio to 2 seconds random_offset: true normalization_signal: input_signal batch_size: 8 # batch size may be increased based on the available memory shuffle: true num_workers: 8 pin_memory: true validation_ds: manifest_filepath: ??? input_key: noisy_filepath target_key: clean_filepath batch_size: 8 shuffle: false num_workers: 4 pin_memory: true encoder: _target_: nemo.collections.audio.modules.transforms.AudioToSpectrogram fft_length: 510 # Number of subbands in the STFT fft_length // 2 1 256 hop_length: 128 magnitude_power: 0.5 scale: 0.33 decoder: _target_: nemo.collections.audio.modules.transforms.SpectrogramToAudio fft_length: ${model.encoder.fft_length} hop_length: ${model.encoder.hop_length} magnitude_power: ${model.encoder.magnitude_power} scale: ${model.encoder.scale} estimator: _target_: nemo.collections.audio.parts.submodules.ncsnpp.SpectrogramNoiseConditionalScoreNetworkPlusPlus in_channels: 1 # single-channel noisy input out_channels: 1 # single-channel estimate num_res_blocks: 3 # increased number of res blocks pad_time_to: 64 # pad to 64 frames for the time dimension pad_dimension_to: 0 # no padding in the frequency dimension loss: _target_: nemo.collections.audio.losses.MSELoss # computed in the time domain metrics: val: sisdr: # output SI-SDR _target_: torchmetrics.audio.ScaleInvariantSignalDistortionRatio optim: name: adam lr: 1e-4 # optimizer arguments betas: [0.9, 0.999] weight_decay: 0.0架构配置的要点type字段predictive对应PredictiveAudioToAudioModel训练脚本据此选择模型类不写该字段时默认使用mask_basedEncMaskDecAudioToAudioModel并打印警告。encoder/decoder通过_target_指向nemo.collections.audio.modules.transforms中的 STFT 变换类。注意fft_length510时子带数为510 // 2 1 256decoder通过${model.encoder.fft_length}等 OmegaConf 插值自动与encoder保持一致避免参数漂移。estimator预测式模型在此使用 NCSNNoise Conditional Score Network Plus Plus网络in_channels/out_channels对应输入输出通道数num_res_blocks控制残差块数量pad_time_to/pad_dimension_to控制时间/频率维度的 padding。lossnemo.collections.audio.losses.MSELoss在时域计算即预测目标为时域波形。metricstorchmetrics.audio.ScaleInvariantSignalDistortionRatio计算 SI-SDR 作为验证指标。optim支持adam/adamw等优化器及lr、betas、weight_decay参数生成式配置如 flow matching还可追加sched调度器小节CosineAnnealingwarmup_steps、min_lr等。不同模型家族的架构段差异较大可通过对比仓库示例快速掌握掩码式mask-basedmask_estimatorRNNmask_processor参考通道掩码损失为SDRLoss支持scale_invariant见 masking.yaml生成式flow matchingestimatorTransformer UNetflow最优传输流samplerEuler 条件流匹配采样器num_steps: 20ssl_pretrain_masking掩码补丁自监督预训练mask_fraction: 0.7指标常用 SI-SDR / ESTOI / PESQ 组合见 flow_matching_generative_ssl_pretraining.yaml。五、微调配置Finetuning所有训练脚本均支持通过部分/完全加载预训练 checkpoint 权重到当前实例化的模型来实现便捷微调。前提是当前实例化模型的参数结构与预训练 checkpoint 匹配否则权重无法正确加载。预训练权重可通过两种方式提供提供 NeMo 模型文件路径init_from_nemo_model提供预训练 NeMo 模型名称init_from_pretrained_model将从云端自动下载。5.1 从零训练python examples/audio/audio_to_audio_train.py \ --config-pathpath to dir of configs --config-namename of config without .yaml) \ model.train_ds.manifest_filepathpath to manifest file \ model.validation_ds.manifest_filepathpath to manifest file \ trainer.devices1 \ trainer.acceleratorgpu \ trainer.max_epochs505.2 基于 NeMo 模型文件微调在命令行中追加init_from_nemo_model参数Hydra 的前缀表示新增配置键指向本地.nemo文件python examples/audio/audio_to_audio_train.py \ --config-pathpath to dir of configs --config-namename of config without .yaml) \ model.train_ds.manifest_filepathpath to manifest file \ model.validation_ds.manifest_filepathpath to manifest file \ trainer.devices1 \ trainer.acceleratorgpu \ trainer.max_epochs50 \ init_from_nemo_modelpath to .nemo model file5.3 基于预训练模型名称微调python examples/audio/audio_to_audio_train.py \ --config-pathpath to dir of configs --config-namename of config without .yaml) \ model.train_ds.manifest_filepathpath to manifest file \ model.validation_ds.manifest_filepathpath to manifest file \ trainer.devices1 \ trainer.acceleratorgpu \ trainer.max_epochs50 \ init_from_pretrained_modelname of pretrained checkpoint从训练脚本源码看model.maybe_init_from_pretrained_checkpoint(cfg)会在模型实例化后统一处理这两种预训练权重注入方式因此三条训练命令共用同一套参数覆盖机制。微调类配置可直接参考 examples/audio/conf/flow_matching_generative_finetuning.yaml。六、trainer 与 exp_manager 常用配置速查虽然trainer/exp_manager属于所有 NeMo 模型通用的配置详见 核心配置文档但 Audio 示例配置中有一批与音频训练强相关的惯用设置值得在编写配置文件时留意trainer.devices: -1使用全部可用 GPUstrategy: ddp开启分布式数据并行trainer.precision: 32O1/O2 自动混合精度场景下应设为 16trainer.enable_checkpointing: false与trainer.logger: falsecheckpoint 与日志交给exp_manager统一管理exp_manager.checkpoint_callback_params用monitor/mode控制保存依据如掩码式用val_lossmin预测式用val_sisdrmaxalways_save_nemo: true表示直接保存.nemo文件而非 PTL checkpointexp_manager.ema启用指数移动平均如decay: 0.999exp_manager.create_early_stopping_callback基于验证指标早停注意strict: false避免恢复训练时的监控指标报错exp_manager.resume_from_checkpoint/resume_if_exists断点续训。综上NeMo Audio 配置体系以 Hydra 覆盖机制为骨架以train_ds/validation_ds/test_ds为数据入口以encoder/decoder/estimator或mask_estimator为架构声明配合 Lhotse 高性能加载与温度重加权实现大规模多数据集训练。实际动手时建议先复制 examples/audio/conf 中与任务最接近的 YAML再用命令行覆盖 manifest 路径与训练参数即可快速跑通。【免费下载链接】SpeechA scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)项目地址: https://gitcode.com/GitHub_Trending/nem/Speech创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考