Diffusers 中 CogVideoX 视频生成管线实战指南从文生视频到显存与推理优化【免费下载链接】diffusers Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch.项目地址: https://gitcode.com/GitHub_Trending/di/diffusers本指南聚焦 Hugging Face Diffusers 仓库中 CogVideoX 系列视频生成管线的完整用法涵盖模型架构要点、文生视频T2V两种官方推荐部署方案显存优先 / 速度优先、__call__核心参数解析、分辨率与帧数选取建议、LoRA 适配以及 I2V / V2V / FunControl 等扩展管线。读完本文你将掌握如何在当前仓库环境下加载、运行并调优 CogVideoX并理解其底层调用链与优化原理。CogVideoX 模型与管线概述CogVideoX 是一个大规模扩散 TransformerDiffusion Transformer模型官方提供2B与5B两种参数规模用于从文本生成更长、更连贯的视频。根据 官方 API 文档 的描述其核心设计包含三点3D 因果 VAE3D Causal Variational Autoencoder通过降低视频数据的序列长度来提升处理效率、减少训练算力开销同时有效抑制生成视频中的闪烁flickering现象。带自适应 LayerNorm 的专家 Transformer提升文本与视频之间的对齐质量。3D 全注意力3D full attention更准确地捕捉生成视频中的运动与时间信息。在 Diffusers 中CogVideoX 由 pipelines/cogvideo 目录 下的多个管线类实现。从源码看管线由五个核心组件构成见 pipeline_cogvideox.py 的__init__注册逻辑组件类型说明tokenizerT5Tokenizer文本分词器text_encoderT5EncoderModel冻结的 T5 文本编码器t5-v1_1-xxl 变体vaeAutoencoderKLCogVideoX3D 因果 VAE负责视频与潜空间的编解码transformerCogVideoXTransformer3DModel文本条件的 3D Transformer负责去噪schedulerCogVideoXDDIMScheduler/CogVideoXDPMScheduler采样调度器管线默认的模型卸载顺序为text_encoder-transformer-vae即model_cpu_offload_seq属性这一点与官方推荐的 CPU offload 用法直接相关。快速开始文本生成视频Text-to-VideoCogVideoXPipeline 支持两种官方推荐用法显存优先memory与推理速度优先inference speed对应下方两个可切换的代码方案。方案一显存优先部署量化 层间类型转换 模型卸载该方案适合显存受限的环境。文档指出量化后的 CogVideoX 5B 模型约需16GB 显存。完整示例代码如下源自 官方文档import torch from diffusers import CogVideoXPipeline, AutoModel, TorchAoConfig from diffusers.quantizers import PipelineQuantizationConfig from diffusers.hooks import apply_group_offloading from diffusers.utils import export_to_video from torchao.quantization import Int8WeightOnlyConfig # quantize weights to int8 with torchao pipeline_quant_config PipelineQuantizationConfig( quant_mapping{transformer: TorchAoConfig(Int8WeightOnlyConfig())} ) # fp8 layerwise weight-casting transformer AutoModel.from_pretrained( THUDM/CogVideoX-5b, subfoldertransformer, dtypetorch.bfloat16 ) transformer.enable_layerwise_casting( storage_dtypetorch.float8_e4m3fn, compute_dtypetorch.bfloat16 ) pipeline CogVideoXPipeline.from_pretrained( THUDM/CogVideoX-5b, transformertransformer, quantization_configpipeline_quant_config, dtypetorch.bfloat16 ) pipeline.to(cuda) # or mps, xpu, cpu # model-offloading pipeline.enable_model_cpu_offload() prompt A detailed wooden toy ship with intricately carved masts and sails is seen gliding smoothly over a plush, blue carpet that mimics the waves of the sea. The ships hull is painted a rich brown, with tiny windows. The carpet, soft and textured, provides a perfect backdrop, resembling an oceanic expanse. Surrounding the ship are various other toys and childrens items, hinting at a playful environment. The scene captures the innocence and imagination of childhood, with the toy ships journey symbolizing endless adventures in a whimsical, indoor setting. video pipeline( promptprompt, guidance_scale6, num_inference_steps50 ).frames[0] export_to_video(video, output.mp4, fps8)这里用到了三层显存优化手段可以叠加使用int8 权重量化通过PipelineQuantizationConfig将transformer组件的权重量化为 int8基于 torchao 的Int8WeightOnlyConfigfp8 层间类型转换layerwise casting调用enable_layerwise_casting以torch.float8_e4m3fn作为存储精度、torch.bfloat16作为计算精度降低权重驻留内存模型 CPU 卸载enable_model_cpu_offload()按text_encoder-transformer-vae的顺序逐模块卸载到 CPU。更详细的各类显存节省技巧可参考 Reduce memory usage 指南。方案二推理速度优先torch.compile 编译加速该方案适合追求吞吐的场景。文档指出首次编译较慢但后续调用管线会显著提速在 80GB A100 上torch.compile 后的平均推理时间为76.27 秒而未编译模型为96.89 秒。import torch from diffusers import CogVideoXPipeline from diffusers.utils import export_to_video pipeline CogVideoXPipeline.from_pretrained( THUDM/CogVideoX-2b, dtypetorch.float16 ).to(cuda) # or mps, xpu, cpu # torch.compile pipeline.transformer.to(memory_formattorch.channels_last) pipeline.transformer torch.compile( pipeline.transformer, modemax-autotune, fullgraphTrue ) prompt A detailed wooden toy ship with intricately carved masts and sails is seen gliding smoothly over a plush, blue carpet that mimics the waves of the sea. The ships hull is painted a rich brown, with tiny windows. The carpet, soft and textured, provides a perfect backdrop, resembling an oceanic expanse. Surrounding the ship are various other toys and childrens items, hinting at a playful environment. The scene captures the innocence and imagination of childhood, with the toy ships journey symbolizing endless adventures in a whimsical, indoor setting. video pipeline( promptprompt, guidance_scale6, num_inference_steps50 ).frames[0] export_to_video(video, output.mp4, fps8)关键点在于先将 Transformer 转为channels_last内存布局再以max-autotune模式、fullgraphTrue进行编译。torch.compile的完整背景可参考 fp16 优化指南中的 torch.compile 章节。__call__核心参数详解在 CogVideoXPipeline.call签名 中官方为每个参数提供了详细说明下面结合源码归纳最常用的参数参数默认值说明promptNone文本提示可为str或list[str]与prompt_embeds二选一negative_promptNone负向提示仅当guidance_scale 1时生效可与negative_prompt_embeds互换height/width由sample_height/sample_width × 8推出默认约 480×720输出视频分辨率必须能被 8 整除源码check_inputs校验num_frames48即sample_frames生成的帧数必须能被vae_scale_factor_temporal4整除CogVideoX 以(秒数×fps1)帧为条件实际输出会比设定多 1 帧num_inference_steps50去噪步数越多质量越高、耗时越长timestepsNone自定义去噪时间步列表需降序覆盖调度器默认排布传入后num_inference_steps须为Noneguidance_scale6无分类器引导强度1 时启用 CFG越接近文本质量可能略降use_dynamic_cfgFalse若为True推理过程中按余弦曲线动态调整引导强度num_videos_per_prompt1每个提示生成的视频数量eta0.0DDIM 采样器专属参数η∈[0,1]其他调度器忽略generatorNone单个或列表形式的torch.Generator用于复现结果latentsNone预生成的噪声潜变量可用于固定随机种子或跨提示复用prompt_embeds/negative_prompt_embedsNone预计算的文本嵌入便于做 prompt weighting 等定制output_typepil输出格式pil、np或latentreturn_dictTrue是否返回CogVideoXPipelineOutput否则返回元组attention_kwargsNone透传给 AttentionProcessor 的 kwargs如 PAG、缓存等callback_on_step_endNone每个去噪步结束时的回调函数callback_on_step_end_tensor_inputs[latents]回调中可访问的张量列表须在_callback_tensor_inputs内max_sequence_length226编码文本的最大序列长度须与transformer.config.max_text_seq_length一致否则可能影响生成质量值得注意的源码细节动态 CFG 实现当use_dynamic_cfgTrue时源码按公式1 guidance_scale * ((1 - cos(π * ((num_inference_steps - t) / num_inference_steps) ** 5)) / 2)在每个时间步动态更新引导强度见 pipeline_cogvideox.py 去噪循环。CogVideoX 1.5 帧数填充若 Transformer 配置了patch_size_t当潜变量帧数不能被其整除时管线会自动填充若干帧解码前再丢弃latents[:, additional_frames:]。DPM-Solver 分支使用CogVideoXDPMScheduler时step调用会额外传入上一轮的old_pred_original_sample这是 DPM 多步求解器的特有逻辑。3D 旋转位置编码管线通过get_3d_rotary_pos_embed生成时空位置编码CogVideoX 1.0 使用网格裁剪坐标1.5 使用grid_typeslice方式见 位置编码准备函数。分辨率、帧数与 fps 建议Notes官方文档对生成参数给出了明确的经验值直接照用可显著提升成片质量T2V 检查点预训练分辨率即1360×768该分辨率下效果最佳。I2V 检查点支持多种分辨率宽度可在 7681360 之间变化但高度必须为 768注原文档写 758实际以官方最新文档为准仓库 README 与社区脚本多使用 768宽高都必须能被 16 整除。帧数T2V 与 I2V 检查点在81 与 161 帧时效果最好建议以16fps导出视频。需要说明的是管线源码默认的num_frames为 486 秒 × 8fps 1 帧的取整基准LoRA 示例中则使用num_frames81、fps16的组合这两套帧率约定都可行按需选择即可。LoRA 适配加载与强度控制CogVideoX 管线原生支持 LoRA底层通过CogVideoXLoraLoaderMixin实现load_lora_weights会把权重注入transformer同时校验所有键名必须包含lora子串。官方提供的完整示例含enable_model_cpu_offload组合使用import torch from diffusers import CogVideoXPipeline from diffusers.hooks import apply_group_offloading from diffusers.utils import export_to_video pipeline CogVideoXPipeline.from_pretrained( THUDM/CogVideoX-5b, dtypetorch.bfloat16 ) pipeline.to(cuda) # or mps, xpu, cpu # load LoRA weights pipeline.load_lora_weights(finetrainers/CogVideoX-1.5-crush-smol-v0, adapter_namecrush-lora) pipeline.set_adapters(crush-lora, 0.9) # model-offloading pipeline.enable_model_cpu_offload() prompt PIKA_CRUSH A large metal cylinder is seen pressing down on a pile of Oreo cookies, flattening them as if they were under a hydraulic press. negative_prompt inconsistent motion, blurry motion, worse quality, degenerate outputs, deformed outputs video pipeline( promptprompt, negative_promptnegative_prompt, num_frames81, height480, width768, num_inference_steps50 ).frames[0] export_to_video(video, output.mp4, fps16)要点load_lora_weights(..., adapter_namecrush-lora)以命名 adapter 方式加载 LoRA便于多 adapter 管理set_adapters(crush-lora, 0.9)设置生效 adapter 及其权重强度本例同时示范了 81 帧、480×768、16fps 的参数组合以及负向提示的写法。如需从训练侧了解 LoRA 适配器的产生过程可参考仓库中的 CogVideoX LoRA 训练脚本 及其 README。扩展管线I2V、V2V 与 FunControl除文生视频的CogVideoXPipeline外cogvideo 管线目录 还提供三种扩展管线全部共用CogVideoXPipelineOutput输出结构CogVideoXImageToVideoPipeline图生视频以单张图片作为起始帧条件生成视频。核心入口 pipeline_cogvideox_image2video.py 中__call__首个参数即为imagePipelineImageInputnum_frames默认 49。官方示例import torch from diffusers import CogVideoXImageToVideoPipeline from diffusers.utils import export_to_video, load_image pipe CogVideoXImageToVideoPipeline.from_pretrained(THUDM/CogVideoX-5b-I2V, torch_dtypetorch.bfloat16) pipe.to(cuda) prompt An astronaut hatching from an egg, on the surface of the moon, the darkness and depth of space realised in the background. High quality, ultrarealistic detail and breath-taking movie-like camera shot. image load_image(astronaut.jpg) video pipe(image, prompt, use_dynamic_cfgTrue) export_to_video(video.frames[0], output.mp4, fps8)CogVideoXVideoToVideoPipeline视频生视频输入一段参考视频通过strength参数控制改造强度并可自行替换调度器示例中使用CogVideoXDPMSchedulerimport torch from diffusers import CogVideoXDPMScheduler, CogVideoXVideoToVideoPipeline from diffusers.utils import export_to_video, load_video pipe CogVideoXVideoToVideoPipeline.from_pretrained(THUDM/CogVideoX-5b, torch_dtypetorch.bfloat16) pipe.to(cuda) pipe.scheduler CogVideoXDPMScheduler.from_config(pipe.scheduler.config) input_video load_video(hiker.mp4) prompt ( An astronaut stands triumphantly at the peak of a towering mountain. Panorama of rugged peaks and valleys. Very futuristic vibe and animated aesthetic. Highlights of purple and golden colors in the scene. The sky is looks like an animated/cartoonish dream of galaxies, nebulae, stars, planets, moons, but the remainder of the scene is mostly realistic. ) video pipe(videoinput_video, promptprompt, strength0.8, guidance_scale6, num_inference_steps50).frames[0] export_to_video(video, output.mp4, fps8)CogVideoXFunControlPipeline可控视频生成面向 Alibaba-PAI 的 CogVideoX-Fun 系列检查点如alibaba-pai/CogVideoX-Fun-V1.1-5b-Pose通过control_video输入姿态等控制信号约束生成示例中调度器替换为DDIMSchedulerimport torch from diffusers import CogVideoXFunControlPipeline, DDIMScheduler from diffusers.utils import export_to_video, load_video pipe CogVideoXFunControlPipeline.from_pretrained( alibaba-pai/CogVideoX-Fun-V1.1-5b-Pose, torch_dtypetorch.bfloat16 ) pipe.scheduler DDIMScheduler.from_config(pipe.scheduler.config) pipe.to(cuda) control_video load_video(hiker.mp4) prompt ( An astronaut stands triumphantly at the peak of a towering mountain. Panorama of rugged peaks and valleys. Very futuristic vibe and animated aesthetic. Highlights of purple and golden colors in the scene. The sky is looks like an animated/cartoonish dream of galaxies, nebulae, stars, planets, moons, but the remainder of the scene is mostly realistic. ) video pipe(promptprompt, control_videocontrol_video).frames[0] export_to_video(video, output.mp4, fps8)统一的输出对象所有 CogVideo 管线均返回CogVideoXPipelineOutput其唯一字段frames的类型为torch.Tensor/np.ndarray或list[list[PIL.Image.Image]]形状为(batch_size, num_frames, channels, height, width)。访问.frames[0]即可取得单个视频的帧序列配合export_to_video导出。显存占用与优化方法对照官方文档给出了开启各类显存优化手段前后的显存占用对照表基于 5B 模型方法启用后显存占用未启用时显存占用enable_model_cpu_offload19GB33GBenable_sequential_cpu_offload4GB~33GB推理速度极慢enable_tiling配合enable_model_cpu_offload11GB—选择建议显存 ≥ 33GB可直接全量加载优先保证速度显存 1633GB优先enable_model_cpu_offload()实测约 19GB必要时叠加 int8/fp8 量化到 16GB 左右显存 8GB考虑enable_sequential_cpu_offload()4GB但需接受明显变慢的推理速度若仍不足可再叠加enable_tiling()将 VAE 解码分块执行进一步压低峰值。这三类方法都属于DiffusionPipeline的通用能力完整原理与更多技巧见 Reduce memory usage 指南。源码结构速览与测试验证管线实现pipeline_cogvideox.pyT2V、pipeline_cogvideox_image2video.pyI2V、pipeline_cogvideox_video2video.pyV2V、pipeline_cogvideox_fun_control.pyFunControl、pipeline_output.py输出定义。测试用例test_cogvideox.py、test_cogvideox_image2video.py、test_cogvideox_video2video.py、test_cogvideox_fun_control.py可用于核对各管线的参数校验、输入输出形状与内存优化开关的行为。LoRA 加载实现CogVideoXLoraLoaderMixinLoRA 权重仅注入transformer模块_lora_loadable_modules [transformer]。训练侧参考train_cogvideox_lora.py 与 train_cogvideox_image_to_video_lora.py 展示了如何为 CogVideoX 训练自定义 LoRA。小结CogVideoX 在 Diffusers 中形成了覆盖 T2V、I2V、V2V、可控生成的完整管线家族配合 int8/fp8 量化、layerwise casting、模型卸载、enable_tiling与torch.compile可以在从 4GB 到 80GB 的各级显存环境下灵活部署。使用时分清三点即可快速上手T2V 优先 1360×768、I2V 高度固定 768 且宽高能被 16 整除、81/161 帧搭配 16fps 导出需要风格定制时通过load_lora_weights加载社区 LoRA显存紧张时按内存对照表逐级叠加优化开关。【免费下载链接】diffusers Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch.项目地址: https://gitcode.com/GitHub_Trending/di/diffusers创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 Diffusers 中 CogVideoX 视频生成管线实战指南：从文生视频到显存与推理优化
Diffusers 中 CogVideoX 视频生成管线实战指南从文生视频到显存与推理优化【免费下载链接】diffusers Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch.项目地址: https://gitcode.com/GitHub_Trending/di/diffusers本指南聚焦 Hugging Face Diffusers 仓库中 CogVideoX 系列视频生成管线的完整用法涵盖模型架构要点、文生视频T2V两种官方推荐部署方案显存优先 / 速度优先、__call__核心参数解析、分辨率与帧数选取建议、LoRA 适配以及 I2V / V2V / FunControl 等扩展管线。读完本文你将掌握如何在当前仓库环境下加载、运行并调优 CogVideoX并理解其底层调用链与优化原理。CogVideoX 模型与管线概述CogVideoX 是一个大规模扩散 TransformerDiffusion Transformer模型官方提供2B与5B两种参数规模用于从文本生成更长、更连贯的视频。根据 官方 API 文档 的描述其核心设计包含三点3D 因果 VAE3D Causal Variational Autoencoder通过降低视频数据的序列长度来提升处理效率、减少训练算力开销同时有效抑制生成视频中的闪烁flickering现象。带自适应 LayerNorm 的专家 Transformer提升文本与视频之间的对齐质量。3D 全注意力3D full attention更准确地捕捉生成视频中的运动与时间信息。在 Diffusers 中CogVideoX 由 pipelines/cogvideo 目录 下的多个管线类实现。从源码看管线由五个核心组件构成见 pipeline_cogvideox.py 的__init__注册逻辑组件类型说明tokenizerT5Tokenizer文本分词器text_encoderT5EncoderModel冻结的 T5 文本编码器t5-v1_1-xxl 变体vaeAutoencoderKLCogVideoX3D 因果 VAE负责视频与潜空间的编解码transformerCogVideoXTransformer3DModel文本条件的 3D Transformer负责去噪schedulerCogVideoXDDIMScheduler/CogVideoXDPMScheduler采样调度器管线默认的模型卸载顺序为text_encoder-transformer-vae即model_cpu_offload_seq属性这一点与官方推荐的 CPU offload 用法直接相关。快速开始文本生成视频Text-to-VideoCogVideoXPipeline 支持两种官方推荐用法显存优先memory与推理速度优先inference speed对应下方两个可切换的代码方案。方案一显存优先部署量化 层间类型转换 模型卸载该方案适合显存受限的环境。文档指出量化后的 CogVideoX 5B 模型约需16GB 显存。完整示例代码如下源自 官方文档import torch from diffusers import CogVideoXPipeline, AutoModel, TorchAoConfig from diffusers.quantizers import PipelineQuantizationConfig from diffusers.hooks import apply_group_offloading from diffusers.utils import export_to_video from torchao.quantization import Int8WeightOnlyConfig # quantize weights to int8 with torchao pipeline_quant_config PipelineQuantizationConfig( quant_mapping{transformer: TorchAoConfig(Int8WeightOnlyConfig())} ) # fp8 layerwise weight-casting transformer AutoModel.from_pretrained( THUDM/CogVideoX-5b, subfoldertransformer, dtypetorch.bfloat16 ) transformer.enable_layerwise_casting( storage_dtypetorch.float8_e4m3fn, compute_dtypetorch.bfloat16 ) pipeline CogVideoXPipeline.from_pretrained( THUDM/CogVideoX-5b, transformertransformer, quantization_configpipeline_quant_config, dtypetorch.bfloat16 ) pipeline.to(cuda) # or mps, xpu, cpu # model-offloading pipeline.enable_model_cpu_offload() prompt A detailed wooden toy ship with intricately carved masts and sails is seen gliding smoothly over a plush, blue carpet that mimics the waves of the sea. The ships hull is painted a rich brown, with tiny windows. The carpet, soft and textured, provides a perfect backdrop, resembling an oceanic expanse. Surrounding the ship are various other toys and childrens items, hinting at a playful environment. The scene captures the innocence and imagination of childhood, with the toy ships journey symbolizing endless adventures in a whimsical, indoor setting. video pipeline( promptprompt, guidance_scale6, num_inference_steps50 ).frames[0] export_to_video(video, output.mp4, fps8)这里用到了三层显存优化手段可以叠加使用int8 权重量化通过PipelineQuantizationConfig将transformer组件的权重量化为 int8基于 torchao 的Int8WeightOnlyConfigfp8 层间类型转换layerwise casting调用enable_layerwise_casting以torch.float8_e4m3fn作为存储精度、torch.bfloat16作为计算精度降低权重驻留内存模型 CPU 卸载enable_model_cpu_offload()按text_encoder-transformer-vae的顺序逐模块卸载到 CPU。更详细的各类显存节省技巧可参考 Reduce memory usage 指南。方案二推理速度优先torch.compile 编译加速该方案适合追求吞吐的场景。文档指出首次编译较慢但后续调用管线会显著提速在 80GB A100 上torch.compile 后的平均推理时间为76.27 秒而未编译模型为96.89 秒。import torch from diffusers import CogVideoXPipeline from diffusers.utils import export_to_video pipeline CogVideoXPipeline.from_pretrained( THUDM/CogVideoX-2b, dtypetorch.float16 ).to(cuda) # or mps, xpu, cpu # torch.compile pipeline.transformer.to(memory_formattorch.channels_last) pipeline.transformer torch.compile( pipeline.transformer, modemax-autotune, fullgraphTrue ) prompt A detailed wooden toy ship with intricately carved masts and sails is seen gliding smoothly over a plush, blue carpet that mimics the waves of the sea. The ships hull is painted a rich brown, with tiny windows. The carpet, soft and textured, provides a perfect backdrop, resembling an oceanic expanse. Surrounding the ship are various other toys and childrens items, hinting at a playful environment. The scene captures the innocence and imagination of childhood, with the toy ships journey symbolizing endless adventures in a whimsical, indoor setting. video pipeline( promptprompt, guidance_scale6, num_inference_steps50 ).frames[0] export_to_video(video, output.mp4, fps8)关键点在于先将 Transformer 转为channels_last内存布局再以max-autotune模式、fullgraphTrue进行编译。torch.compile的完整背景可参考 fp16 优化指南中的 torch.compile 章节。__call__核心参数详解在 CogVideoXPipeline.call签名 中官方为每个参数提供了详细说明下面结合源码归纳最常用的参数参数默认值说明promptNone文本提示可为str或list[str]与prompt_embeds二选一negative_promptNone负向提示仅当guidance_scale 1时生效可与negative_prompt_embeds互换height/width由sample_height/sample_width × 8推出默认约 480×720输出视频分辨率必须能被 8 整除源码check_inputs校验num_frames48即sample_frames生成的帧数必须能被vae_scale_factor_temporal4整除CogVideoX 以(秒数×fps1)帧为条件实际输出会比设定多 1 帧num_inference_steps50去噪步数越多质量越高、耗时越长timestepsNone自定义去噪时间步列表需降序覆盖调度器默认排布传入后num_inference_steps须为Noneguidance_scale6无分类器引导强度1 时启用 CFG越接近文本质量可能略降use_dynamic_cfgFalse若为True推理过程中按余弦曲线动态调整引导强度num_videos_per_prompt1每个提示生成的视频数量eta0.0DDIM 采样器专属参数η∈[0,1]其他调度器忽略generatorNone单个或列表形式的torch.Generator用于复现结果latentsNone预生成的噪声潜变量可用于固定随机种子或跨提示复用prompt_embeds/negative_prompt_embedsNone预计算的文本嵌入便于做 prompt weighting 等定制output_typepil输出格式pil、np或latentreturn_dictTrue是否返回CogVideoXPipelineOutput否则返回元组attention_kwargsNone透传给 AttentionProcessor 的 kwargs如 PAG、缓存等callback_on_step_endNone每个去噪步结束时的回调函数callback_on_step_end_tensor_inputs[latents]回调中可访问的张量列表须在_callback_tensor_inputs内max_sequence_length226编码文本的最大序列长度须与transformer.config.max_text_seq_length一致否则可能影响生成质量值得注意的源码细节动态 CFG 实现当use_dynamic_cfgTrue时源码按公式1 guidance_scale * ((1 - cos(π * ((num_inference_steps - t) / num_inference_steps) ** 5)) / 2)在每个时间步动态更新引导强度见 pipeline_cogvideox.py 去噪循环。CogVideoX 1.5 帧数填充若 Transformer 配置了patch_size_t当潜变量帧数不能被其整除时管线会自动填充若干帧解码前再丢弃latents[:, additional_frames:]。DPM-Solver 分支使用CogVideoXDPMScheduler时step调用会额外传入上一轮的old_pred_original_sample这是 DPM 多步求解器的特有逻辑。3D 旋转位置编码管线通过get_3d_rotary_pos_embed生成时空位置编码CogVideoX 1.0 使用网格裁剪坐标1.5 使用grid_typeslice方式见 位置编码准备函数。分辨率、帧数与 fps 建议Notes官方文档对生成参数给出了明确的经验值直接照用可显著提升成片质量T2V 检查点预训练分辨率即1360×768该分辨率下效果最佳。I2V 检查点支持多种分辨率宽度可在 7681360 之间变化但高度必须为 768注原文档写 758实际以官方最新文档为准仓库 README 与社区脚本多使用 768宽高都必须能被 16 整除。帧数T2V 与 I2V 检查点在81 与 161 帧时效果最好建议以16fps导出视频。需要说明的是管线源码默认的num_frames为 486 秒 × 8fps 1 帧的取整基准LoRA 示例中则使用num_frames81、fps16的组合这两套帧率约定都可行按需选择即可。LoRA 适配加载与强度控制CogVideoX 管线原生支持 LoRA底层通过CogVideoXLoraLoaderMixin实现load_lora_weights会把权重注入transformer同时校验所有键名必须包含lora子串。官方提供的完整示例含enable_model_cpu_offload组合使用import torch from diffusers import CogVideoXPipeline from diffusers.hooks import apply_group_offloading from diffusers.utils import export_to_video pipeline CogVideoXPipeline.from_pretrained( THUDM/CogVideoX-5b, dtypetorch.bfloat16 ) pipeline.to(cuda) # or mps, xpu, cpu # load LoRA weights pipeline.load_lora_weights(finetrainers/CogVideoX-1.5-crush-smol-v0, adapter_namecrush-lora) pipeline.set_adapters(crush-lora, 0.9) # model-offloading pipeline.enable_model_cpu_offload() prompt PIKA_CRUSH A large metal cylinder is seen pressing down on a pile of Oreo cookies, flattening them as if they were under a hydraulic press. negative_prompt inconsistent motion, blurry motion, worse quality, degenerate outputs, deformed outputs video pipeline( promptprompt, negative_promptnegative_prompt, num_frames81, height480, width768, num_inference_steps50 ).frames[0] export_to_video(video, output.mp4, fps16)要点load_lora_weights(..., adapter_namecrush-lora)以命名 adapter 方式加载 LoRA便于多 adapter 管理set_adapters(crush-lora, 0.9)设置生效 adapter 及其权重强度本例同时示范了 81 帧、480×768、16fps 的参数组合以及负向提示的写法。如需从训练侧了解 LoRA 适配器的产生过程可参考仓库中的 CogVideoX LoRA 训练脚本 及其 README。扩展管线I2V、V2V 与 FunControl除文生视频的CogVideoXPipeline外cogvideo 管线目录 还提供三种扩展管线全部共用CogVideoXPipelineOutput输出结构CogVideoXImageToVideoPipeline图生视频以单张图片作为起始帧条件生成视频。核心入口 pipeline_cogvideox_image2video.py 中__call__首个参数即为imagePipelineImageInputnum_frames默认 49。官方示例import torch from diffusers import CogVideoXImageToVideoPipeline from diffusers.utils import export_to_video, load_image pipe CogVideoXImageToVideoPipeline.from_pretrained(THUDM/CogVideoX-5b-I2V, torch_dtypetorch.bfloat16) pipe.to(cuda) prompt An astronaut hatching from an egg, on the surface of the moon, the darkness and depth of space realised in the background. High quality, ultrarealistic detail and breath-taking movie-like camera shot. image load_image(astronaut.jpg) video pipe(image, prompt, use_dynamic_cfgTrue) export_to_video(video.frames[0], output.mp4, fps8)CogVideoXVideoToVideoPipeline视频生视频输入一段参考视频通过strength参数控制改造强度并可自行替换调度器示例中使用CogVideoXDPMSchedulerimport torch from diffusers import CogVideoXDPMScheduler, CogVideoXVideoToVideoPipeline from diffusers.utils import export_to_video, load_video pipe CogVideoXVideoToVideoPipeline.from_pretrained(THUDM/CogVideoX-5b, torch_dtypetorch.bfloat16) pipe.to(cuda) pipe.scheduler CogVideoXDPMScheduler.from_config(pipe.scheduler.config) input_video load_video(hiker.mp4) prompt ( An astronaut stands triumphantly at the peak of a towering mountain. Panorama of rugged peaks and valleys. Very futuristic vibe and animated aesthetic. Highlights of purple and golden colors in the scene. The sky is looks like an animated/cartoonish dream of galaxies, nebulae, stars, planets, moons, but the remainder of the scene is mostly realistic. ) video pipe(videoinput_video, promptprompt, strength0.8, guidance_scale6, num_inference_steps50).frames[0] export_to_video(video, output.mp4, fps8)CogVideoXFunControlPipeline可控视频生成面向 Alibaba-PAI 的 CogVideoX-Fun 系列检查点如alibaba-pai/CogVideoX-Fun-V1.1-5b-Pose通过control_video输入姿态等控制信号约束生成示例中调度器替换为DDIMSchedulerimport torch from diffusers import CogVideoXFunControlPipeline, DDIMScheduler from diffusers.utils import export_to_video, load_video pipe CogVideoXFunControlPipeline.from_pretrained( alibaba-pai/CogVideoX-Fun-V1.1-5b-Pose, torch_dtypetorch.bfloat16 ) pipe.scheduler DDIMScheduler.from_config(pipe.scheduler.config) pipe.to(cuda) control_video load_video(hiker.mp4) prompt ( An astronaut stands triumphantly at the peak of a towering mountain. Panorama of rugged peaks and valleys. Very futuristic vibe and animated aesthetic. Highlights of purple and golden colors in the scene. The sky is looks like an animated/cartoonish dream of galaxies, nebulae, stars, planets, moons, but the remainder of the scene is mostly realistic. ) video pipe(promptprompt, control_videocontrol_video).frames[0] export_to_video(video, output.mp4, fps8)统一的输出对象所有 CogVideo 管线均返回CogVideoXPipelineOutput其唯一字段frames的类型为torch.Tensor/np.ndarray或list[list[PIL.Image.Image]]形状为(batch_size, num_frames, channels, height, width)。访问.frames[0]即可取得单个视频的帧序列配合export_to_video导出。显存占用与优化方法对照官方文档给出了开启各类显存优化手段前后的显存占用对照表基于 5B 模型方法启用后显存占用未启用时显存占用enable_model_cpu_offload19GB33GBenable_sequential_cpu_offload4GB~33GB推理速度极慢enable_tiling配合enable_model_cpu_offload11GB—选择建议显存 ≥ 33GB可直接全量加载优先保证速度显存 1633GB优先enable_model_cpu_offload()实测约 19GB必要时叠加 int8/fp8 量化到 16GB 左右显存 8GB考虑enable_sequential_cpu_offload()4GB但需接受明显变慢的推理速度若仍不足可再叠加enable_tiling()将 VAE 解码分块执行进一步压低峰值。这三类方法都属于DiffusionPipeline的通用能力完整原理与更多技巧见 Reduce memory usage 指南。源码结构速览与测试验证管线实现pipeline_cogvideox.pyT2V、pipeline_cogvideox_image2video.pyI2V、pipeline_cogvideox_video2video.pyV2V、pipeline_cogvideox_fun_control.pyFunControl、pipeline_output.py输出定义。测试用例test_cogvideox.py、test_cogvideox_image2video.py、test_cogvideox_video2video.py、test_cogvideox_fun_control.py可用于核对各管线的参数校验、输入输出形状与内存优化开关的行为。LoRA 加载实现CogVideoXLoraLoaderMixinLoRA 权重仅注入transformer模块_lora_loadable_modules [transformer]。训练侧参考train_cogvideox_lora.py 与 train_cogvideox_image_to_video_lora.py 展示了如何为 CogVideoX 训练自定义 LoRA。小结CogVideoX 在 Diffusers 中形成了覆盖 T2V、I2V、V2V、可控生成的完整管线家族配合 int8/fp8 量化、layerwise casting、模型卸载、enable_tiling与torch.compile可以在从 4GB 到 80GB 的各级显存环境下灵活部署。使用时分清三点即可快速上手T2V 优先 1360×768、I2V 高度固定 768 且宽高能被 16 整除、81/161 帧搭配 16fps 导出需要风格定制时通过load_lora_weights加载社区 LoRA显存紧张时按内存对照表逐级叠加优化开关。【免费下载链接】diffusers Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch.项目地址: https://gitcode.com/GitHub_Trending/di/diffusers创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考