Qwen3-VL-235B-A22B-Instruct-w8a8-QuaRot 单机部署

本文介绍了在昇腾单机服务器上部署Qwen3-VL-235B-A22B-Instruct-w8a8-QuaRot模型适配Ascend 910B单节点 8 卡 TP8 镜像quay.io/ascend/vllm-ascend:v0.18.0rc1一、模型信息属性信息模型 IDEco-Tech/Qwen3-VL-235B-A22B-Instruct-w8a8-QuaRot量化格式w8a8 QuaRotINT8 权重 INT8 激活权重数量56 个 safetensors~222GB权重路径/root/.cache/modelscope/hub/models/Eco-Tech/Qwen3-VL-235B-A22B-Instruct-w8a8-QuaRot/模型名qwen-vlmax_model_len262144256k二、拉起容器2.1 安装 Docker如未安装yum install -y docker systemctl start docker systemctl enable docker2.2 创建软链接ln -s /root/.cache/modelscope/hub/models/Eco-Tech/Qwen3-VL-235B-A22B-Instruct-w8a8-QuaRot \ /root/Qwen3-VL-235B-A22B-Instruct-w8a8-QuaRot2.3 拉起容器docker run -d \ --name qwen3-vl-deploy \ --network host \ --shm-size1200g \ --privilegedtrue \ --device/dev/davinci_manager \ --device/dev/hisi_hdc \ --device/dev/devmm_svm \ -v /usr/local/dcmi:/usr/local/dcmi \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \ -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \ -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /root/Qwen3-VL-235B-A22B-Instruct-w8a8-QuaRot:/model \ -v /root/.cache:/root/.cache \ quay.io/ascend/vllm-ascend:v0.18.0rc1 \ bash -c sleep infinity必须挂载/usr/lib64/libstdc.so.6。如果启动失败加入-v /usr/lib64/libstdc.so.6:/usr/lib64/libstdc.so.6三、启动 vLLM3.1 先停其他容器释放显存gpu-memory-utilization0.94 需要几乎全部显存必须停掉其他容器。docker stop $(docker ps -q)3.2 启动 qwen3-vl-deploy 容器docker start qwen3-vl-deploy3.3 启动 vLLMdocker exec -d qwen3-vl-deploy bash -c vllm serve /root/.cache/modelscope/hub/models/Eco-Tech/Qwen3-VL-235B-A22B-Instruct-w8a8-QuaRot/ \ --served-model-name qwen-vl \ --host 0.0.0.0 --port 8800 \ --data-parallel-size 1 \ --tensor-parallel-size 8 \ --max-model-len 131072 \ --max-num-batched-tokens 16384 \ --max-num-seqs 128 \ --gpu-memory-utilization 0.94 \ --compilation-config {\cudagraph_mode\: \FULL_DECODE_ONLY\} \ --trust-remote-code \ --dtype bfloat16 \ --enable-auto-tool-choice \ --tool-call-parser hermes \ --async-scheduling \ --allowed-local-media-path / \ --quantization ascend \ --additional-config {\enable_cpu_binding\:true}四、验证等待约 3-5 分钟后vLLM 加载模型需要时间执行curl http://localhost:8800/v1/models应返回 qwen-vl。对话测试curl --location http://localhost:8800/v1/chat/completions \ --header Content-Type: application/json \ --data {model:qwen-vl,messages:[{role:user,content:hello}],max_tokens:50}五、访问服务服务启动后通过容器所在机器的 8800 端口访问http://{机器IP}:8800API 端点- GET /v1/models — 查看可用模型- POST /v1/chat/completions — 对话- POST /v1/completions — 文本补全六、重启恢复6.1 启动容器docker start qwen3-vl-deploy6.2 重新启动 vLLM先停其他容器再启动 vLLMdocker stop $(docker ps -q) docker exec -d qwen3-vl-deploy bash -c vllm serve /root/.cache/modelscope/hub/models/Eco-Tech/Qwen3-VL-235B-A22B-Instruct-w8a8-QuaRot/ \ --served-model-name qwen-vl \ --host 0.0.0.0 --port 8800 \ --data-parallel-size 1 \ --tensor-parallel-size 8 \ --max-model-len 36864 \ --max-num-batched-tokens 16384 \ --max-num-seqs 128 \ --gpu-memory-utilization 0.94 \ --compilation-config {\cudagraph_mode\: \FULL_DECODE_ONLY\} \ --trust-remote-code \ --dtype bfloat16 \ --enable-auto-tool-choice \ --tool-call-parser hermes \ --async-scheduling \ --allowed-local-media-path / \ --quantization ascend \ --additional-config {\enable_cpu_binding\:true}6.3 确认进程运行docker exec qwen3-vl-deploy bash -c ps -ef | grep vllm serve | grep -v grep七、常见问题排查7.1 查看启动日志docker logs -f qwen3-vl-deploy7.2 确认端口在监听docker exec qwen3-vl-deploy bash -c netstat -tlnp | grep 88007.3 确认 NPU 进程docker exec qwen3-vl-deploy bash -c npu-smi info | grep Process id应该看到 8 个进程TP8。7.4 常见错误显存不足ValueError: Free memory on device (4.6/60.96 GiB) ... less than desired GPU memory utilization (0.94)→docker stop $(docker ps -q)停掉其他容器tool-call-parser 错误KeyError: invalid tool call parser: qwen2 或 qwen3_coder→ Qwen3-VL 用hermes不是 qwen3_coderqwen3_coder 是编程模型专用PyTorch OpenMP 错误pool INTERNAL ASSERT FAILED at ParallelOpenMP.cpp→ 挂载主机的 libstdc.so.6/usr/lib64/libstdc.so.6vLLM 启动后立即退出→ 检查显存是否足够需停其他容器检查日志确认具体报错curl 失败但 vLLM 正在运行→ vLLM 初始化约需 3-5 分钟期间端口不监听是正常现象等待即可