vSAN手动定位及删除Inaccessible及VM相关对象

场景分享在一个健康的vSAN环境中一个健康的VM包含多个Object对象VMDK/Snapshot/Namespace/Swap等等通常不需要进行手动操作在某些特殊环境下则需要人工介入下面分享一个生产环境的案例背景某用户由于某些历史原因vSAN环境里大量的VM使用了一个自定义的存储策略RAID0FTT0在此种存储策略下虽然提升了性能及存储容量的可用性但牺牲了数据的安全性一旦有磁盘或磁盘组异常导致的Object下线则相关联的VM将完全不可用若该对象无法修复则VM彻底无法修复只能依赖备份好巧不巧某个磁盘组的SSD缓存盘故障导致整个磁盘组下线众所周知SSD损坏几乎不可恢复因此在该磁盘组上的对象彻底丢失相关VM也彻底无法访问期间想办法去做恢复等就不做展开了最后由于这些不可访问的VM占用了大量的vSAN空间也无法从常规途径删除因此需要评估将这些不可访问的VM彻底删除以回收vSAN空间初步排查及需求分析前文提到VM的正常工作依赖所有对象的相互关系整个VM中的对象异常都会导致VM无法正常运行但对于VM本身来讲并不是所有Object都完全丢失有些像vmdk对象虽然还在也占用了大量空间但缺少了相关索引关系已无法正常使用因此本次需求简单总结为以下两个任务删除不可访问VM的正常对象以释放空间清除相关Inaccessible的对象操作流程环境说明作为信息保护本文在POC环境中进行演示本演示环境为一套3节点嵌套vSAN8 OSA环境测试机调用新建的RAID0存储策略存储策略配置为RAID0FTT0进入虚拟机监控可以看到所有对象仅存在一个副本位于集群的不同主机此时嵌套环境强行关闭一个节点可以看到原来位于该节点的组件丢失VM也无法正常访问删除VM正常组件/对象首先删除VM正常对象以回收vSAN空间因此需要先定位到这些对象的UUID此时VM的目录已经混乱常规Web方式不太能准确定位推荐两种常用方式vCenter RVCESXCLI经测试esxcli方式较为靠谱Skyline也推荐该方式SSH到集群内一台正常的ESXi#grep筛选VM的名称B30为往前30行[rootlocalhost:~]esxcli vsan debug object list|grepLab-VM-RAID0-B30Group UUID: 7806af69-2c7c-cee7-9665-005056b3940e Directory Name: N/A Object UUID: a507af69-6c67-2e3d-ed21-005056b31ecc Version:20Health: healthy Owner: localhost Size:8.00GB Used:0.00GB Used 4K Blocks:0.00GB Policy: stripeWidth:1cacheReservation:0proportionalCapacity:0hostFailuresToTolerate:0forceProvisioning:1spbmProfileId: bd2cbaac-3b56-4d58-9b61-852b2f1765aa spbmProfileGenerationNumber:0iopsLimit:0checksumDisabled:0CSN:68spbmProfileName: vSAN-RAID0 Configuration: Component: a507af69-e254-4b3f-bf59-005056b31ecc Component State: ACTIVE, Address Space(B):8589934592(8.00GB), Disk UUID: 5273c4df-60cd-a59d-1e34-32e699d0076a, Disk Name: mpx.vmhba0:C0:T2:L0:2 Votes:1, Capacity Used(B):12582912(0.01GB), Physical Capacity Used(B):4194304(0.00GB), Total 4K Blocks Used(B):4194304(0.00GB), Host Name: localhost Type: vmswap Path: /vmfs/volumes/vsan:5251e1d0e576fe4f-55ce6965022d2690/4f06af69-cad0-383f-d8e9-005056b31ecc/Lab-VM-RAID0-e5d0c140.vswp(Missing)-- Group UUID: 7806af69-2c7c-cee7-9665-005056b3940e Directory Name: N/A Object UUID: 5006af69-5e31-7ae6-6bec-005056b31ecc Version:20Health: healthy Owner: localhost Size:120.00GB Used:10.06GB Used 4K Blocks:9.95GB Policy: stripeWidth:1cacheReservation:0proportionalCapacity:0hostFailuresToTolerate:0forceProvisioning:0spbmProfileId: bd2cbaac-3b56-4d58-9b61-852b2f1765aa spbmProfileGenerationNumber:0iopsLimit:0checksumDisabled:0CSN:60spbmProfileName: vSAN-RAID0 Configuration: Component: 5006af69-bc7f-80e8-3983-005056b31ecc Component State: ACTIVE, Address Space(B):128849018880(120.00GB), Disk UUID: 526c5c34-e232-85f4-2341-4180a5685d75, Disk Name: mpx.vmhba0:C0:T2:L0:2 Votes:1, Capacity Used(B):10913579008(10.16GB), Physical Capacity Used(B):10804527104(10.06GB), Total 4K Blocks Used(B):10684493824(9.95GB), Host Name: localhost Type: vdisk Path: /vmfs/volumes/vsan:5251e1d0e576fe4f-55ce6965022d2690/4f06af69-cad0-383f-d8e9-005056b31ecc/Lab-VM-RAID0.vmdk(Missing)可以看到有两个匹配的Object UUID健康状态是Healthy但是Path是Missing记录下这两个UUIDObject UUID: a507af69-6c67-2e3d-ed21-005056b31eccObject UUID: 5006af69-5e31-7ae6-6bec-005056b31ecc知道UUID后可以通过esxcli再次进行确认[rootlocalhost:~]esxcli vsan debug object list-u5006af69-5e31-7ae6-6bec-005056b31ecc Object UUID: 5006af69-5e31-7ae6-6bec-005056b31ecc Version:20Health: healthy Owner: localhost Size:120.00GB Used:10.06GB Used 4K Blocks:9.95GB Policy: stripeWidth:1cacheReservation:0proportionalCapacity:0hostFailuresToTolerate:0forceProvisioning:0spbmProfileId: bd2cbaac-3b56-4d58-9b61-852b2f1765aa spbmProfileGenerationNumber:0iopsLimit:0checksumDisabled:0CSN:60spbmProfileName: vSAN-RAID0 Configuration: Component: 5006af69-bc7f-80e8-3983-005056b31ecc Component State: ACTIVE, Address Space(B):128849018880(120.00GB), Disk UUID: 526c5c34-e232-85f4-2341-4180a5685d75, Disk Name: mpx.vmhba0:C0:T2:L0:2 Votes:1, Capacity Used(B):10913579008(10.16GB), Physical Capacity Used(B):10804527104(10.06GB), Total 4K Blocks Used(B):10684493824(9.95GB), Host Name: localhost Type: vdisk Path: /vmfs/volumes/vsan:5251e1d0e576fe4f-55ce6965022d2690/4f06af69-cad0-383f-d8e9-005056b31ecc/Lab-VM-RAID0.vmdk(Missing)Group UUID: 4f06af69-cad0-383f-d8e9-005056b31ecc Directory Name: N/A使用命令将这两个对象进行删除[rootlocalhost:~]/usr/lib/vmware/osfs/bin/objtool delete-u5006af69-5e31-7ae6-6bec-005056b31ecc-f[rootlocalhost:~]/usr/lib/vmware/osfs/bin/objtool delete-ua507af69-6c67-2e3d-ed21-005056b31ecc-f删除后可以进入WebUI确认对象已删除并且vSAN空间已正常回收清除Inaccessible对象同样先命令行找到Inaccessible对象并进行确认[rootlocalhost:~]esxcli vsan debug object list|grepinaccessible-B5Group UUID: 7806af69-2c7c-cee7-9665-005056b3940e Directory Name: N/A Object UUID: 4f06af69-cad0-383f-d8e9-005056b31ecc Version:20Health: inaccessible - Lost data availability.(APD)[rootlocalhost:~]esxcli vsan debug object list-u4f06af69-cad0-383f-d8e9-005056b31ecc Object UUID: 4f06af69-cad0-383f-d8e9-005056b31ecc Version:20Health: inaccessible - Lost data availability.(APD)Owner: 00000000-0000-0000-0000-000000000000 Size:0.00GB Used:0.14GB Used 4K Blocks:0.03GB Policy: stripeWidth:1cacheReservation:0proportionalCapacity:[0,100]hostFailuresToTolerate:0forceProvisioning:0spbmProfileId: bd2cbaac-3b56-4d58-9b61-852b2f1765aa spbmProfileGenerationNumber:0iopsLimit:0checksumDisabled:0CSN:30SCSN:2spbmProfileName: vSAN-RAID0 Configuration: Component: 4f06af69-3e86-7c41-d87c-005056b31ecc Component State: ABSENT, Address Space(B):273804165120(255.00GB), Disk UUID: 522f739e-2c27-89f3-f681-d8544aa9333c, Disk Name: mpx.vmhba0:C0:T2:L0:2 Votes:1, Capacity Used(B):155189248(0.14GB), Physical Capacity Used(B):150994944(0.14GB), Total 4K Blocks Used(B):33001472(0.03GB), Host Name: localhost Type: N/A Path: N/A Group UUID: 4f06af69-cad0-383f-d8e9-005056b31ecc Directory Name: N/A需要通过cmmds-tool进行删除[rootlocalhost:~]cmmds-tool delete-tDOM_OBJECT-u4f06af69-cad0-383f-d8e9-005056b31ecc至此完成删除总结最后希望大家除了Lab环境外在生产环境不要用到这篇文章最后两点vSAN环境非必要千万不要用RAID0牺牲数据安全性换回的那些性能确实不值如何非要用数据空间极度紧张等做好备份做好备份