CANN/hcomm taskException模块代码分析

taskException Module Code Analysis【免费下载链接】hcommHCOMMHuawei Communication是HCCL的通信基础库提供通信域以及通信资源的管理能力。项目地址: https://gitcode.com/cann/hcommFeature DescriptionThe taskException module is theDFX (Design for eXcellence) exception diagnosis subsystemin the HCCL collective communication library. It is responsible for exception capture, error information parsing, diagnostic log printing, and error reporting when collective communication tasks fail to execute. This module spans two runtime environments: the Host side and the AICPU side.Host side: Receives exception callbacks from the Runtime, distributes them to different exception handling flows based on task type (general task exception or CCU task exception), reads AICPU error information, prints UB DFX register information, and reports cluster monitoring errors.AICPU side: Runs as a daemon thread that periodically checks for CQE (Completion Queue Entry) exceptions on each communication domain stream. It parses exception CQEs, organizes error information, reports to the Host side through the HDC channel, and notifies TSFW through Mailbox.Core capabilities include:Registering and unregistering exception callback functions with the RuntimeSearching for abnormal TaskInfo based on GlobalMirrorTasksReading ErrorMessageReport from the AICPU side through the HDC channelParsing CCU (Cube Compute Unit) task exceptions and restoring the error instruction context based on the CcuRep instruction representation systemPrinting UB DFX register information for hardware diagnosticsReporting CQE error information to cluster monitoringPrinting the context of up to 50 preceding tasks before the exception taskDirectory DescriptiontaskException/ ├── host/ # Host-side exception handling │ ├── hcclCommTaskException.h # TaskExceptionHost / TaskExceptionHostManager class declarations │ ├── hcclCommTaskException.cc # Host-side general exception handling implementation callback registration management │ ├── ccuTaskException.h # CcuTaskException class declaration │ ├── ccuTaskException.cc # CCU task exception handling implementation (instruction parsing, register reading, error information generation) │ └── ccu_error_info_v1.h # CCU error information data structure definitions (CcuErrorInfo, CcuLoopContext, CcuMissionContext, and so on) └── aicpu/ # AICPU-side exception handling ├── hcclCommTaskExceptionLite.h # HcclCommTaskExceptionLite class declaration └── hcclCommTaskExceptionLite.cc # AICPU-side exception detection, CQE parsing, and error reporting implementationFile RelationshipsFileFunctionDependencieshcclCommTaskException.h/.ccHost-side main entry point, registers Runtime exception callbacks, distributes to general or CCU exception handlingDepends onccuTaskException.hfor CCU type exception handling; depends onglobal_mirror_tasks.hfor finding TaskInfo; obtains ErrorMessageReport from the AICPU side through callbacksccuTaskException.h/.ccCCU task exception-specific handling, parses CcuRep instruction representation, reads hardware registersDepends on data structures inccu_error_info_v1.h; depends onccu_kernel_mgr.hfor CcuRepContext; depends onccu_urma_channel.hfor channel informationccu_error_info_v1.hCCU error information data structure definitionsReferenced byccuTaskException.h/.cc; depends onccu_rep_type_v1.hfor the CcuRepType enum definitionhcclCommTaskExceptionLite.h/.ccAICPU-side daemon thread, detects CQE exceptions and reports to HostDepends oncoll_comm_aicpu.hfor the AICPU communication domain; depends onerror_message_v2.hfor organizing ErrorMessageReport and reporting to the Host through HDCTaskException File InteractionFlow DescriptionRegistration FlowAicpu Mode Host-Side Registration FlowAicpu Mode Device-Side Registration FlowCCU Mode Registration FlowException Handling FlowAicpu Mode Host-Side Exception Handling FlowAicpu Mode Device-Side Exception Handling FlowSendTaskExceptionByMBox Error Code Conversion Table(Source:hcclCommTaskExceptionLite.cc:429-435, constants defined in:hcomm_task_scheduler_error.h)CQE sqeTypeCQE errorCodeTS Error CodeValueDescriptionUB (sqeType9)0x02TS_ERROR_HCCL_OP_UB_DDRC_FAILED0x3eaUB local side returns ERRORUB (sqeType9)0x03TS_ERROR_HCCL_OP_UB_POISON_FAILED0x3ebUB remote side returns ERRORUB (sqeType9)0x05TS_ERROR_HCCL_OP_UB_LINK_FAILED0x3ecUB network exception, taack timeoutUB (sqeType9)OtherTS_ERROR_HCCL_OTHER_ERROR0x223UB other errorSDMA (sqeType11)0x09TS_ERROR_SDMA_LINK_ERROR0x222SDMA write copy timeout acknowledgment or address decoding errorSDMA (sqeType11)0x0aTS_ERROR_SDMA_POISON_ERROR0x221SDMA read copy timeout acknowledgment or read HBM returns ERRORSDMA (sqeType11)0x08TS_ERROR_SDMA_DDRC_ERROR0x220SDMA read HBM returns ERRORSDMA (sqeType11)OtherTS_ERROR_HCCL_OTHER_ERROR0x223SDMA other errorOther sqeType-TS_ERROR_HCCL_OTHER_ERROR0x223Non-UB or SDMA errorCCU Mode Exception Handling FlowInterface Description (Class Diagram)Interface DescriptionTaskExceptionHostInterfaceTypeParametersReturn ValueDescriptionGetInstance(s32)Public static[in] deviceLogicIDTaskExceptionHost*Gets the exception handler for the specified device. Supports up to 65 devices.Register(u64)Public[in] commHandleHcclResultWrites commHandle to CommRegisterMap_. The first comm registration callsaclrtSetExceptionInfoCallback(ProcessCallback)to register the exception callback with RTS.UnRegister(u64)Public[in] commHandleHcclResultRemoves commHandle from CommRegisterMap_. When the last comm is unregistered, sets the callback to nullptr.ProcessCallback(rtExceptionInfo_t*)Public static[in] exceptionInfovoidRuntime exception callback entry point. Obtains the handler through GetInstance and forwards to Process.HandleAicpuErrorReport(rtExceptionInfo_t*, const ErrorMessageReport, const TaskInfo)Private[in] exceptionInfo, [in] errorMessage, [in] taskInfovoidHandles errors already reported by the AICPU side: prints BaseInfo/ParaInfo/GroupInfo/OpDataInfo, calls PrintUbDfxInfo, ReportErrorMsg. When ubCqeStatus is non-zero, calls GetAicpuCqeErrInfo.HandleHostErrorReport(rtExceptionInfo_t*, const TaskInfo)Private[in] exceptionInfo, [in] taskInfovoidHost-side independent exception handling: prints preceding task context for TASK_NOTIFY_WAIT and reports EI0002, prints cluster monitoring error info.ReportErrorMsg(const TaskInfo, const string, const ErrorMessageReport, rtExceptionInfo_t*)Private[in] exceptionTaskInfo, [in] groupRankContent, [in] errorMessage, [in] exceptionInfovoidReports different error codes based on task type: TASK_NOTIFY_WAIT→EI0002, UB/Write type→EI0018.ProcessException(rtExceptionInfo_t*, const TaskInfo)Private[in] exceptionInfo, [in] taskInfovoidGeneral exception handling entry point: checks hasAicpuReport_, obtains ErrorMessageReport through CollComm::GetAicpuTaskException if not reported, dispatches to HandleAicpuErrorReport or HandleHostErrorReport.PrintTaskContextInfo(uint32_t, uint32_t, uint32_t)Private[in] deviceId, [in] streamId, [in] taskIdvoidPrints up to 50 preceding task context entries before the exception task.PrintUbDfxInfo(rtExceptionInfo_t*, const ErrorMessageReport)Private[in] exceptionInfo, [in] errorMessagevoidFor UB type tasks (TASK_WRITE_WITH_NOTIFY/TASK_UB, and so on), prints UB CQE status and EID information, and reads UB DFX registers.PrintUbRegisters(s32, RdmaHandle)Private[in] devLogicId, [in] rdmaHandleHcclResultReads and prints UB DFX register information through RaGetAuxInfo.GetGroupRankInfo(const TaskInfo)Private[in] taskInfostringGets group/rankSize/rankId information from TaskInfo.GetAicpuCqeErrRemoteLocalIdByRankId(CollComm*, uint32_t, u32)Private[in] collComm, [in] rankid, [out] remoteLocalIdvoidGets the LocalId corresponding to the remote rank through RankGraph.GetAicpuCqeErrNetInstanceByRankId(CollComm*, uint32_t, string)Private[in] collComm, [in] rankid, [out] netInstanceIdvoidGets the NetInstanceId corresponding to the remote rank through RankGraph.CcuTaskExceptionInterfaceTypeParametersReturn ValueDescriptionProcessCcuException(const rtExceptionInfo_t*, const TaskInfo)Public static[in] exceptionInfo, [in] taskInfovoidMain entry point for CCU task exception handling: initializes channel mapping, traverses Missions, prints errors and registers, cleans up TaskKill status.InitChannelMap(s32, u64)Private static[in] deviceId, [in] ccuKernelHandleHcclResultInitializes the global channelId→channelHandle mappingg_channelIdToHandle.GetCcuErrorMsg(int32_t, uint16_t, const ParaCcu, vectorCcuErrorInfo)Private static[in] deviceId, [in] missionStatus, [in] ccuTaskParam, [out] errorInfoHcclResultCore method for obtaining CCU error information: reads MissionContext, finds the exception Rep, dispatches to different GenErrorInfo methods.GenErrorInfoByRepType(const ErrorInfoBase, shared_ptrCcuRepBase, vectorCcuErrorInfo)Private static[in] baseInfo, [in] repBase, [out] errorInfovoidDispatches to the corresponding GenErrorInfo method based on CcuRepType (uses HANDLER_MAP function table).GenErrorInfoLoop(const ErrorInfoBase, CcuRepContext, vectorCcuErrorInfo)Private static[in] baseInfo, [in] ctx, [out] errorInfoHcclResultParses Loop type exceptions: reads LoopContext registers, recursively parses Reps within the Loop.GenErrorInfoLoopGroup(const ErrorInfoBase, shared_ptrCcuRepBase, CcuRepContext, vectorCcuErrorInfo)Private static[in] baseInfo, [in] repBase, [in] ctx, [out] errorInfoHcclResultParses LoopGroup type exceptions: expands all Loops and parses them individually.PrintCcuUbRegisters(const vectorCcuErrorInfo, s32, const TaskInfo)Private static[in] errorInfos, [in] devLogicId, [in] taskInfoHcclResultGets CCU Jetty status and prints UB registers for error Jetties.GetCcuJettys(const CcuErrorInfo, pairCcuChannelInfo, vectorCcuJetty*)Private static[in] errorInfo, [out] ctxHcclResultGets channelId from CcuErrorInfo and obtains Jetties through the channel→endpoint→ctxPool chain.GetCcuCqeErrorInfo(const CcuErrorInfo, const TaskInfo, u32, uint8_t)Private static[in] ccuErrorInfo, [in] taskInfo, [in] locDeviceId, [in] missionStatusvoidGets remoteRankId and NetInstanceId through channelId, calls ClusterMoniterGetCcuCqeErrInfo to report to cluster monitoring.HcclCommTaskExceptionLiteInterfaceTypeParametersReturn ValueDescriptionGetInstance()Public staticNoneHcclCommTaskExceptionLiteGets the singleton instance.Init(u32)Public[in] devIdvoidInitializes the device ID.Call()PublicNonevoidDaemon thread callback entry point. Calls HandleExceptionCqe. Sets stopCall_ on failure to prevent flooding.HandleExceptionCqe()PrivateNoneHcclResultTraverses all threads in all communication domains, detects and handles CQE exceptions.GetThreadCqe(Thread*, rtLogicCqReport_t, CqeStatus)Private[in] thread, [out] cqeException, [out] cqeStatusHcclResultGets CQE exception information for the specified thread through CqReportRecv.ProcessCqe(CollCommAicpu*, const rtLogicCqReport_t, const CqeStatus, const vectorpairstring,CollCommAicpuMgr*)Private[in] aicpuComm, [in] exceptionInfo, [in] cqeStatus, [in] aicpuCommInfoHcclResultProcesses CQE exceptions: prints TaskException, reports to Host, prints all communication domain information on NotifyWait timeout.ReportErrMsg(CollCommAicpu*, const rtLogicCqReport_t)Private[in] aicpuComm, [in] exceptionInfoHcclResultChecks IsErrorReported. If not reported, generates ErrorMessageReport, reports to Host through HDC, notifies TSFW through Mailbox, and sets ErrorReported(true).GenerateErrorMessageReport(CollCommAicpu*, const TaskInfo, const rtLogicCqReport_t, ErrorMessageReport)Private[in] aicpuComm, [in] taskInfo, [in] exceptionInfo, [out] errMsgInfoHcclResultFills common fields of ErrorMessageReport based on task information, then calls GenerateTaskErrMsg to fill task-type-specific fields.GenerateTaskErrMsg(const TaskInfo, ErrorMessageReport, const rtLogicCqReport_t)Private[in] taskInfo, [out] errMsgInfo, [in] exceptionInfovoidDispatches to FillNotifyErrMsg/FillReduceErrMsg/FillDmaErrMsg/FillUbErrMsg/FillSdmaErrMsg/FillReduceInlineErrMsg based on taskType.FillNotifyErrMsg(const TaskInfo, ErrorMessageReport)Private[in] taskInfo, [out] errMsgInfovoidFills notifyId and notifyValue for NOTIFY_WAIT/NOTIFY_RECORD types.FillReduceErrMsg(const TaskInfo, ErrorMessageReport, const rtLogicCqReport_t)Private[in] taskInfo, [out] errMsgInfo, [in] exceptionInfovoidFills reduceOp, notifyId, locEid, rmtEid, ubCqeStatus, and so on for UB_REDUCE_INLINE/WRITE_REDUCE_WITH_NOTIFY types.FillDmaErrMsg(const TaskInfo, ErrorMessageReport, const rtLogicCqReport_t)Private[in] taskInfo, [out] errMsgInfo, [in] exceptionInfovoidFills for UB_INLINE_WRITE/WRITE_WITH_NOTIFY types. Internally calls FillUbErrMsg.FillUbErrMsg(const TaskInfo, ErrorMessageReport, const rtLogicCqReport_t)Private[in] taskInfo, [out] errMsgInfo, [in] exceptionInfovoidFills locEid, rmtEid, ubCqeStatus, linkType, size, and so on for UB types.FillSdmaErrMsg(const TaskInfo, ErrorMessageReport)Private[in] taskInfo, [out] errMsgInfovoidFills linkType, size, srcAddr, dstAddr for SDMA types.FillReduceInlineErrMsg(const TaskInfo, ErrorMessageReport)Private[in] taskInfo, [out] errMsgInfovoidFills reduceOp for REDUCE_INLINE types.SendTaskExceptionByMBox(u32, u32, const rtLogicCqReport_t)Private[in] notifyId, [in] tsId, [in] exceptionInfoHcclResultReports task exception events to TSFW through Mailbox, including error code conversion (UB/SDMA error codes to TS error codes).SwitchUBCqeErrCodeToTsErrCode(u32)Private[in] cqeErrCodeuint16_tConverts UB CQE error codes to TS error codes.SwitchSdmaCqeErrCodeToTsErrCode(u32)Private[in] cqeErrCodeuint16_tConverts SDMA CQE error codes to TS error codes.PrintAllCommTaskException()PublicNoneHcclResultPrints all Task exception information for all communication domains.PrintCommTaskException(CollCommAicpu*)Public[in] aicpuCommHcclResultPrints Task exception information for all threads in the specified communication domain.PrintTaskExceptionBySqeId(CollCommAicpu*, u32, u32)Private[in] aicpuComm, [in] sqId, [in] sqeIdHcclResultPrints Task exception information for the specified sqId/sqeId: BaseInfo/ParaInfo, EID, GroupInfo, OpData/TaskContext.PrintTaskContextInfo(CollCommAicpu*, u32, u32)Private[in] aicpuComm, [in] sqId, [in] taskIdHcclResultPrints up to 50 preceding task context entries before the exception task (printed in segments by opIndex).CollectTaskContext(CollCommAicpu*, u32, u32, vectorshared_ptrTaskInfo)Private[in] aicpuComm, [in] sqId, [in] taskId, [out] taskContextHcclResultCollects up to 50 task entries before the exception task from the MirrorTaskManagerLite queue.PrintEid(const TaskInfo)Private[in] taskInfovoidPrints localEid and remoteEid for UB type tasks.GetGroupInfo(CollCommAicpu*)Private[in] aicpuCommstringGets the group/rankSize/localRank information of the communication domain.GetSqeId(uint16_t, uint16_t)Private[in] taskId, [in] streamIdu32Combines taskId and streamId into sqeId.Global Callback Registration InterfacesInterfaceParametersDescriptionRegisterGetAicpuCqeErrInfoCallBackHcomm(callback)AICPU CQE error info callbackRegisters the callback for AICPU CQE error information reporting to cluster monitoring.RegisterAicpuGetErrStatusVecCallBack(callback)AICPU error status vector callbackRegisters the callback for getting the AICPU-side abnormal device status list.RegisterGetCcuCqeErrInfoCallBackHcomm(callback)CCU CQE error info callbackRegisters the callback for CCU CQE error information reporting to cluster monitoring.RegisterCcuGetErrStatusVecCallBack(callback)CCU error status vector callbackRegisters the callback for getting the CCU-side abnormal device status list.Usage LimitationsSupported ScenariosChipModeHost SideAICPU SideDescriptionAscend 950PR/Ascend 950DTAICPUSupportedSupportedIndop tasks use the new flow; non-Indop tasks fall back to legacy TaskExceptionHandler.Ascend 950PR/Ascend 950DTCCUSupportedNot applicableTASK_CCU type is handled by CcuTaskException.Ascend 950PR/Ascend 950DTMC2Fall back to legacyNot applicableFalls back to TaskExceptionHandler under the legacy directory.Constraint SpecificationsDevice count limit: Supports up to 65 devices (MAX_MODULE_DEVICE_NUM_V2 65).TaskExceptionHostManager::GetHandlerreturns nullptr when devId 65.CCU message length: CCU transfer length must not exceed 256MB (CCU_MSG_256MB_LEN). A warning is printed when exceeded.AICPU duplicate report prevention: The Host side uses thehasAicpuReport_flag to prevent the same TaskExceptionHost from repeatedly handling AICPU-reported errors. The AICPU side uses theCollCommAicpu::IsErrorReported()flag to prevent duplicate reporting from the same communication domain.Host-side comm registration management: UsesCommRegisterMap_to manage commHandle registration. The Process callback validates whether the commHandle is registered. The RTS callback is cleared when the last comm is unregistered.AICPU-side flooding prevention:HcclCommTaskExceptionLitesetsstopCall_trueafterHandleExceptionCqefails, stopping subsequent calls.CQE error information retrieval: The CCU side uses theisGetCqeErrInfoatomic flag to avoid redundant CQE error information retrieval.Cluster monitoring report limit: The abnormal device information list is limited to up to 3 entries (maxListSize 3).Task context print limit: Prints up to 50 task entries before the exception task (TASK_CONTEXT_SIZE 50). The single print length does not exceedTASK_CONTEXT_INFO_SIZE.CCU Mission count: CurrentlyccuMissionNumis 1, only handling a single Mission exception.CCU instruction backtracking limit: Prints up to 10 instructions before the error instruction (loopUpInstrNum 10).CCU Loop expansion: Instructions within a Loop are recursively expanded throughGenErrorInfoLoop. LoopGroup expands all Loops throughGenErrorInfoLoopGroup.Error code conversion: AICPU-side UB and SDMA error codes must be converted to TS error codes before reporting through Mailbox. Unrecognized error codes are uniformly converted toTS_ERROR_HCCL_OTHER_ERROR.Thread safety:g_communicatorCallbackMapV2andg_channelIdToHandleare both protected by mutexes for concurrent access.【免费下载链接】hcommHCOMMHuawei Communication是HCCL的通信基础库提供通信域以及通信资源的管理能力。项目地址: https://gitcode.com/cann/hcomm创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考