cann/asc-devkit L2缓存模式优化实践

L2 Cache Mode Best Practices Sample【免费下载链接】asc-devkit本项目是CANN 推出的昇腾AI处理器专用的算子程序开发语言原生支持C和C标准规范主要由类库和语言扩展层构成提供多层级API满足多维场景算子开发诉求。项目地址: https://gitcode.com/cann/asc-devkitOverviewWhen MTE2 moves data from Global Memory (GM) to Unified Buffer (UB), thel2_cache_modeparameter of theasc_copy_gm2ub_aligninterface (this sample uses pointer-based C programming) can explicitly configure the L2 Cache management strategy. This sample describes how to select the appropriate L2 Cache mode to optimize MTE2 data movement performance forreuse dataandstreaming datascenarios, and how to improve the L2 Cache hit rate through tiling strategies when L2 Cache is enabled.Reuse Data Scenario (data needs to be read multiple times)Case1: Repeat the entire block 4 times,l2_cache_mode0(NORMAL) — the entire block far exceeds the L2 capacity, hit rate is extremely low, demonstrating the performance bottleneck without tiling.Case2: Split into 4 tiles along the N direction, repeat each tile 4 times,l2_cache_mode0(NORMAL) — after tiling, the working set per tile fits within the L2 capacity, improving the hit rate.Streaming Data Scenario (data is read only once)Case3: Add double buffer,l2_cache_mode0(NORMAL) (baseline).Case4: Same as Case3, but setl2_cache_modeto 4 (DISABLE), bypassing L2 Cache — compare with Case3.Supported Products and CANN Software VersionsProductCANN Software VersionAscend 950PR/Ascend 950DT CANN 9.1.0Directory Structure├── set_l2_cache_mode │ ├── scripts │ │ └── gen_data.py // Script for generating input data and ground truth data │ ├── CMakeLists.txt // Build configuration file │ ├── set_l2_cache_mode.asc // Sample entry (kernel call main function) │ ├── set_l2_cache_mode.h // Kernel implementation (DataCopyRepeat Add) │ ├── data_utils.h // Data read and write utility functions │ ├── README.md // Sample documentation (Chinese) │ └── README_en.md // Sample documentation (English)Sample DescriptionThe C-API GM→UB data movement interface uses thel2_cache_modeparameter to control how the data being moved is managed in the L2 Cache. For details on the values and their meanings, see the asc_copy_gm2ub_align interface documentation.Based on the two scenarios described above, this sample designs 4 Cases for comparison and verification:Data reuse scenario (Case1-2)This group uses a half-type 2D matrix with shape [12288, 12288] as input.asc_copy_gm2ub_alignmoves data from GM to UB. By comparing the L2 Cache hit rate under different data movement strategies, observe the performance difference between whole-block repetition and tiled repetition. This group does not include computation logic and only performs data movement operations.Streaming data scenario (Case3-4)This group implements the addition of two half-type matrices (z x y) with shape [8192, 8192]. It uses double buffering (Ping-Pong) to pipeline data movement and vector computation. By comparingl2_cache_mode0(NORMAL) vsl2_cache_mode4(DISABLE), demonstrate the optimization effect of bypassing L2 Cache for streaming data.Sample ImplementationPerformance Metrics DescriptionThe performance data in this section is collected on Ascend 950 series products. Ascend 950PR/Ascend 950DT is referred to as Ascend 950 series for short.Different performance collection commands are used to obtain different performance metrics:msprof ./demo— Collects AI Core instruction-level latency and ratios (MTE2/MTE3/Vector), corresponding toTable 1.msprof --ai-coreon --aic-metricsL2Cache ./demo— Collects L2 Cache read/write hit and miss counts, corresponding toTable 2.Table 1AI Core Performance Metrics DescriptionFieldDescriptionTask Duration(μs)Total execution time of the entire task. Operator execution time is measured by this parameter.aiv_total_cyclesTotal cycles for task execution.aiv_mte2_time(μs)Execution time of MTE2 type instructions (GM to AI Core data movement), in μs.aiv_mte2_ratioRatio of MTE2 instruction cycles to total cycles.aiv_mte3_time(μs)Execution time of MTE3 type instructions (UB to GM data movement), in μs.aiv_mte3_ratioRatio of MTE3 instruction cycles to total cycles.aiv_vec_time(μs)Execution time of vec type instructions (vector computation), in μs.aiv_vec_ratioRatio of vec instruction cycles to total cycles.Table 2L2 Cache Performance Metrics DescriptionFieldDescriptionTask Duration(μs)Total execution time of the entire task.aiv_total_cyclesTotal cycles for task execution.aiv_write_cache_hitNumber of write cache hits.aiv_write_cache_miss_allocateNumber of write cache misses that trigger cache reallocation.aiv_read_local_l2_hitNumber of read cache hits.aiv_read_local_l2_missNumber of read cache misses.aiv_read_local_l2_victimNumber of read cache misses that caused cache data eviction.L2 Cache hit rate aiv_read_local_l2_hit / (aiv_read_local_l2_hit aiv_read_local_l2_miss aiv_read_local_l2_victim)Data Reuse Pattern DiagramThe data movement patterns between GM and UB for Case1 and Case2 are shown below.Case1: Repeat the entire matrix 4 timesGM Matrix: [M, N] ┌─────────────────────────────── N ───────────────────────────────┐ │ All columns moved at once │ └──────────────────────────────────────────────────────────────────┘ All cores start and move the entire matrix along the same path: Round 1: All cores read the entire matrix from GM to UB Round 2: All cores read the entire matrix again to UB Round 3: All cores read the entire matrix again to UB Round 4: All cores read the entire matrix again to UB Note: The working set in each round is the entire matrix, making it difficult for L2 Cache to retain data from the previous round.Case2: Split into 4 tiles along the N direction, repeat each tile 4 timesGM Matrix: [M, N] ┌─────── N/4 ────────┬─────── N/4 ────────┬─────── N/4 ────────┬─────── N/4 ────────┐ │ Tile 0 │ Tile 1 │ Tile 2 │ Tile 3 │ └─────────────────────┴─────────────────────┴─────────────────────┴─────────────────────┘ All cores start, move Tile 0 data 4 times consecutively, then process the next tile: Tile 0: Round 1 reads from GM, Rounds 2-4 preferentially read from L2 Cache Tile 1: Round 1 reads from GM, Rounds 2-4 preferentially read from L2 Cache Tile 2: Round 1 reads from GM, Rounds 2-4 preferentially read from L2 Cache Tile 3: Round 1 reads from GM, Rounds 2-4 preferentially read from L2 Cache Note: The working set per tile is smaller, making it easier to retain in L2 Cache during consecutive repeated accesses.Case1: Repeat the entire matrix 4 times l2_cache_mode0(NORMAL)Design intent: The entire matrix (301.99MB) is larger than the L2 Cache capacity (approximately 128MB). Each full matrix movement causes old data to be evicted, resulting in a low L2 hit rate. Compare with Case2 (tiled repetition) to demonstrate the importance of controlling the working set size through tiling — even with L2 Cache enabled, if the single working set exceeds the L2 capacity, tiling is necessary to control the working set size.Sample configuration:Repeat the entire matrix along the same path 4 times.Total data moved: 301.99MB x 4 1207.96MB.Tile: [64, 1024], eachasc_copy_gm2ub_alignmoves 131,072B.L2 strategy:l2_cache_mode0(NORMAL)Performance data:ArchScenarioConfigurationTask Duration(μs)aiv_total_cyclesaiv_time(μs)aiv_read_local_l2_hitaiv_read_local_l2_missaiv_read_local_l2_victimDescriptionL2Cache Hit RateAscend 950 seriesCase1Tensor[12288,12288]Tile[64,1024]Block Num64690.3872348218689.487315345231777672752Entire matrix repeated 4 times along the same path8.2%Optimization analysis:The L2 Cache hit rate is only8.2%, meaning most repeated accesses still need to be read from GM.The entire matrix (302MB) far exceeds the L2 Cache capacity (approximately 128MB), making L2 Cache practically ineffective.The victim count is as high as 7,672,752, indicating that almost all cached data from the previous round has been evicted.Principle:When moving the entire matrix in each round, the data cached in L2 from the previous round is almost entirely evicted (victim count as high as 7,672,752). In the next round, repeated accesses still need to be read from GM again.Each miss triggers Cache Line allocation and old data eviction. These ineffective cache management operations consume additional L2 controller management bandwidth, reducing MTE2 data movement efficiency.Performance optimization suggestions:When the single working set far exceeds the L2 capacity, enabling L2 Cache alone cannot provide reuse benefits. Use tiling to control the working set size within the L2 capacity.Case2: Split into 4 tiles along N repeat each tile 4 times l2_cache_mode0(NORMAL)Design intent: For data that needs to be read multiple times, enabling L2 Cache caches the data read in the first access. Subsequent repeated accesses read directly from L2, significantly reducing MTE2 data movement latency. The input matrix (301.99MB) exceeds the L2 capacity (approximately 128MB), so the entire matrix cannot be fully cached. Splitting into 4 tiles along the N direction results in each tile being approximately 75.50MB, allowing each tile to achieve L2 hit rate improvements within a certain range.Sample configuration:Split into 4 tiles along the N direction. Repeat each tile 4 times.Total data moved: 301.99MB x 4 1207.96MB.Tile: [64, 1024], eachasc_copy_gm2ub_alignmoves 131,072B.L2 strategy:l2_cache_mode0(NORMAL)Performance data:ArchScenarioConfigurationTask Duration(μs)aiv_total_cyclesaiv_time(μs)aiv_read_local_l2_hitaiv_read_local_l2_missaiv_read_local_l2_victimDescriptionL2Cache Hit RateAscend 950 seriesCase2Tensor[12288,12288]Tile[64,1024]Block Num64350.9536233475350.03666877452867617209844 tiles along N, each tile repeated 4 times74.8%Optimization analysis:The L2 Cache hit rate increased from8.2%(Case1) to74.8%, close to the theoretical limit of 75% (in 4 rounds of data movement, the first round always misses, and rounds 2-4 ideally hit every time).Task Duration decreased from 690.38μs to 350.95μs, a reduction of49.2%.The victim count decreased from 7,672,752 to 1,720,984, indicating significantly reduced cache thrashing.Principle:The entire matrix (302MB) far exceeds the L2 capacity (128MB), so it cannot be fully cached. After splitting into 4 tiles along the N direction, each tile is approximately 75.5MB and can be effectively cached within the L2 capacity.When consecutively accessing the same tile, data from rounds 2-4 hits directly in L2, greatly reducing GM read operations.Performance optimization suggestions:For repeated accesses to data exceeding the L2 capacity, first use tiling to control the single working set within the L2 capacity, then enable L2 Cache to obtain reuse benefits.Case3: Add with double buffer l2_cache_mode0(NORMAL)Design intent: The inputs x and y of the Add operator are both streaming data (each element is read only once). Double buffering is used for pipeline parallelism. Withl2_cache_mode0(NORMAL), MTE2 writes data to the L2 Cache while moving it to UB. However, the data is never accessed again, making the L2 write operations completely wasteful and consuming additional L2 controller management bandwidth.Sample configuration:Matrix: [8192, 8192], half type (128MB per matrix).Evenly distributed across 64 AIV Cores.dataCopyLen: 21760.L2 strategy:l2_cache_mode0(NORMAL)Performance data:Task Duration(μs)aiv_time(μs)aiv_vec_time(μs)aiv_vec_ratioaiv_scalar_time(μs)aiv_scalar_ratioaiv_mte2_time(μs)aiv_mte2_ratioaiv_mte3_time(μs)aiv_mte3_ratio240.01239.1383.140.3481.8150.008233.8690.978121.440.507Optimization analysis:MTE2 writes data to the L2 Cache while moving it, wasting L2 controller management bandwidth (aiv_mte2_time reaches233.869μs, accounting for97.8%).Principle:The inputs x and y of the Add operator are both streaming data. Each element is accessed only once and is never reused.l2_cache_mode0(NORMAL) causes MTE2 to perform additional L2 Cache write operations during GM to UB data movement. These writes do not help performance and instead consume L2 controller management bandwidth and pollute the L2 Cache space.Performance optimization suggestions:For streaming data, configurel2_cache_mode4(DISABLE) to bypass L2 Cache writes, eliminating unnecessary cache management overhead.Case4: Add with double buffer l2_cache_mode4(DISABLE)Design intent: The only difference from Case3 is thatasc_copy_gm2ub_alignis called withl2_cache_mode4(DISABLE) instead ofl2_cache_mode0(NORMAL). For streaming data that is read only once, disabling L2 Cache allows MTE2 to move data directly from GM to UB, avoiding unnecessary cache write overhead and preventing streaming data from occupying L2 space.Key code:// Case 3 (l2_cache_mode0, NORMAL) and Case 4 (l2_cache_mode4, DISABLE) // The only difference is the l2_cache_mode parameter value __aicore__ inline void ProcessDoubleBufferImpl(uint8_t l2CacheMode) { // ... asc_copy_gm2ub_align(xLocal, xGm startElement, 1, (uint32_t)(curLen * sizeof(half)), 0, 0, 0, l2CacheMode, 0, 0); asc_copy_gm2ub_align(yLocal, yGm startElement, 1, (uint32_t)(curLen * sizeof(half)), 0, 0, 0, l2CacheMode, 0, 0); // ... }L2 strategy:l2_cache_mode4(DISABLE)Performance data:Task Duration(μs)aiv_time(μs)aiv_vec_time(μs)aiv_vec_ratioaiv_scalar_time(μs)aiv_scalar_ratioaiv_mte2_time(μs)aiv_mte2_ratioaiv_mte3_time(μs)aiv_mte3_ratio172.996171.8182.4620.481.8010.01166.7520.97118.7230.109Optimization analysis:Task Duration decreased from 240.01μs to 172.996μs, a reduction of27.9%.MTE2 time decreased from 233.869μs to 166.752μs, a reduction of28.7%.MTE3 time decreased from 121.44μs to 18.723μs, a reduction of84.6%.aiv_vec_time remained nearly unchanged (83.14μs to 82.462μs, a change of0.8%), indicating that L2 Cache bypass does not affect vector computation time.Principle:l2_cache_mode4(DISABLE) bypasses L2 Cache writes. MTE2 moves data directly from GM to UB, eliminating unnecessary cache write operations and Cache Line allocation overhead.Performance optimization suggestions:For large amounts of streaming data that are read only once (such as inputs to element-wise operators like Add and Mul), configurel2_cache_mode4(DISABLE) inasc_copy_gm2ub_alignto bypass unnecessary cache writes and improve MTE2 data movement efficiency.Optimization SummaryOptimization MethodCore PrincipleApplicable ScenarioL2 Cache bypassAvoid cache pollution, reduce overheadStreaming access (read once)Enable L2 CacheCache the data read in the first access using L2 Cache, subsequent repeated accesses read directly from L2 (L2 Cache bandwidth is larger than GM bandwidth)Data that is read multiple timesTiling with repeated accessLimit repeated access to a smaller data range to improve L2 Cache hit chancesReusable data size exceeds L2 Cache capacityL2 Cache Mode Decision Tree:Is the data read multiple times? ├── Yes → Use l2_cache_mode0 (NORMAL) │ Single working set L2 capacity? │ └── Yes → Tile first, then use l2_cache_mode0 (NORMAL) within each tile └── No → Use l2_cache_mode4 (DISABLE, bypass for streaming data)Build and RunIn the sample root directory, perform the following steps to build and run the sample.Set environment variablessource ${install_path}/cann/set_env.shNote:${install_path}is the CANN installation directory.Run the sampleSCENARIO_NUM1 ASC_ARCHdav-3510 mkdir -p build cd build cmake -DSCENARIO_NUM$SCENARIO_NUM -DCMAKE_ASC_ARCHITECTURES$ASC_ARCH .. make -j python3 ../scripts/gen_data.py -scenarioNum $SCENARIO_NUM ./demoBuild optionsParameterDescriptionValuesDefaultSCENARIO_NUMScenario number1-41CMAKE_ASC_RUN_MODERun modenpu,simnpuCMAKE_ASC_ARCHITECTURESNPU hardware architecturedav-3510(only this architecture is supported)dav-3510Performance profilingUse themsproftool to obtain detailed performance data:msprof ./demo # Collect MTE2/MTE3/Vector performance data. See Table 1 for field descriptions. msprof --ai-coreon --aic-metricsL2Cache ./demo # Collect L2 Cache performance data. See Table 2 for field descriptions.After collection, aPROF_-prefixed directory is generated in the current directory. The performance summary file is located in themindstudio_profiler_outputdirectory.PROF_xxxx_XXXXXX ├── device_{id} ├── host ├── mindstudio_profiler_log └── mindstudio_profiler_output ├── msprof_*.json ├── op_summary_*.csv └── README.txtTo view the specific performance analysis results:# View Task Duration and other data cat ./PROF_*/mindstudio_profiler_output/op_summary_*.csv【免费下载链接】asc-devkit本项目是CANN 推出的昇腾AI处理器专用的算子程序开发语言原生支持C和C标准规范主要由类库和语言扩展层构成提供多层级API满足多维场景算子开发诉求。项目地址: https://gitcode.com/cann/asc-devkit创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考