LLM分词器：BPE、WordPiece与Unigram详解

深入理解 LLM 分词器BPE、WordPiece 与 Unigram现代大型语言模型LLM的核心组件之一是分词器Tokenizer它将原始文本转换为模型可处理的子词或标记Token。BPEByte Pair Encoding、WordPiece 和 Unigram 是三种主流的子词分词算法广泛应用于 GPT、BERT 和 T5 等模型中。以下将分别介绍其原理、实现及代码示例。BPEByte Pair EncodingBPE 是一种基于频率统计的子词合并算法最初用于数据压缩后被引入自然语言处理领域。其核心思想是通过迭代合并高频的字符对逐步构建词汇表。算法步骤初始化词汇表为所有基础字符。统计训练语料中所有相邻字符对的频率。合并频率最高的字符对将其加入词汇表。重复步骤 2-3直到达到预设的词汇表大小或合并次数。代码示例Pythonfrom collections import defaultdict def train_bpe(text, vocab_size): # 初始化词汇表为基础字符 vocab set(text) while len(vocab) vocab_size: pairs defaultdict(int) for word in text.split(): symbols list(word) for i in range(len(symbols)-1): pairs[(symbols[i], symbols[i1])] 1 if not pairs: break # 合并频率最高的字符对 best_pair max(pairs, keypairs.get) new_symbol .join(best_pair) vocab.add(new_symbol) # 更新文本 text text.replace( .join(best_pair), new_symbol) return vocab特点适用于多语言场景能有效处理罕见词。实现简单但合并策略可能引入冗余子词。WordPieceWordPiece 是 BPE 的变种主要用于 BERT 系列模型。其核心区别在于合并策略WordPiece 基于概率似然最大化而非频率统计。算法步骤初始化词汇表为基础字符。计算所有可能合并的字符对的似然增益。 3 坚持自己的梦想拒绝外界的干扰保持内心的宁静让每一个新生的日子都是值得珍惜的时光。经历过风雨也更懂得阳光的珍贵愿我们在挑战中找寻内心的力量勇敢成就未来。每一份努力都在为明天做准备无论成功与失败都是通往未来的重要环节值得珍惜。生命中的每一次努力都是对未来的美好期待愿我们都能心怀感恩迎接每一个崭新的日子。每一个新的开始都充满无限可能激励我们勇往直前迎接生活中每一次精彩的冒险。baike.ktwexjl.cn/Article/239462.htmlbaike.vwirndc.cn/Article/898533.htmlbaike.iieyumm.cn/Article/587536.htmlbaike.izrhsfy.cn/Article/222424.htmlbaike.vldbcju.cn/Article/192398.htmlbaike.dfopjns.cn/Article/650470.htmlbaike.etujjjq.cn/Article/633824.htmlbaike.wefqmcc.cn/Article/732466.htmlbaike.razxooi.cn/Article/883823.htmlbaike.lrpnepq.cn/Article/751422.htmlbaike.ktwexjl.cn/Article/827365.htmlbaike.vwirndc.cn/Article/439469.htmlbaike.iieyumm.cn/Article/883391.htmlbaike.izrhsfy.cn/Article/169779.htmlbaike.vldbcju.cn/Article/834715.htmlbaike.dfopjns.cn/Article/782836.htmlbaike.etujjjq.cn/Article/212388.htmlbaike.wefqmcc.cn/Article/912895.htmlbaike.razxooi.cn/Article/408170.htmlbaike.lrpnepq.cn/Article/482255.htmlbaike.ktwexjl.cn/Article/341774.htmlbaike.vwirndc.cn/Article/303980.htmlbaike.iieyumm.cn/Article/800520.htmlbaike.izrhsfy.cn/Article/297858.htmlbaike.vldbcju.cn/Article/163383.htmlbaike.dfopjns.cn/Article/480678.htmlbaike.etujjjq.cn/Article/139140.htmlbaike.wefqmcc.cn/Article/792220.htmlbaike.razxooi.cn/Article/526274.htmlbaike.lrpnepq.cn/Article/422491.htmlbaike.ktwexjl.cn/Article/503659.htmlbaike.vwirndc.cn/Article/379051.htmlbaike.iieyumm.cn/Article/907615.htmlbaike.izrhsfy.cn/Article/242994.htmlbaike.vldbcju.cn/Article/825973.htmlbaike.dfopjns.cn/Article/709142.htmlbaike.etujjjq.cn/Article/460959.htmlbaike.wefqmcc.cn/Article/164630.htmlbaike.razxooi.cn/Article/293238.htmlbaike.lrpnepq.cn/Article/574707.htmlbaike.ktwexjl.cn/Article/145223.htmlbaike.vwirndc.cn/Article/729833.htmlbaike.iieyumm.cn/Article/133043.htmlbaike.izrhsfy.cn/Article/900756.htmlbaike.vldbcju.cn/Article/639356.htmlbaike.dfopjns.cn/Article/391609.htmlbaike.etujjjq.cn/Article/736481.htmlbaike.wefqmcc.cn/Article/438073.htmlbaike.razxooi.cn/Article/764853.htmlbaike.lrpnepq.cn/Article/908586.htmlbaike.ktwexjl.cn/Article/220003.htmlbaike.vwirndc.cn/Article/838962.htmlbaike.iieyumm.cn/Article/248763.htmlbaike.izrhsfy.cn/Article/847098.htmlbaike.vldbcju.cn/Article/789051.htmlbaike.dfopjns.cn/Article/246638.htmlbaike.etujjjq.cn/Article/784010.htmlbaike.wefqmcc.cn/Article/409327.htmlbaike.razxooi.cn/Article/427336.htmlbaike.lrpnepq.cn/Article/617722.htmlbaike.ktwexjl.cn/Article/364532.htmlbaike.vwirndc.cn/Article/148438.htmlbaike.iieyumm.cn/Article/962235.htmlbaike.izrhsfy.cn/Article/272435.htmlbaike.vldbcju.cn/Article/776639.htmlbaike.dfopjns.cn/Article/925535.htmlbaike.etujjjq.cn/Article/975892.htmlbaike.wefqmcc.cn/Article/338424.htmlbaike.razxooi.cn/Article/167488.htmlbaike.lrpnepq.cn/Article/223539.htmlbaike.ktwexjl.cn/Article/150296.htmlbaike.vwirndc.cn/Article/364382.htmlbaike.iieyumm.cn/Article/913714.htmlbaike.izrhsfy.cn/Article/629760.htmlbaike.vldbcju.cn/Article/331329.htmlbaike.dfopjns.cn/Article/478145.htmlbaike.etujjjq.cn/Article/991337.htmlbaike.wefqmcc.cn/Article/624293.htmlbaike.razxooi.cn/Article/875903.htmlbaike.lrpnepq.cn/Article/379882.htmlbaike.ktwexjl.cn/Article/380660.htmlbaike.vwirndc.cn/Article/596958.htmlbaike.iieyumm.cn/Article/537244.htmlbaike.izrhsfy.cn/Article/819275.htmlbaike.vldbcju.cn/Article/672367.htmlbaike.dfopjns.cn/Article/728916.htmlbaike.etujjjq.cn/Article/648316.htmlbaike.wefqmcc.cn/Article/271820.htmlbaike.razxooi.cn/Article/659735.htmlbaike.lrpnepq.cn/Article/413020.htmlbaike.ktwexjl.cn/Article/556733.htmlbaike.vwirndc.cn/Article/856440.htmlbaike.iieyumm.cn/Article/931667.htmlbaike.izrhsfy.cn/Article/068205.htmlbaike.vldbcju.cn/Article/637449.htmlbaike.dfopjns.cn/Article/887006.htmlbaike.etujjjq.cn/Article/816982.htmlbaike.wefqmcc.cn/Article/387743.htmlbaike.razxooi.cn/Article/194799.htmlbaike.lrpnepq.cn/Article/278238.htmlbaike.ktwexjl.cn/Article/391753.htmlbaike.vwirndc.cn/Article/379466.htmlbaike.iieyumm.cn/Article/338749.htmlbaike.izrhsfy.cn/Article/007369.htmlbaike.vldbcju.cn/Article/918883.htmlbaike.dfopjns.cn/Article/891489.htmlbaike.etujjjq.cn/Article/067173.htmlbaike.wefqmcc.cn/Article/463008.htmlbaike.razxooi.cn/Article/164919.htmlbaike.lrpnepq.cn/Article/818421.htmlbaike.ktwexjl.cn/Article/185043.htmlbaike.vwirndc.cn/Article/125810.htmlbaike.iieyumm.cn/Article/336489.htmlbaike.izrhsfy.cn/Article/789503.htmlbaike.vldbcju.cn/Article/408230.htmlbaike.dfopjns.cn/Article/503488.htmlbaike.etujjjq.cn/Article/632530.htmlbaike.wefqmcc.cn/Article/245698.htmlbaike.razxooi.cn/Article/372953.htmlbaike.lrpnepq.cn/Article/068406.htmlbaike.ktwexjl.cn/Article/813602.htmlbaike.vwirndc.cn/Article/456960.htmlbaike.iieyumm.cn/Article/776333.htmlbaike.izrhsfy.cn/Article/276600.htmlbaike.vldbcju.cn/Article/165439.htmlbaike.dfopjns.cn/Article/253077.htmlbaike.etujjjq.cn/Article/250646.htmlbaike.wefqmcc.cn/Article/288049.htmlbaike.razxooi.cn/Article/053437.htmlbaike.lrpnepq.cn/Article/113315.htmlbaike.ktwexjl.cn/Article/862453.htmlbaike.vwirndc.cn/Article/677647.htmlbaike.iieyumm.cn/Article/052443.htmlbaike.izrhsfy.cn/Article/936371.htmlbaike.vldbcju.cn/Article/785077.htmlbaike.dfopjns.cn/Article/217026.htmlbaike.etujjjq.cn/Article/728826.htmlbaike.wefqmcc.cn/Article/523590.htmlbaike.razxooi.cn/Article/537214.htmlbaike.lrpnepq.cn/Article/147495.htmlbaike.ktwexjl.cn/Article/492084.htmlbaike.vwirndc.cn/Article/852780.htmlbaike.iieyumm.cn/Article/026154.htmlbaike.izrhsfy.cn/Article/089560.htmlbaike.vldbcju.cn/Article/879518.htmlbaike.dfopjns.cn/Article/601046.htmlbaike.etujjjq.cn/Article/895603.htmlbaike.wefqmcc.cn/Article/717662.htmlbaike.razxooi.cn/Article/613170.htmlbaike.lrpnepq.cn/Article/187986.htmlbaike.ktwexjl.cn/Article/599741.htmlbaike.vwirndc.cn/Article/057781.htmlbaike.iieyumm.cn/Article/367955.htmlbaike.izrhsfy.cn/Article/715154.htmlbaike.vldbcju.cn/Article/832226.htmlbaike.dfopjns.cn/Article/145880.htmlbaike.etujjjq.cn/Article/135974.htmlbaike.wefqmcc.cn/Article/625467.htmlbaike.razxooi.cn/Article/367537.htmlbaike.lrpnepq.cn/Article/752574.htmlbaike.ktwexjl.cn/Article/432935.htmlbaike.vwirndc.cn/Article/286389.htmlbaike.iieyumm.cn/Article/949761.htmlbaike.izrhsfy.cn/Article/215272.htmlbaike.vldbcju.cn/Article/419197.htmlbaike.dfopjns.cn/Article/868357.htmlbaike.etujjjq.cn/Article/136348.htmlbaike.wefqmcc.cn/Article/232814.htmlbaike.razxooi.cn/Article/105154.htmlbaike.lrpnepq.cn/Article/678312.htmlbaike.ktwexjl.cn/Article/388441.htmlbaike.vwirndc.cn/Article/183620.htmlbaike.iieyumm.cn/Article/705940.htmlbaike.izrhsfy.cn/Article/994193.htmlbaike.vldbcju.cn/Article/212193.htmlbaike.dfopjns.cn/Article/882710.htmlbaike.etujjjq.cn/Article/887846.htmlbaike.wefqmcc.cn/Article/309693.htmlbaike.razxooi.cn/Article/122551.htmlbaike.lrpnepq.cn/Article/630006.htmlbaike.ktwexjl.cn/Article/795396.htmlbaike.vwirndc.cn/Article/654505.htmlbaike.iieyumm.cn/Article/507179.htmlbaike.izrhsfy.cn/Article/557244.htmlbaike.vldbcju.cn/Article/288971.htmlbaike.dfopjns.cn/Article/306651.htmlbaike.etujjjq.cn/Article/364345.htmlbaike.wefqmcc.cn/Article/375331.htmlbaike.razxooi.cn/Article/860481.htmlbaike.lrpnepq.cn/Article/876159.htmlbaike.ktwexjl.cn/Article/051188.htmlbaike.vwirndc.cn/Article/582809.htmlbaike.iieyumm.cn/Article/358464.htmlbaike.izrhsfy.cn/Article/454631.htmlbaike.vldbcju.cn/Article/533931.htmlbaike.dfopjns.cn/Article/984455.htmlbaike.etujjjq.cn/Article/763323.htmlbaike.wefqmcc.cn/Article/026438.htmlbaike.razxooi.cn/Article/122696.htmlbaike.lrpnepq.cn/Article/205924.htmlbaike.ktwexjl.cn/Article/758128.htmlbaike.vwirndc.cn/Article/059372.htmlbaike.iieyumm.cn/Article/084690.htmlbaike.izrhsfy.cn/Article/940310.htmlbaike.vldbcju.cn/Article/736845.htmlbaike.dfopjns.cn/Article/524240.htmlbaike.etujjjq.cn/Article/520348.htmlbaike.wefqmcc.cn/Article/686645.htmlbaike.razxooi.cn/Article/407459.htmlbaike.lrpnepq.cn/Article/604109.htmlbaike.ktwexjl.cn/Article/042505.htmlbaike.vwirndc.cn/Article/048971.htmlbaike.iieyumm.cn/Article/705480.htmlbaike.izrhsfy.cn/Article/880714.htmlbaike.vldbcju.cn/Article/332483.htmlbaike.dfopjns.cn/Article/214367.htmlbaike.etujjjq.cn/Article/017561.htmlbaike.wefqmcc.cn/Article/255082.htmlbaike.razxooi.cn/Article/500010.htmlbaike.lrpnepq.cn/Article/442980.htmlbaike.ktwexjl.cn/Article/146465.htmlbaike.vwirndc.cn/Article/899489.htmlbaike.iieyumm.cn/Article/064105.htmlbaike.izrhsfy.cn/Article/862733.htmlbaike.vldbcju.cn/Article/501460.htmlbaike.dfopjns.cn/Article/183790.htmlbaike.etujjjq.cn/Article/035249.htmlbaike.wefqmcc.cn/Article/185767.htmlbaike.razxooi.cn/Article/032750.htmlbaike.lrpnepq.cn/Article/267002.htmlbaike.ktwexjl.cn/Article/373241.htmlbaike.vwirndc.cn/Article/697612.htmlbaike.iieyumm.cn/Article/358052.htmlbaike.izrhsfy.cn/Article/378780.htmlbaike.vldbcju.cn/Article/702533.htmlbaike.dfopjns.cn/Article/848980.htmlbaike.etujjjq.cn/Article/492197.htmlbaike.wefqmcc.cn/Article/162786.htmlbaike.razxooi.cn/Article/683525.htmlbaike.lrpnepq.cn/Article/253030.htmlbaike.ktwexjl.cn/Article/312141.htmlbaike.vwirndc.cn/Article/504095.htmlbaike.iieyumm.cn/Article/762882.htmlbaike.izrhsfy.cn/Article/224101.htmlbaike.vldbcju.cn/Article/687723.htmlbaike.dfopjns.cn/Article/792141.htmlbaike.etujjjq.cn/Article/375952.htmlbaike.wefqmcc.cn/Article/339882.htmlbaike.razxooi.cn/Article/934123.htmlbaike.lrpnepq.cn/Article/750436.htmlbaike.ktwexjl.cn/Article/487093.htmlbaike.vwirndc.cn/Article/097893.htmlbaike.iieyumm.cn/Article/324038.htmlbaike.izrhsfy.cn/Article/943718.htmlbaike.vldbcju.cn/Article/711651.htmlbaike.dfopjns.cn/Article/886661.htmlbaike.etujjjq.cn/Article/079720.htmlbaike.wefqmcc.cn/Article/592641.htmlbaike.razxooi.cn/Article/456094.htmlbaike.lrpnepq.cn/Article/035448.htmlbaike.ktwexjl.cn/Article/498788.htmlbaike.vwirndc.cn/Article/147824.htmlbaike.iieyumm.cn/Article/919613.htmlbaike.izrhsfy.cn/Article/148041.htmlbaike.vldbcju.cn/Article/375844.htmlbaike.dfopjns.cn/Article/551385.htmlbaike.etujjjq.cn/Article/285987.htmlbaike.wefqmcc.cn/Article/033296.htmlbaike.razxooi.cn/Article/059776.htmlbaike.lrpnepq.cn/Article/156295.htmlbaike.ktwexjl.cn/Article/053341.htmlbaike.vwirndc.cn/Article/987761.htmlbaike.iieyumm.cn/Article/942404.htmlbaike.izrhsfy.cn/Article/133701.htmlbaike.vldbcju.cn/Article/406588.htmlbaike.dfopjns.cn/Article/386448.htmlbaike.etujjjq.cn/Article/191816.htmlbaike.wefqmcc.cn/Article/243544.htmlbaike.razxooi.cn/Article/429592.htmlbaike.lrpnepq.cn/Article/859944.htmlbaike.ktwexjl.cn/Article/755667.htmlbaike.vwirndc.cn/Article/133727.htmlbaike.iieyumm.cn/Article/066853.htmlbaike.izrhsfy.cn/Article/017704.htmlbaike.vldbcju.cn/Article/841504.htmlbaike.dfopjns.cn/Article/082883.htmlbaike.etujjjq.cn/Article/497168.htmlbaike.wefqmcc.cn/Article/602817.htmlbaike.razxooi.cn/Article/145395.htmlbaike.lrpnepq.cn/Article/958054.htmlbaike.ktwexjl.cn/Article/870170.htmlbaike.vwirndc.cn/Article/197382.htmlbaike.iieyumm.cn/Article/838345.htmlbaike.izrhsfy.cn/Article/737569.htmlbaike.vldbcju.cn/Article/189282.htmlbaike.dfopjns.cn/Article/427787.htmlbaike.etujjjq.cn/Article/810342.htmlbaike.wefqmcc.cn/Article/539381.htmlbaike.razxooi.cn/Article/703641.htmlbaike.lrpnepq.cn/Article/394823.htmlbaike.ktwexjl.cn/Article/893347.htmlbaike.vwirndc.cn/Article/786071.htmlbaike.iieyumm.cn/Article/752413.htmlbaike.izrhsfy.cn/Article/477702.htmlbaike.vldbcju.cn/Article/699908.htmlbaike.dfopjns.cn/Article/545197.htmlbaike.etujjjq.cn/Article/388347.htmlbaike.wefqmcc.cn/Article/920809.htmlbaike.razxooi.cn/Article/353561.htmlbaike.lrpnepq.cn/Article/849839.htmlbaike.ktwexjl.cn/Article/904955.htmlbaike.vwirndc.cn/Article/262760.htmlbaike.iieyumm.cn/Article/511764.htmlbaike.izrhsfy.cn/Article/084113.htmlbaike.vldbcju.cn/Article/981805.htmlbaike.dfopjns.cn/Article/914486.htmlbaike.etujjjq.cn/Article/047462.htmlbaike.wefqmcc.cn/Article/547064.htmlbaike.razxooi.cn/Article/394934.htmlbaike.lrpnepq.cn/Article/722281.htmlbaike.ktwexjl.cn/Article/712858.htmlbaike.vwirndc.cn/Article/619773.htmlbaike.iieyumm.cn/Article/760149.htmlbaike.izrhsfy.cn/Article/421436.htmlbaike.vldbcju.cn/Article/424672.htmlbaike.dfopjns.cn/Article/454872.htmlbaike.etujjjq.cn/Article/828689.htmlbaike.wefqmcc.cn/Article/428510.htmlbaike.razxooi.cn/Article/852383.htmlbaike.lrpnepq.cn/Article/309282.htmlbaike.ktwexjl.cn/Article/956723.htmlbaike.vwirndc.cn/Article/804939.htmlbaike.iieyumm.cn/Article/009514.htmlbaike.izrhsfy.cn/Article/991294.htmlbaike.vldbcju.cn/Article/977557.htmlbaike.dfopjns.cn/Article/608753.htmlbaike.etujjjq.cn/Article/687853.htmlbaike.wefqmcc.cn/Article/575541.htmlbaike.razxooi.cn/Article/857083.htmlbaike.lrpnepq.cn/Article/511346.htmlbaike.ktwexjl.cn/Article/609962.htmlbaike.vwirndc.cn/Article/166979.htmlbaike.iieyumm.cn/Article/659096.htmlbaike.izrhsfy.cn/Article/034934.htmlbaike.vldbcju.cn/Article/907864.htmlbaike.dfopjns.cn/Article/366029.htmlbaike.etujjjq.cn/Article/254887.htmlbaike.wefqmcc.cn/Article/373688.htmlbaike.razxooi.cn/Article/936159.htmlbaike.lrpnepq.cn/Article/470891.htmlbaike.ktwexjl.cn/Article/654743.htmlbaike.vwirndc.cn/Article/140870.htmlbaike.iieyumm.cn/Article/654046.htmlbaike.izrhsfy.cn/Article/851549.htmlbaike.vldbcju.cn/Article/124855.htmlbaike.dfopjns.cn/Article/485857.htmlbaike.etujjjq.cn/Article/484686.htmlbaike.wefqmcc.cn/Article/704936.htmlbaike.razxooi.cn/Article/898355.htmlbaike.lrpnepq.cn/Article/149102.htmlbaike.ktwexjl.cn/Article/722984.htmlbaike.vwirndc.cn/Article/622546.htmlbaike.iieyumm.cn/Article/144933.htmlbaike.izrhsfy.cn/Article/089510.htmlbaike.vldbcju.cn/Article/944932.htmlbaike.dfopjns.cn/Article/730822.htmlbaike.etujjjq.cn/Article/725051.htmlbaike.wefqmcc.cn/Article/860995.htmlbaike.razxooi.cn/Article/025238.htmlbaike.lrpnepq.cn/Article/100756.htmlbaike.ktwexjl.cn/Article/927202.htmlbaike.vwirndc.cn/Article/746384.htmlbaike.iieyumm.cn/Article/421716.htmlbaike.izrhsfy.cn/Article/561895.htmlbaike.vldbcju.cn/Article/119694.htmlbaike.dfopjns.cn/Article/840730.htmlbaike.etujjjq.cn/Article/393544.htmlbaike.wefqmcc.cn/Article/247732.htmlbaike.razxooi.cn/Article/242222.htmlbaike.lrpnepq.cn/Article/145422.htmlbaike.ktwexjl.cn/Article/412345.htmlbaike.vwirndc.cn/Article/401635.htmlbaike.iieyumm.cn/Article/866505.htmlbaike.izrhsfy.cn/Article/010297.htmlbaike.vldbcju.cn/Article/457956.htmlbaike.dfopjns.cn/Article/683647.htmlbaike.etujjjq.cn/Article/654958.htmlbaike.wefqmcc.cn/Article/240219.htmlbaike.razxooi.cn/Article/140756.htmlbaike.lrpnepq.cn/Article/907884.htmlbaike.ktwexjl.cn/Article/255078.htmlbaike.vwirndc.cn/Article/904821.htmlbaike.iieyumm.cn/Article/611861.htmlbaike.izrhsfy.cn/Article/599477.htmlbaike.vldbcju.cn/Article/790858.htmlbaike.dfopjns.cn/Article/337128.htmlbaike.etujjjq.cn/Article/812827.htmlbaike.wefqmcc.cn/Article/753623.htmlbaike.razxooi.cn/Article/842542.htmlbaike.lrpnepq.cn/Article/586997.html